docs/userguide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Run the blueprint processor locally in an IDE, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15 Check out the code
  16 ~~~~~~~~~~~~~~~~~~~
  17
  18 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  19
  20 Build it locally
  21 ~~~~~~~~~~~~~~~~~~
  22
  23 In the checked out directory, type
  24
  25 .. code-block:: bash
  26
  27    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  28
  29 Wait for the maven install command to finish until you go further.
  30
  31 Spin up a Docker container with the database
  32 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  33
  34 The Blueprints Processor project uses a database to store information about the blueprints
  35 and therefore it needs to be online before attempting to run it.
  36
  37 One way to create the database is by using the :file:`docker-compose.yaml` file.
  38 This database will require a local directory to mount a volume, therefore before running docker-compose create following directory:
  39
  40 .. code-block:: bash
  41
  42    mkdir -p -m 755 /opt/app/cds/mysql/data
  43
  44 Navigate to the docker-compose file in the distribution module:
  45
  46 .. tabs::
  47
  48    .. group-tab:: Frankfurt - Latest
  49
  50       .. code-block:: bash
  51
  52          cd ms/blueprintsprocessor/application/src/main/dc
  53
  54    .. group-tab:: El Alto - Dublin
  55
  56       .. code-block:: bash
  57
  58          ms/blueprintsprocessor/distribution/src/main/dc
  59
  60 And run docker-composer:
  61
  62 .. code-block:: bash
  63
  64     docker-compose up -d db
  65
  66 This should spin up a container of the MariaDB image in the background.
  67 To check if it has worked, this command can be used:
  68
  69 .. code-block:: bash
  70
  71     docker-compose logs -f
  72
  73 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  74
  75 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  76 it can be started again by the command:
  77
  78 .. code-block:: bash
  79
  80    docker start <id of mariadb container>
  81
  82 Set permissions on the local file system
  83 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  84
  85 Blueprints processor uses the local file system for some operations and, therefore,
  86 need some existing and accessible paths to run properly.
  87
  88 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
  89
  90 .. code-block:: bash
  91
  92    mkdir -p -m 755 /opt/app/onap/blueprints/archive
  93    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
  94    mkdir -p -m 755 /opt/app/onap/scripts
  95    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
  96
  97 Import the project into the IDE
  98 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  99
 100 .. tabs::
 101
 102    .. tab:: IntelliJ IDEA
 103
 104       .. note::
 105          This is the recommended IDE for running CDS blueprint processor.
 106
 107       Go to *File | Open* and choose the :file:`pom.xml` file of the cds/ms/blueprintprocessor directory:
 108
 109       |imageImportProject|
 110
 111       Import as a project. Sometimes it may be necessary to reimport Maven project, e.g. if some dependencies can't be found:
 112
 113       |imageReimportMaven|
 114
 115       **Override some application properties:**
 116
 117       Next steps will create a run configuration profile overriding some application properties with custom values,
 118       to reflect the local environment characteristics.
 119
 120       .. tabs::
 121
 122          .. group-tab:: Frankfurt - Latest
 123
 124             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 125
 126             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 127
 128             After dependencies are imported and indexes are set up you will see a green arrow
 129             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 130             created. Right-click inside the class at any point to load the context menu and select create
 131             a run configuration from context:
 132
 133             |imageCreateRunConfigKt|
 134
 135             **The following window will open:**
 136
 137             |imageRunConfigKt|
 138
 139             **Add the following in the field `VM Options`:**
 140
 141             .. code-block:: bash
 142                :caption: **Custom values for properties**
 143
 144                -Dspring.profiles.active=dev
 145
 146             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 147
 148             .. code-block:: java
 149
 150                -D<application-dev.properties key>=<application-dev.properties value>
 151
 152          .. group-tab:: El Alto
 153
 154             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 155
 156             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 157
 158             After dependencies are imported and indexes are set up you will see a green arrow
 159             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 160             created. Right-click inside the class at any point to load the context menu and select create
 161             a run configuration from context:
 162
 163             |imageCreateRunConfigJava|
 164
 165             **The following window will open:**
 166
 167             |imageRunConfigJava|
 168
 169             **Add the following in the field `VM Options`:**
 170
 171             .. code-block:: bash
 172                :caption: **Custom values for properties**
 173
 174                -Dspring.profiles.active=dev
 175
 176             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 177
 178             .. code-block:: java
 179
 180                -D<application-dev.properties key>=<application-dev.properties value>
 181
 182          .. group-tab:: Dublin
 183
 184             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 185
 186             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 187
 188             After dependencies are imported and indexes are set up you will see a green arrow
 189             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 190             created. Right-click inside the class at any point to load the context menu and select create
 191             a run configuration from context:
 192
 193             |imageCreateRunConfigJava|
 194
 195             **The following window will open:**
 196
 197             |imageRunConfigJava|
 198
 199             **Add the following in the field `VM Options`**
 200
 201             .. code-block:: java
 202                :caption: **Custom values for properties**
 203
 204                -DappName=ControllerBluePrints
 205                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 206                -DappVersion=1.0.0
 207                -Dspring.config.location=opt/app/onap/config/
 208                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 209                -Dspring.datasource.username=sdnctl
 210                -Dspring.datasource.password=sdnctl
 211                -Dcontrollerblueprints.loadInitialData=true
 212                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 213                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 214                -Dblueprintsprocessor.db.primary.username=sdnctl
 215                -Dblueprintsprocessor.db.primary.password=sdnctl
 216                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 217                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 218                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 219                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 220                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 221                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 222                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 223                -Dblueprintsprocessor.restconfEnabled=true
 224                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 225                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 226                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 227                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 228                -Dblueprintsprocessor.grpcEnable=false
 229                -Dblueprintsprocessor.grpcPort=9111
 230                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 231                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 232                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 233                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 234                -Dsecurity.user.name=ccsdkapps
 235                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 236                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 237                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 238                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 239                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 240                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 241                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 242                -Dspring.profiles.active=dev
 243                -Dblueprintsprocessor.httpPort=8080
 244                -Dserver.port=55555
 245
 246
 247       **In the field 'Working Directory' browse to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 248       **if path is not already specified correctly.**
 249
 250       Run configuration should now look something like this:
 251
 252       |imageRunConfigSetUp|
 253
 254       **Add/replace the following in Blueprint's application-dev.properties file.**
 255
 256       .. code-block:: java
 257
 258          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 259          blueprintsprocessor.grpcclient.remote-python.host=localhost
 260          blueprintsprocessor.grpcclient.remote-python.port=50051
 261          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 262
 263          blueprintprocessor.remoteScriptCommand.enabled=true
 264
 265       Take care that if a parameter already exist you need to change the value of the existing parameter to avoid duplicates.
 266
 267
 268       **Run the application:**
 269
 270       Before running Blueprint Processor check that you use the correct Java version in IntelliJ.
 271       Select either run or debug for the created Run Configuration to start the Blueprints Processor:
 272
 273       |imageRunDebug|
 274
 275       |imageBuildLogs|
 276
 277    .. tab:: Visual Studio Code
 278
 279       .. tabs::
 280
 281          .. group-tab:: Frankfurt - Latest
 282
 283             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 284             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 285             * **Step #3** - On the top menu click *Run | Open Configurations*
 286
 287             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 288                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 289
 290             * **Step #4** - add configuration shown below to your configurations list.
 291
 292             .. code-block:: json
 293
 294                {
 295                  "type": "kotlin",
 296                  "request": "launch",
 297                  "name": "Blueprint Processor",
 298                  "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 299                  "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 300                }
 301
 302             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 303
 304             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 305                If you have a cleaner idea how to solve this - please let us know.
 306
 307             **Add/replace the following in Blueprint's application-dev.properties file:**
 308
 309             .. code-block:: java
 310
 311                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 312                blueprintsprocessor.grpcclient.remote-python.host=localhost
 313                blueprintsprocessor.grpcclient.remote-python.port=50051
 314                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 315
 316                blueprintprocessor.remoteScriptCommand.enabled=true
 317
 318             **Currently the following entries need to be added in VSC too:**
 319
 320             .. code-block:: java
 321
 322                logging.level.web=DEBUG
 323                logging.level.org.springframework.web: DEBUG
 324
 325                #Encrypted username and password for health check service
 326                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 327                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 328
 329                #BaseUrls for health check blueprint processor services
 330                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 331                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 332
 333                #BaseUrls for health check Cds Listener services
 334                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 335                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 336
 337                #Actuator properties
 338                management.endpoints.web.exposure.include=*
 339                management.endpoint.health.show-details=always
 340                management.info.git.mode=full
 341
 342             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 343
 344             .. code-block:: bash
 345
 346                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 347
 348             Click Run in Menu bar.
 349
 350             |imageLogsVSC|
 351
 352
 353 Testing the application
 354 ~~~~~~~~~~~~~~~~~~~~~~~
 355
 356 There are two main features of the Blueprints Processor that can be of interest of a developer:
 357 blueprint publish and blueprint process.
 358
 359 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 360
 361 To process, the endpoint is ``api/v1/execution-service/process``.
 362
 363 Postman is a software that can be used to send these request, and an example of
 364 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 365
 366 A detailed description of the usage of different APIs of CDS will follow.
 367
 368
 369 Possible Fixes
 370 ~~~~~~~~~~~~~~
 371
 372 Imported packages or annotiations are not found, Run Config not available?
 373 *****************************************************************************
 374
 375 1. Rebuild with ``maven install ...`` (see above)
 376 2. Potentially change Maven home directory in Settings
 377 3. Maven reimport in IDE
 378
 379 Compilation error?
 380 *******************
 381 * Change Java Version to 11
 382
 383
 384 .. image alignment inside tabs doesn't work
 385
 386 .. |imageRunConfigJava| image:: media/run_config_java.png
 387    :width: 500pt
 388    :align: middle
 389
 390 .. |imageRunConfigKt| image:: media/run_config_kt.png
 391    :width: 500pt
 392    :align: middle
 393
 394 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 395    :width: 500pt
 396    :align: middle
 397
 398 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 399    :width: 500pt
 400    :align: middle
 401
 402 .. |imageImportProject| image:: media/import_project.png
 403    :width: 300pt
 404    :align: middle
 405
 406 .. |imageReimportMaven| image:: media/reimport_maven.png
 407    :width: 400pt
 408    :align: middle
 409
 410 .. |imageRunDebug| image:: media/run_debug.png
 411    :width: 500pt
 412    :align: middle
 413
 414 .. |imageBuildLogs| image:: media/build_logs.png
 415    :width: 500pt
 416    :align: middle
 417
 418 .. |imageLogsVSC| image:: media/vsc_logs.png
 419    :width: 500pt
 420    :align: middle
 421
 422 .. |imageRunConfigSetUp| image:: media/run-config-set-up.png
 423    :width: 500pt
 424    :align: middle