docs/userguides/developer-guide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Run the blueprint processor locally in an IDE, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15 Check out the code
  16 ~~~~~~~~~~~~~~~~~~~
  17
  18 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  19
  20 Build it locally
  21 ~~~~~~~~~~~~~~~~~~
  22
  23 In the checked out directory, type
  24
  25 .. code-block:: bash
  26
  27    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  28
  29 .. note::
  30    If an error ``invalid flag: --release`` appears when executing the maven install command, you need to upgrade Java version of your local
  31    Maven installation. Use something like ``export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64``.
  32
  33 Wait for the maven install command to finish until you go further.
  34
  35 Spin up a Docker container with the database
  36 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  37
  38 The Blueprints Processor project uses a database to store information about the blueprints
  39 and therefore it needs to be online before attempting to run it.
  40
  41 One way to create the database is by using the :file:`docker-compose.yaml` file.
  42 This database will require a local directory to mount a volume, therefore before running docker-compose create following directory:
  43
  44 .. code-block:: bash
  45
  46    mkdir -p -m 755 /opt/app/cds/mysql/data
  47
  48 Navigate to the docker-compose file in the distribution module:
  49
  50 .. tabs::
  51
  52    .. group-tab:: Frankfurt - Latest
  53
  54       .. code-block:: bash
  55
  56          cd ms/blueprintsprocessor/application/src/main/dc
  57
  58    .. group-tab:: El Alto - Dublin
  59
  60       .. code-block:: bash
  61
  62          ms/blueprintsprocessor/distribution/src/main/dc
  63
  64 And run docker-composer:
  65
  66 .. code-block:: bash
  67
  68     docker-compose up -d db
  69
  70 This should spin up a container of the MariaDB image in the background.
  71 To check if it has worked, this command can be used:
  72
  73 .. code-block:: bash
  74
  75     docker-compose logs -f
  76
  77 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  78
  79 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  80 it can be started again by the command:
  81
  82 .. code-block:: bash
  83
  84    docker start <id of mariadb container>
  85
  86 Set permissions on the local file system
  87 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  88
  89 Blueprints processor uses the local file system for some operations and, therefore,
  90 need some existing and accessible paths to run properly.
  91
  92 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
  93
  94 .. code-block:: bash
  95
  96    mkdir -p -m 755 /opt/app/onap/blueprints/archive
  97    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
  98    mkdir -p -m 755 /opt/app/onap/scripts
  99    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
 100
 101 Import the project into the IDE
 102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 103
 104 .. tabs::
 105
 106    .. tab:: IntelliJ IDEA
 107
 108       .. note::
 109          This is the recommended IDE for running CDS blueprint processor.
 110
 111       Go to *File | Open* and choose the :file:`pom.xml` file of the cds/ms/blueprintprocessor directory:
 112
 113       |imageImportProject|
 114
 115       Import as a project. Sometimes it may be necessary to reimport Maven project, e.g. if some dependencies can't be found:
 116
 117       |imageReimportMaven|
 118
 119       **Override some application properties:**
 120
 121       Next steps will create a run configuration profile overriding some application properties with custom values,
 122       to reflect the local environment characteristics.
 123
 124       .. tabs::
 125
 126          .. group-tab:: Frankfurt - Latest
 127
 128             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 129
 130             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 131
 132             After dependencies are imported and indexes are set up you will see a green arrow
 133             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 134             created. Right-click inside the class at any point to load the context menu and select create
 135             a run configuration from context:
 136
 137             |imageCreateRunConfigKt|
 138
 139             **The following window will open:**
 140
 141             |imageRunConfigKt|
 142
 143             **Add the following in the field `VM Options`:**
 144
 145             .. code-block:: bash
 146                :caption: **Custom values for properties**
 147
 148                -Dspring.profiles.active=dev
 149
 150             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 151
 152             .. code-block:: java
 153
 154                -D<application-dev.properties key>=<application-dev.properties value>
 155
 156          .. group-tab:: El Alto
 157
 158             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 159
 160             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 161
 162             After dependencies are imported and indexes are set up you will see a green arrow
 163             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 164             created. Right-click inside the class at any point to load the context menu and select create
 165             a run configuration from context:
 166
 167             |imageCreateRunConfigJava|
 168
 169             **The following window will open:**
 170
 171             |imageRunConfigJava|
 172
 173             **Add the following in the field `VM Options`:**
 174
 175             .. code-block:: bash
 176                :caption: **Custom values for properties**
 177
 178                -Dspring.profiles.active=dev
 179
 180             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 181
 182             .. code-block:: java
 183
 184                -D<application-dev.properties key>=<application-dev.properties value>
 185
 186          .. group-tab:: Dublin
 187
 188             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 189
 190             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 191
 192             After dependencies are imported and indexes are set up you will see a green arrow
 193             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 194             created. Right-click inside the class at any point to load the context menu and select create
 195             a run configuration from context:
 196
 197             |imageCreateRunConfigJava|
 198
 199             **The following window will open:**
 200
 201             |imageRunConfigJava|
 202
 203             **Add the following in the field `VM Options`**
 204
 205             .. code-block:: java
 206                :caption: **Custom values for properties**
 207
 208                -DappName=ControllerBluePrints
 209                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 210                -DappVersion=1.0.0
 211                -Dspring.config.location=opt/app/onap/config/
 212                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 213                -Dspring.datasource.username=sdnctl
 214                -Dspring.datasource.password=sdnctl
 215                -Dcontrollerblueprints.loadInitialData=true
 216                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 217                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 218                -Dblueprintsprocessor.db.primary.username=sdnctl
 219                -Dblueprintsprocessor.db.primary.password=sdnctl
 220                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 221                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 222                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 223                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 224                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 225                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 226                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 227                -Dblueprintsprocessor.restconfEnabled=true
 228                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 229                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 230                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 231                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 232                -Dblueprintsprocessor.grpcEnable=false
 233                -Dblueprintsprocessor.grpcPort=9111
 234                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 235                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 236                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 237                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 238                -Dsecurity.user.name=ccsdkapps
 239                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 240                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 241                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 242                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 243                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 244                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 245                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 246                -Dspring.profiles.active=dev
 247                -Dblueprintsprocessor.httpPort=8080
 248                -Dserver.port=55555
 249
 250
 251       **In the field 'Working Directory' browse to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 252       **if path is not already specified correctly.**
 253
 254       Run configuration should now look something like this:
 255
 256       |imageRunConfigSetUp|
 257
 258       **Add/replace the following in Blueprint's application-dev.properties file.**
 259
 260       .. code-block:: java
 261
 262          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 263          blueprintsprocessor.grpcclient.remote-python.host=localhost
 264          blueprintsprocessor.grpcclient.remote-python.port=50051
 265          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 266
 267          blueprintprocessor.remoteScriptCommand.enabled=true
 268
 269       Take care that if a parameter already exist you need to change the value of the existing parameter to avoid duplicates.
 270
 271
 272       **Run the application:**
 273
 274       Before running Blueprint Processor check that you use the correct Java version in IntelliJ.
 275       Select either run or debug for the created Run Configuration to start the Blueprints Processor:
 276
 277       |imageRunDebug|
 278
 279       |imageBuildLogs|
 280
 281    .. tab:: Visual Studio Code
 282
 283       .. tabs::
 284
 285          .. group-tab:: Frankfurt - Latest
 286
 287             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 288             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 289             * **Step #3** - On the top menu click *Run | Open Configurations*
 290
 291             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 292                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 293
 294             * **Step #4** - add configuration shown below to your configurations list.
 295
 296             .. code-block:: json
 297
 298                {
 299                  "type": "kotlin",
 300                  "request": "launch",
 301                  "name": "Blueprint Processor",
 302                  "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 303                  "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 304                }
 305
 306             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 307
 308             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 309                If you have a cleaner idea how to solve this - please let us know.
 310
 311             **Add/replace the following in Blueprint's application-dev.properties file:**
 312
 313             .. code-block:: java
 314
 315                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 316                blueprintsprocessor.grpcclient.remote-python.host=localhost
 317                blueprintsprocessor.grpcclient.remote-python.port=50051
 318                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 319
 320                blueprintprocessor.remoteScriptCommand.enabled=true
 321
 322             **Currently the following entries need to be added in VSC too:**
 323
 324             .. code-block:: java
 325
 326                logging.level.web=DEBUG
 327                logging.level.org.springframework.web: DEBUG
 328
 329                #Encrypted username and password for health check service
 330                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 331                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 332
 333                #BaseUrls for health check blueprint processor services
 334                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 335                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 336
 337                #BaseUrls for health check Cds Listener services
 338                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 339                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 340
 341                #Actuator properties
 342                management.endpoints.web.exposure.include=*
 343                management.endpoint.health.show-details=always
 344                management.info.git.mode=full
 345
 346             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 347
 348             .. code-block:: bash
 349
 350                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 351
 352             Click Run in Menu bar.
 353
 354             |imageLogsVSC|
 355
 356
 357 Testing the application
 358 ~~~~~~~~~~~~~~~~~~~~~~~
 359
 360 There are two main features of the Blueprints Processor that can be of interest of a developer:
 361 blueprint publish and blueprint process.
 362
 363 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 364
 365 To process, the endpoint is ``api/v1/execution-service/process``.
 366
 367 Postman is a software that can be used to send these request, and an example of
 368 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 369
 370 A detailed description of the usage of different APIs of CDS will follow.
 371
 372
 373 Possible Fixes
 374 ~~~~~~~~~~~~~~
 375
 376 Imported packages or annotiations are not found, Run Config not available?
 377 *****************************************************************************
 378
 379 1. Rebuild with ``maven install ...`` (see above)
 380 2. Potentially change Maven home directory in Settings
 381 3. Maven reimport in IDE
 382
 383 Compilation error?
 384 *******************
 385 * Change Java Version to 11
 386
 387
 388 .. image alignment inside tabs doesn't work
 389
 390 .. |imageRunConfigJava| image:: media/run_config_java.png
 391    :width: 500pt
 392    :align: middle
 393
 394 .. |imageRunConfigKt| image:: media/run_config_kt.png
 395    :width: 500pt
 396    :align: middle
 397
 398 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 399    :width: 500pt
 400    :align: middle
 401
 402 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 403    :width: 500pt
 404    :align: middle
 405
 406 .. |imageImportProject| image:: media/import_project.png
 407    :width: 300pt
 408    :align: middle
 409
 410 .. |imageReimportMaven| image:: media/reimport_maven.png
 411    :width: 400pt
 412    :align: middle
 413
 414 .. |imageRunDebug| image:: media/run_debug.png
 415    :width: 500pt
 416    :align: middle
 417
 418 .. |imageBuildLogs| image:: media/build_logs.png
 419    :width: 500pt
 420    :align: middle
 421
 422 .. |imageLogsVSC| image:: media/vsc_logs.png
 423    :width: 500pt
 424    :align: middle
 425
 426 .. |imageRunConfigSetUp| image:: media/run_config_set_up.png
 427    :width: 500pt
 428    :align: middle