docs/userguides/developer-guide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 .. _running_bp_processor_in_ide:
   7
   8 Running Blueprints Processor Microservice in an IDE
   9 ====================================================
  10
  11 Objective
  12 ~~~~~~~~~~~~
  13
  14 Run the blueprint processor locally in an IDE, while having the database running in a container.
  15 This way, code changes can be conveniently tested and debugged.
  16
  17 Check out the code
  18 ~~~~~~~~~~~~~~~~~~~
  19
  20 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  21
  22 Build it locally
  23 ~~~~~~~~~~~~~~~~~~
  24
  25 In the checked out directory, type
  26
  27 .. code-block:: bash
  28
  29    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  30
  31 .. note::
  32    If an error ``invalid flag: --release`` appears when executing the maven install command, you need to upgrade Java version of your local
  33    Maven installation. Use something like ``export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64``.
  34
  35 Wait for the maven install command to finish until you go further.
  36
  37 Spin up a Docker container with the database
  38 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  39
  40 The Blueprints Processor project uses a database to store information about the blueprints
  41 and therefore it needs to be online before attempting to run it.
  42
  43 One way to create the database is by using the :file:`docker-compose.yaml` file.
  44 This database will require a local directory to mount a volume, therefore before running docker-compose create following directory:
  45
  46 .. code-block:: bash
  47
  48    mkdir -p -m 755 /opt/app/cds/mysql/data
  49
  50 Navigate to the docker-compose file in the distribution module:
  51
  52 .. tabs::
  53
  54    .. group-tab:: Frankfurt - Latest
  55
  56       .. code-block:: bash
  57
  58          cd ms/blueprintsprocessor/application/src/main/dc
  59
  60    .. group-tab:: El Alto - Dublin
  61
  62       .. code-block:: bash
  63
  64          ms/blueprintsprocessor/distribution/src/main/dc
  65
  66 And run docker-composer:
  67
  68 .. code-block:: bash
  69
  70     docker-compose up -d db
  71
  72 This should spin up a container of the MariaDB image in the background.
  73 To check if it has worked, this command can be used:
  74
  75 .. code-block:: bash
  76
  77     docker-compose logs -f
  78
  79 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  80
  81 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  82 it can be started again by the command:
  83
  84 .. code-block:: bash
  85
  86    docker start <id of mariadb container>
  87
  88 Set permissions on the local file system
  89 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  90
  91 Blueprints processor uses the local file system for some operations and, therefore,
  92 need some existing and accessible paths to run properly.
  93
  94 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
  95
  96 .. code-block:: bash
  97
  98    mkdir -p -m 755 /opt/app/onap/blueprints/archive
  99    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
 100    mkdir -p -m 755 /opt/app/onap/scripts
 101    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
 102
 103 Import the project into the IDE
 104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 105
 106 .. tabs::
 107
 108    .. tab:: IntelliJ IDEA
 109
 110       .. note::
 111          This is the recommended IDE for running CDS blueprint processor.
 112
 113       Go to *File | Open* and choose the :file:`pom.xml` file of the cds/ms/blueprintprocessor directory:
 114
 115       |imageImportProject|
 116
 117       Import as a project. Sometimes it may be necessary to reimport Maven project, e.g. if some dependencies can't be found:
 118
 119       |imageReimportMaven|
 120
 121       **Override some application properties:**
 122
 123       Next steps will create a run configuration profile overriding some application properties with custom values,
 124       to reflect the local environment characteristics.
 125
 126       .. tabs::
 127
 128          .. group-tab:: Frankfurt - Latest
 129
 130             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 131
 132             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 133
 134             After dependencies are imported and indexes are set up you will see a green arrow
 135             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 136             created. Right-click inside the class at any point to load the context menu and select create
 137             a run configuration from context:
 138
 139             |imageCreateRunConfigKt|
 140
 141             **The following window will open:**
 142
 143             |imageRunConfigKt|
 144
 145             **Add the following in the field `VM Options`:**
 146
 147             .. code-block:: bash
 148                :caption: **Custom values for properties**
 149
 150                -Dspring.profiles.active=dev
 151
 152             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 153
 154             .. code-block:: java
 155
 156                -D<application-dev.properties key>=<application-dev.properties value>
 157
 158          .. group-tab:: El Alto
 159
 160             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 161
 162             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 163
 164             After dependencies are imported and indexes are set up you will see a green arrow
 165             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 166             created. Right-click inside the class at any point to load the context menu and select create
 167             a run configuration from context:
 168
 169             |imageCreateRunConfigJava|
 170
 171             **The following window will open:**
 172
 173             |imageRunConfigJava|
 174
 175             **Add the following in the field `VM Options`:**
 176
 177             .. code-block:: bash
 178                :caption: **Custom values for properties**
 179
 180                -Dspring.profiles.active=dev
 181
 182             Optional: You can override any value from **application-dev.properties** file here. In this case use the following pattern:
 183
 184             .. code-block:: java
 185
 186                -D<application-dev.properties key>=<application-dev.properties value>
 187
 188          .. group-tab:: Dublin
 189
 190             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 191
 192             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 193
 194             After dependencies are imported and indexes are set up you will see a green arrow
 195             next to main function of BlueprintProcessorApplication class, indicating that the run configuration can now be
 196             created. Right-click inside the class at any point to load the context menu and select create
 197             a run configuration from context:
 198
 199             |imageCreateRunConfigJava|
 200
 201             **The following window will open:**
 202
 203             |imageRunConfigJava|
 204
 205             **Add the following in the field `VM Options`**
 206
 207             .. code-block:: java
 208                :caption: **Custom values for properties**
 209
 210                -DappName=ControllerBluePrints
 211                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 212                -DappVersion=1.0.0
 213                -Dspring.config.location=opt/app/onap/config/
 214                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 215                -Dspring.datasource.username=sdnctl
 216                -Dspring.datasource.password=sdnctl
 217                -Dcontrollerblueprints.loadInitialData=true
 218                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 219                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 220                -Dblueprintsprocessor.db.primary.username=sdnctl
 221                -Dblueprintsprocessor.db.primary.password=sdnctl
 222                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 223                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 224                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 225                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 226                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 227                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 228                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 229                -Dblueprintsprocessor.restconfEnabled=true
 230                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 231                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 232                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 233                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 234                -Dblueprintsprocessor.grpcEnable=false
 235                -Dblueprintsprocessor.grpcPort=9111
 236                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 237                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 238                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 239                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 240                -Dsecurity.user.name=ccsdkapps
 241                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 242                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 243                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 244                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 245                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 246                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 247                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 248                -Dspring.profiles.active=dev
 249                -Dblueprintsprocessor.httpPort=8080
 250                -Dserver.port=55555
 251
 252
 253       **In the field 'Working Directory' browse to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 254       **if path is not already specified correctly.**
 255
 256       Run configuration should now look something like this:
 257
 258       |imageRunConfigSetUp|
 259
 260       **Add/replace the following in Blueprint's application-dev.properties file.**
 261
 262       .. code-block:: java
 263
 264          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 265          blueprintsprocessor.grpcclient.remote-python.host=localhost
 266          blueprintsprocessor.grpcclient.remote-python.port=50051
 267          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 268
 269          blueprintprocessor.remoteScriptCommand.enabled=true
 270
 271       Take care that if a parameter already exist you need to change the value of the existing parameter to avoid duplicates.
 272
 273
 274       **Run the application:**
 275
 276       Before running Blueprint Processor check that you use the correct Java version in IntelliJ.
 277       Select either run or debug for the created Run Configuration to start the Blueprints Processor:
 278
 279       |imageRunDebug|
 280
 281       |imageBuildLogs|
 282
 283    .. tab:: Visual Studio Code
 284
 285       .. tabs::
 286
 287          .. group-tab:: Frankfurt - Latest
 288
 289             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 290             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 291             * **Step #3** - On the top menu click *Run | Open Configurations*
 292
 293             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 294                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 295
 296             * **Step #4** - add configuration shown below to your configurations list.
 297
 298             .. code-block:: json
 299
 300                {
 301                  "type": "kotlin",
 302                  "request": "launch",
 303                  "name": "Blueprint Processor",
 304                  "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 305                  "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 306                }
 307
 308             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 309
 310             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 311                If you have a cleaner idea how to solve this - please let us know.
 312
 313             **Add/replace the following in Blueprint's application-dev.properties file:**
 314
 315             .. code-block:: java
 316
 317                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 318                blueprintsprocessor.grpcclient.remote-python.host=localhost
 319                blueprintsprocessor.grpcclient.remote-python.port=50051
 320                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 321
 322                blueprintprocessor.remoteScriptCommand.enabled=true
 323
 324             **Currently the following entries need to be added in VSC too:**
 325
 326             .. code-block:: java
 327
 328                logging.level.web=DEBUG
 329                logging.level.org.springframework.web: DEBUG
 330
 331                #Encrypted username and password for health check service
 332                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 333                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 334
 335                #BaseUrls for health check blueprint processor services
 336                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 337                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 338
 339                #BaseUrls for health check Cds Listener services
 340                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 341                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 342
 343                #Actuator properties
 344                management.endpoints.web.exposure.include=*
 345                management.endpoint.health.show-details=always
 346                management.info.git.mode=full
 347
 348             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 349
 350             .. code-block:: bash
 351
 352                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 353
 354             Click Run in Menu bar.
 355
 356             |imageLogsVSC|
 357
 358
 359 Testing the application
 360 ~~~~~~~~~~~~~~~~~~~~~~~
 361
 362 There are two main features of the Blueprints Processor that can be of interest of a developer:
 363 blueprint publish and blueprint process.
 364
 365 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 366
 367 To process, the endpoint is ``api/v1/execution-service/process``.
 368
 369 Postman is a software that can be used to send these request, and an example of
 370 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 371
 372 A detailed description of the usage of different APIs of CDS will follow.
 373
 374
 375 Possible Fixes
 376 ~~~~~~~~~~~~~~
 377
 378 Imported packages or annotiations are not found, Run Config not available?
 379 *****************************************************************************
 380
 381 1. Rebuild with ``maven install ...`` (see above)
 382 2. Potentially change Maven home directory in Settings
 383 3. Maven reimport in IDE
 384
 385 Compilation error?
 386 *******************
 387 * Change Java Version to 11
 388
 389
 390 .. image alignment inside tabs doesn't work
 391
 392 .. |imageRunConfigJava| image:: media/run_config_java.png
 393    :width: 500pt
 394    :align: middle
 395
 396 .. |imageRunConfigKt| image:: media/run_config_kt.png
 397    :width: 500pt
 398    :align: middle
 399
 400 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 401    :width: 500pt
 402    :align: middle
 403
 404 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 405    :width: 500pt
 406    :align: middle
 407
 408 .. |imageImportProject| image:: media/import_project.png
 409    :width: 300pt
 410    :align: middle
 411
 412 .. |imageReimportMaven| image:: media/reimport_maven.png
 413    :width: 400pt
 414    :align: middle
 415
 416 .. |imageRunDebug| image:: media/run_debug.png
 417    :width: 500pt
 418    :align: middle
 419
 420 .. |imageBuildLogs| image:: media/build_logs.png
 421    :width: 500pt
 422    :align: middle
 423
 424 .. |imageLogsVSC| image:: media/vsc_logs.png
 425    :width: 500pt
 426    :align: middle
 427
 428 .. |imageRunConfigSetUp| image:: media/run_config_set_up.png
 429    :width: 500pt
 430    :align: middle