docs/userguide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Have the blueprint processor running locally is to use the IDE to run the code, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15 Check out the code
  16 ~~~~~~~~~~~~~~~~~~~
  17
  18 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  19
  20 Build it locally
  21 ~~~~~~~~~~~~~~~~~~
  22
  23 In the checked out directory, type
  24
  25 .. code-block:: bash
  26
  27    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  28
  29 Spin up a Docker container with the database
  30 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  31
  32 The Blueprints Processor project uses a database to store information about the blueprints
  33 and therefore it needs to be online before attempting to run it.
  34
  35 One way to create the database is by using the :file:`docker-compose.yaml` file.
  36 This database will require a local directory to mount a volume, before running docker-compose remember to create following directory:
  37
  38 .. code-block:: bash
  39
  40    mkdir -p -m 755 /opt/app/cds/mysql/data
  41
  42 Navigate to the docker-compose file in the distribution module:
  43
  44 .. tabs::
  45
  46    .. group-tab:: Frankfurt - Latest
  47
  48       .. code-block:: bash
  49
  50          cd ms/blueprintsprocessor/application/src/main/dc
  51
  52    .. group-tab:: El Alto - Dublin
  53
  54       .. code-block:: bash
  55
  56          ms/blueprintsprocessor/distribution/src/main/dc
  57
  58 And run docker-composer:
  59
  60 .. code-block:: bash
  61
  62     docker-compose up -d db
  63
  64 This should spin up a container of the MariaDB image in the background.
  65 To check if it has worked, this command can be used:
  66
  67 .. code-block:: bash
  68
  69     docker-compose logs -f
  70
  71 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  72
  73 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  74 it can be started again by the command:
  75
  76 .. code-block:: bash
  77
  78    docker start <id of mariadb container>
  79
  80 Set permissions on the local file system
  81 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  82
  83 Blueprints processor uses the local file system for some operations and, therefore,
  84 need some existing and accessible paths to run properly.
  85
  86 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
  87
  88 .. code-block:: bash
  89
  90    mkdir -p -m 755 /opt/app/onap/blueprints/archive
  91    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
  92    mkdir -p -m 755 /opt/app/onap/scripts
  93    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
  94
  95 Import the project into the IDE
  96 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  97
  98 .. tabs::
  99
 100    .. tab:: IntelliJ IDEA
 101
 102       Go to *File | Open* and choose the :file:`pom.xml` file of the cds directory:
 103
 104       |imageImportProject|
 105
 106       Sometimes it may be necessary to reimport Maven project:
 107
 108       |imageReimportMaven|
 109
 110       **Override some application properties:**
 111
 112       After the project is compiled, a Run Configuration profile overriding some application properties
 113       with custom values needs to be created, to reflect the local environment characteristics.
 114
 115       .. tabs::
 116
 117          .. group-tab:: Frankfurt - Latest
 118
 119             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 120
 121             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 122
 123             Right-click inside it, at any point, to load the context menu and select create
 124             BlueprintProcessorApplication configuration from context:
 125
 126             |imageCreateRunConfigKt|
 127
 128             **The following window will open:**
 129
 130             |imageRunConfigKt|
 131
 132             **Add the following in the field `VM Options`:**
 133
 134             .. code-block:: bash
 135                :caption: **Custom values for properties**
 136
 137                -Dspring.profiles.active=dev
 138
 139             You can override any value from **application-dev.properties** file here. Use the following pattern:
 140
 141             .. code-block:: java
 142
 143                -D<application-dev.properties key>=<application-dev.properties value>
 144
 145          .. group-tab:: El Alto
 146
 147             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 148
 149             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 150
 151             Right-click inside it, at any point, to load the context menu and select create
 152             BlueprintProcessorApplication configuration from context:
 153
 154             |imageCreateRunConfigJava|
 155
 156             **The following window will open:**
 157
 158             |imageRunConfigJava|
 159
 160             **Add the following in the field `VM Options`:**
 161
 162             .. code-block:: bash
 163                :caption: **Custom values for properties**
 164
 165                -Dspring.profiles.active=dev
 166
 167             You can override any value from **application-dev.properties** file here. Use the following pattern:
 168
 169             .. code-block:: java
 170
 171                -D<application-dev.properties key>=<application-dev.properties value>
 172
 173          .. group-tab:: Dublin
 174
 175             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 176
 177             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 178
 179             Right-click inside it, at any point, to load the context menu and select create
 180             BlueprintProcessorApplication configuration from context:
 181
 182             |imageCreateRunConfigJava|
 183
 184             **The following window will open:**
 185
 186             |imageRunConfigJava|
 187
 188             **Add the following in that field:**
 189
 190             .. code-block:: java
 191                :caption: **Custom values for properties**
 192
 193                -DappName=ControllerBluePrints
 194                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 195                -DappVersion=1.0.0
 196                -Dspring.config.location=opt/app/onap/config/
 197                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 198                -Dspring.datasource.username=sdnctl
 199                -Dspring.datasource.password=sdnctl
 200                -Dcontrollerblueprints.loadInitialData=true
 201                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 202                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 203                -Dblueprintsprocessor.db.primary.username=sdnctl
 204                -Dblueprintsprocessor.db.primary.password=sdnctl
 205                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 206                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 207                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 208                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 209                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 210                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 211                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 212                -Dblueprintsprocessor.restconfEnabled=true
 213                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 214                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 215                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 216                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 217                -Dblueprintsprocessor.grpcEnable=false
 218                -Dblueprintsprocessor.grpcPort=9111
 219                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 220                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 221                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 222                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 223                -Dsecurity.user.name=ccsdkapps
 224                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 225                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 226                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 227                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 228                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 229                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 230                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 231                -Dspring.profiles.active=dev
 232                -Dblueprintsprocessor.httpPort=8080
 233                -Dserver.port=55555
 234
 235
 236       **Browse Working Directory to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 237       **if path is not already specified correctly.**
 238
 239       **Add/replace the following in Blueprint's application-dev.properties file:**
 240
 241       .. code-block:: java
 242
 243          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 244          blueprintsprocessor.grpcclient.remote-python.host=localhost
 245          blueprintsprocessor.grpcclient.remote-python.port=50051
 246          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 247
 248          blueprintprocessor.remoteScriptCommand.enabled=true
 249
 250
 251       **Run the application:**
 252
 253       Select either run or debug for this Run Configuration to start the Blueprints Processor:
 254
 255       |imageRunDebug|
 256
 257       |imageBuildLogs|
 258
 259    .. tab:: Visual Studio Code
 260
 261       .. tabs::
 262
 263          .. group-tab:: Frankfurt - Latest
 264
 265             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 266             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 267             * **Step #3** - On the top menu click *Run | Open Configurations*
 268
 269             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 270                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 271
 272             * **Step #4** - add configuration shown below to your configurations list.
 273
 274             .. code-block:: json
 275
 276                {
 277                  "type": "kotlin",
 278                  "request": "launch",
 279                  "name": "Blueprint Processor",
 280                  "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 281                  "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 282                }
 283
 284             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 285
 286             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 287                If you have a cleaner idea how to solve this - please let us know.
 288
 289             **Add/replace the following in Blueprint's application-dev.properties file:**
 290
 291             .. code-block:: java
 292
 293                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 294                blueprintsprocessor.grpcclient.remote-python.host=localhost
 295                blueprintsprocessor.grpcclient.remote-python.port=50051
 296                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 297
 298                blueprintprocessor.remoteScriptCommand.enabled=true
 299
 300             **Currently the following entries need to be added in VSC too:**
 301
 302             .. code-block:: java
 303
 304                logging.level.web=DEBUG
 305                logging.level.org.springframework.web: DEBUG
 306
 307                #Encrypted username and password for health check service
 308                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 309                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 310
 311                #BaseUrls for health check blueprint processor services
 312                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 313                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 314
 315                #BaseUrls for health check Cds Listener services
 316                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 317                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 318
 319                #Actuator properties
 320                management.endpoints.web.exposure.include=*
 321                management.endpoint.health.show-details=always
 322                management.info.git.mode=full
 323
 324             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 325
 326             .. code-block:: bash
 327
 328                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 329
 330             Click Run in Menu bar.
 331
 332             |imageLogsVSC|
 333
 334
 335 Testing the application
 336 ~~~~~~~~~~~~~~~~~~~~~~~
 337
 338 There are two main features of the Blueprints Processor that can be of interest of a developer:
 339 blueprint publish and blueprint process.
 340
 341 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 342
 343 To process, the endpoint is ``api/v1/execution-service/process``.
 344
 345 Postman is a software that can be used to send these request, and an example of
 346 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 347
 348 A detailed description of the usage of different APIs of CDS will follow.
 349
 350
 351 Possible Fixes
 352 ~~~~~~~~~~~~~~
 353
 354 Imported packages or annotiations are not found, Run Config not available?
 355 *****************************************************************************
 356
 357 1. Rebuild with ``maven install ...`` (see above)
 358 2. Potentially change Maven home directory in Settings
 359 3. Maven reimport in IDE
 360
 361 Compilation error?
 362 *******************
 363
 364 * Change Java Version to 11
 365
 366
 367 .. image alignment inside tabs doesn't work
 368
 369 .. |imageRunConfigJava| image:: media/run_config_java.png
 370    :width: 500pt
 371    :align: middle
 372
 373 .. |imageRunConfigKt| image:: media/run_config_kt.png
 374    :width: 500pt
 375    :align: middle
 376
 377 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 378    :width: 500pt
 379    :align: middle
 380
 381 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 382    :width: 500pt
 383    :align: middle
 384
 385 .. |imageImportProject| image:: media/import_project.png
 386    :width: 300pt
 387    :align: middle
 388
 389 .. |imageReimportMaven| image:: media/reimport_maven.png
 390    :width: 400pt
 391    :align: middle
 392
 393 .. |imageRunDebug| image:: media/run_debug.png
 394    :width: 500pt
 395    :align: middle
 396
 397 .. |imageBuildLogs| image:: media/build_logs.png
 398    :width: 500pt
 399    :align: middle
 400
 401 .. |imageLogsVSC| image:: media/vsc_logs.png
 402    :width: 500pt
 403    :align: middle