docs/userguide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Have the blueprint processor running locally is to use the IDE to run the code, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15 Check out the code
  16 ~~~~~~~~~~~~~~~~~~~
  17
  18 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  19
  20 Build it locally
  21 ~~~~~~~~~~~~~~~~~~
  22
  23 In the checked out directory, type
  24
  25 .. code-block:: bash
  26
  27    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  28
  29 Spin up a Docker container with the database
  30 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  31
  32 The Blueprints Processor project uses a database to store information about the blueprints
  33 and therefore it needs to be online before attempting to run it.
  34
  35 One way to create the database is by using the :file:`docker-compose.yaml`
  36 file present on the distribution module. This database will require a local directory to mount a volume, before running docker-compose remember to create following directory:
  37
  38 .. code-block:: bash
  39
  40    mkdir -p -m 755 /opt/app/cds/mysql/data
  41
  42 Navigate to the docker-compose file in the distribution module:
  43
  44 .. code-block:: bash
  45
  46    ms/blueprintsprocessor/distribution/src/main/dc
  47
  48 And run docker-composer:
  49
  50 .. code-block:: bash
  51
  52     docker-compose up -d db
  53
  54 This should spin up a container of the MariaDB image in the background.
  55 To check if it has worked, this command can be used:
  56
  57 .. code-block:: bash
  58
  59     docker-compose logs -f
  60
  61 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  62
  63 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  64 it can be started again by the command:
  65
  66 .. code-block:: bash
  67
  68    docker start <id of mariadb container>
  69
  70 Set permissions on the local file system
  71 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  72
  73 Blueprints processor uses the local file system for some operations and, therefore,
  74 need some existing and accessible paths to run properly.
  75
  76 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
  77
  78 .. code-block:: bash
  79
  80    mkdir -p -m 755 /opt/app/onap/blueprints/archive
  81    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
  82    mkdir -p -m 755 /opt/app/onap/scripts
  83    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
  84
  85 Import the project into the IDE
  86 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  87
  88 .. tabs::
  89
  90    .. tab:: IntelliJ IDEA
  91
  92       Go to *File | Open* and choose the :file:`pom.xml` file of the cds directory:
  93
  94       |imageImportProject|
  95
  96       Sometimes it may be necessary to reimport Maven project:
  97
  98       |imageReimportMaven|
  99
 100       **Override some application properties:**
 101
 102       After the project is compiled, a Run Configuration profile overriding some application properties
 103       with custom values needs to be created, to reflect the local environment characteristics.
 104
 105       .. tabs::
 106
 107          .. group-tab:: Latest
 108
 109             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 110
 111             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 112
 113             Right-click inside it, at any point, to load the context menu and select create
 114             BlueprintProcessorApplication configuration from context:
 115
 116             |imageCreateRunConfigkt|
 117
 118             **The following window will open:**
 119
 120             |imageRunConfigKt|
 121
 122             **Add the following in the field `VM Options`:**
 123
 124             .. code-block:: bash
 125                :caption: **Custom values for properties**
 126
 127                -Dspring.profiles.active=dev
 128
 129             You can override any value from **application-dev.properties** file here. Use the following pattern:
 130
 131             .. code-block:: java
 132
 133                -D<application-dev.properties key>=<application-dev.properties value>
 134
 135          .. group-tab:: Frankfurt
 136
 137             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 138
 139             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 140
 141             Right-click inside it, at any point, to load the context menu and select create
 142             BlueprintProcessorApplication configuration from context:
 143
 144             |imageCreateRunConfigkt|
 145
 146             **The following window will open:**
 147
 148             |imageRunConfigKt|
 149
 150             **Add the following in the field `VM Options`:**
 151
 152             .. code-block:: bash
 153                :caption: **Custom values for properties**
 154
 155                -Dspring.profiles.active=dev
 156
 157             You can override any value from **application-dev.properties** file here. Use the following pattern:
 158
 159             .. code-block:: java
 160
 161                -D<application-dev.properties key>=<application-dev.properties value>
 162
 163          .. group-tab:: El Alto
 164
 165             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 166
 167             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 168
 169             Right-click inside it, at any point, to load the context menu and select create
 170             BlueprintProcessorApplication configuration from context:
 171
 172             |imageCreateRunConfigJava|
 173
 174             **The following window will open:**
 175
 176             |imageRunConfigJava|
 177
 178             **Add the following in the field `VM Options`:**
 179
 180             .. code-block:: bash
 181                :caption: **Custom values for properties**
 182
 183                -Dspring.profiles.active=dev
 184
 185             You can override any value from **application-dev.properties** file here. Use the following pattern:
 186
 187             .. code-block:: java
 188
 189                -D<application-dev.properties key>=<application-dev.properties value>
 190
 191          .. group-tab:: Dublin
 192
 193             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 194
 195             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 196
 197             Right-click inside it, at any point, to load the context menu and select create
 198             BlueprintProcessorApplication configuration from context:
 199
 200             |imageCreateRunConfigJava|
 201
 202             **The following window will open:**
 203
 204             |imageRunConfigJava|
 205
 206             **Add the following in that field:**
 207
 208             .. code-block:: java
 209                :caption: **Custom values for properties**
 210
 211                -DappName=ControllerBluePrints
 212                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 213                -DappVersion=1.0.0
 214                -Dspring.config.location=opt/app/onap/config/
 215                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 216                -Dspring.datasource.username=sdnctl
 217                -Dspring.datasource.password=sdnctl
 218                -Dcontrollerblueprints.loadInitialData=true
 219                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 220                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 221                -Dblueprintsprocessor.db.primary.username=sdnctl
 222                -Dblueprintsprocessor.db.primary.password=sdnctl
 223                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 224                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 225                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 226                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 227                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 228                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 229                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 230                -Dblueprintsprocessor.restconfEnabled=true
 231                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 232                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 233                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 234                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 235                -Dblueprintsprocessor.grpcEnable=false
 236                -Dblueprintsprocessor.grpcPort=9111
 237                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 238                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 239                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 240                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 241                -Dsecurity.user.name=ccsdkapps
 242                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 243                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 244                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 245                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 246                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 247                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 248                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 249                -Dspring.profiles.active=dev
 250                -Dblueprintsprocessor.httpPort=8080
 251                -Dserver.port=55555
 252
 253
 254       **Browse Working Directory to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 255       **if path is not already specified correctly.**
 256
 257       **Add/replace the following in Blueprint's application-dev.properties file:**
 258
 259       .. code-block:: java
 260
 261          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 262          blueprintsprocessor.grpcclient.remote-python.host=localhost
 263          blueprintsprocessor.grpcclient.remote-python.port=50051
 264          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 265
 266          blueprintprocessor.remoteScriptCommand.enabled=true
 267
 268
 269       **Run the application:**
 270
 271       Select either run or debug for this Run Configuration to start the Blueprints Processor:
 272
 273       |imageRunDebug|
 274
 275       |imageBuildLogs|
 276
 277    .. tab:: Visual Studio Code
 278
 279       .. tabs::
 280
 281          .. group-tab:: Latest
 282
 283             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 284             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 285             * **Step #3** - On the top menu click *Run | Open Configurations*
 286
 287             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 288                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 289
 290             * **Step #4** - add configuration shown below to your configurations list.
 291
 292             .. code-block:: json
 293
 294                {
 295                  "type": "kotlin",
 296                  "request": "launch",
 297                  "name": "Blueprint Processor",
 298                  "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 299                  "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 300                }
 301
 302             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 303
 304             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 305                If you have a cleaner idea how to solve this - please let us know.
 306
 307             **Add/replace the following in Blueprint's application-dev.properties file:**
 308
 309             .. code-block:: java
 310
 311                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 312                blueprintsprocessor.grpcclient.remote-python.host=localhost
 313                blueprintsprocessor.grpcclient.remote-python.port=50051
 314                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 315
 316                blueprintprocessor.remoteScriptCommand.enabled=true
 317
 318             **Currently the following entries need to be added in VSC too:**
 319
 320             .. code-block:: java
 321
 322                logging.level.web=DEBUG
 323                logging.level.org.springframework.web: DEBUG
 324
 325                #Encrypted username and password for health check service
 326                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 327                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 328
 329                #BaseUrls for health check blueprint processor services
 330                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 331                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 332
 333                #BaseUrls for health check Cds Listener services
 334                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 335                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 336
 337                #Actuator properties
 338                management.endpoints.web.exposure.include=*
 339                management.endpoint.health.show-details=always
 340                management.info.git.mode=full
 341
 342             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 343
 344             .. code-block:: bash
 345
 346                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 347
 348             Click Run in Menu bar.
 349
 350             |imageLogsVSC|
 351
 352
 353 Testing the application
 354 ~~~~~~~~~~~~~~~~~~~~~~~
 355
 356 There are two main features of the Blueprints Processor that can be of interest of a developer:
 357 blueprint publish and blueprint process.
 358
 359 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 360
 361 To process, the endpoint is ``api/v1/execution-service/process``.
 362
 363 Postman is a software that can be used to send these request, and an example of
 364 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 365
 366 A detailed description of the usage of different APIs of CDS will follow.
 367
 368
 369 Possible Fixes
 370 ~~~~~~~~~~~~~~
 371
 372 Imported packages or annotiations are not found, Run Config not available?
 373 *****************************************************************************
 374
 375 1. Rebuild with ``maven install ...`` (see above)
 376 2. Potentially change Maven home directory in Settings
 377 3. Maven reimport in IDE
 378
 379 Compilation error?
 380 *******************
 381
 382 * Change Java Version to 11
 383
 384
 385 .. image alignment inside tabs doesn't work
 386
 387 .. |imageRunConfigJava| image:: media/run_config_java.png
 388    :width: 500pt
 389    :align: middle
 390
 391 .. |imageRunConfigKt| image:: media/run_config_kt.png
 392    :width: 500pt
 393    :align: middle
 394
 395 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 396    :width: 500pt
 397    :align: middle
 398
 399 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 400    :width: 500pt
 401    :align: middle
 402
 403 .. |imageImportProject| image:: media/import_project.png
 404    :width: 300pt
 405    :align: middle
 406
 407 .. |imageReimportMaven| image:: media/reimport_maven.png
 408    :width: 400pt
 409    :align: middle
 410
 411 .. |imageRunDebug| image:: media/run_debug.png
 412    :width: 500pt
 413    :align: middle
 414
 415 .. |imageBuildLogs| image:: media/build_logs.png
 416    :width: 500pt
 417    :align: middle
 418
 419 .. |imageLogsVSC| image:: media/vsc_logs.png
 420    :width: 500pt
 421    :align: middle