docs/userguide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Have the blueprint processor running locally is to use the IDE to run the code, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15
  16 Check out the code
  17 ~~~~~~~~~~~~~~~~~~~
  18
  19 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  20
  21 Build it locally
  22 ~~~~~~~~~~~~~~~~~~
  23
  24 In the checked out directory, type
  25
  26 .. code-block:: bash
  27
  28    mvn clean install -Pq -Dadditionalparam=-Xdoclint:none
  29
  30 Spin up a Docker container with the database
  31 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  32
  33 The Blueprints Processor project uses a database to store information about the blueprints
  34 and therefore it needs to be online before attempting to run it.
  35
  36 One way to create the database is by using the :file:`docker-compose.yaml`
  37 file present on the distribution module. This database will require a local directory to mount a volume, before running docker-compose remember to create following directory:
  38
  39 .. code-block:: bash
  40
  41    mkdir -p -m 755 /opt/app/cds/mysql/data
  42
  43 Navigate to the docker-compose file in the distribution module:
  44
  45 .. tabs::
  46
  47    .. group-tab:: Latest
  48
  49       .. code-block:: bash
  50
  51          cd ms/blueprintsprocessor/application/src/main/dc
  52
  53    .. group-tab:: Frankfurt
  54
  55       .. code-block:: bash
  56
  57          cd ms/blueprintsprocessor/application/src/main/dc
  58
  59    .. group-tab:: El Alto
  60
  61       .. code-block:: bash
  62
  63          ms/blueprintsprocessor/distribution/src/main/dc
  64
  65    .. group-tab:: Dublin
  66
  67       .. code-block:: bash
  68
  69          ms/blueprintsprocessor/distribution/src/main/dc
  70
  71 And run docker-composer:
  72
  73 .. code-block:: bash
  74
  75     docker-compose up -d db
  76
  77 This should spin up a container of the MariaDB image in the background.
  78 To check if it has worked, this command can be used:
  79
  80 .. code-block:: bash
  81
  82     docker-compose logs -f
  83
  84 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  85
  86 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  87 it can be started again by the command:
  88
  89 .. code-block:: bash
  90
  91    docker start <id of mariadb container>
  92
  93
  94 Set permissions on the local file system
  95 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  96
  97 Blueprints processor uses the local file system for some operations and, therefore,
  98 need some existing and accessible paths to run properly.
  99
 100 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
 101
 102 .. code-block:: bash
 103
 104    mkdir -p -m 755 /opt/app/onap/blueprints/archive
 105    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
 106    mkdir -p -m 755 /opt/app/onap/scripts
 107    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
 108
 109 Import the project into the IDE
 110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 111
 112 .. tabs::
 113
 114    .. tab:: IntelliJ IDEA
 115
 116       Go to *File | Open* and choose the :file:`pom.xml` file of the cds directory:
 117
 118       |imageImportProject|
 119
 120       Sometimes it may be necessary to reimport Maven project:
 121
 122       |imageReimportMaven|
 123
 124
 125       **Override some application properties:**
 126
 127       After the project is compiled, a Run Configuration profile overriding some application properties
 128       with custom values needs to be created, to reflect the local environment characteristics.
 129
 130       .. tabs::
 131
 132          .. group-tab:: Latest
 133
 134             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 135
 136             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 137
 138             Right-click inside it, at any point, to load the context menu and select create
 139             BlueprintProcessorApplication configuration from context:
 140
 141             |imageCreateRunConfigkt|
 142
 143             **The following window will open:**
 144
 145             |imageRunConfigKt|
 146
 147             **Add the following in the field `VM Options`:**
 148
 149             .. code-block:: bash
 150                :caption: **Custom values for properties**
 151
 152                -Dspring.profiles.active=dev
 153
 154             You can override any value from **application-dev.properties** file here. Use the following pattern:
 155
 156             .. code-block:: java
 157
 158                -D<application-dev.properties key>=<application-dev.properties value>
 159
 160          .. group-tab:: Frankfurt
 161
 162             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 163
 164             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 165
 166             Right-click inside it, at any point, to load the context menu and select create
 167             BlueprintProcessorApplication configuration from context:
 168
 169             |imageCreateRunConfigkt|
 170
 171             **The following window will open:**
 172
 173             |imageRunConfigKt|
 174
 175             **Add the following in the field `VM Options`:**
 176
 177             .. code-block:: bash
 178                :caption: **Custom values for properties**
 179
 180                -Dspring.profiles.active=dev
 181
 182             You can override any value from **application-dev.properties** file here. Use the following pattern:
 183
 184             .. code-block:: java
 185
 186                -D<application-dev.properties key>=<application-dev.properties value>
 187
 188          .. group-tab:: El Alto
 189
 190             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 191
 192             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 193
 194             Right-click inside it, at any point, to load the context menu and select create
 195             BlueprintProcessorApplication configuration from context:
 196
 197             |imageCreateRunConfigJava|
 198
 199             **The following window will open:**
 200
 201             |imageRunConfigJava|
 202
 203             **Add the following in the field `VM Options`:**
 204
 205             .. code-block:: bash
 206                :caption: **Custom values for properties**
 207
 208                -Dspring.profiles.active=dev
 209
 210             You can override any value from **application-dev.properties** file here. Use the following pattern:
 211
 212             .. code-block:: java
 213
 214                -D<application-dev.properties key>=<application-dev.properties value>
 215
 216          .. group-tab:: Dublin
 217
 218             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 219
 220             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 221
 222             Right-click inside it, at any point, to load the context menu and select create
 223             BlueprintProcessorApplication configuration from context:
 224
 225             |imageCreateRunConfigJava|
 226
 227             **The following window will open:**
 228
 229             |imageRunConfigJava|
 230
 231             **Add the following in that field:**
 232
 233             .. code-block:: java
 234                :caption: **Custom values for properties**
 235
 236                -DappName=ControllerBluePrints
 237                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 238                -DappVersion=1.0.0
 239                -Dspring.config.location=opt/app/onap/config/
 240                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 241                -Dspring.datasource.username=sdnctl
 242                -Dspring.datasource.password=sdnctl
 243                -Dcontrollerblueprints.loadInitialData=true
 244                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 245                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 246                -Dblueprintsprocessor.db.primary.username=sdnctl
 247                -Dblueprintsprocessor.db.primary.password=sdnctl
 248                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 249                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 250                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 251                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 252                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 253                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 254                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 255                -Dblueprintsprocessor.restconfEnabled=true
 256                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 257                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 258                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 259                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 260                -Dblueprintsprocessor.grpcEnable=false
 261                -Dblueprintsprocessor.grpcPort=9111
 262                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 263                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 264                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 265                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 266                -Dsecurity.user.name=ccsdkapps
 267                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 268                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 269                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 270                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 271                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 272                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 273                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 274                -Dspring.profiles.active=dev
 275                -Dblueprintsprocessor.httpPort=8080
 276                -Dserver.port=55555
 277
 278
 279       **Browse Working Directory to your application path**  ``.../cds/ms/blueprintsprocessor/application``
 280       **if path is not already specified correctly.**
 281
 282       **Add/replace the following in Blueprint's application-dev.properties file:**
 283
 284       .. code-block:: java
 285
 286          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 287          blueprintsprocessor.grpcclient.remote-python.host=localhost
 288          blueprintsprocessor.grpcclient.remote-python.port=50051
 289          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 290
 291          blueprintprocessor.remoteScriptCommand.enabled=true
 292
 293
 294       **Run the application:**
 295
 296       Select either run or debug for this Run Configuration to start the Blueprints Processor:
 297
 298       |imageRunDebug|
 299
 300       |imageBuildLogs|
 301
 302    .. tab:: Visual Studio Code
 303
 304       .. tabs::
 305
 306          .. group-tab:: Latest
 307
 308             * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 309             * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 310             * **Step #3** - On the top menu click *Run | Open Configurations*
 311
 312             .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 313                Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 314
 315             * **Step #4** - add configuration shown below to your configurations list.
 316
 317             .. code-block:: json
 318
 319                {
 320                      "type": "kotlin",
 321                      "request": "launch",
 322                      "name": "Blueprint Processor",
 323                      "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 324                      "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 325                }
 326
 327             .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 328
 329             .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 330                If you have a cleaner idea how to solve this - please let us know.
 331
 332             **Add/replace the following in Blueprint's application-dev.properties file:**
 333
 334             .. code-block:: java
 335
 336                blueprintsprocessor.grpcclient.remote-python.type=token-auth
 337                blueprintsprocessor.grpcclient.remote-python.host=localhost
 338                blueprintsprocessor.grpcclient.remote-python.port=50051
 339                blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 340
 341                blueprintprocessor.remoteScriptCommand.enabled=true
 342
 343             **Currently the following entries need to be added in VSC too:**
 344
 345             .. code-block:: java
 346
 347                logging.level.web=DEBUG
 348                logging.level.org.springframework.web: DEBUG
 349
 350                #Encrypted username and password for health check service
 351                endpoints.user.name=eHbVUbJAj4AG2522cSbrOQ==
 352                endpoints.user.password=eHbVUbJAj4AG2522cSbrOQ==
 353
 354                #BaseUrls for health check blueprint processor services
 355                blueprintprocessor.healthcheck.baseUrl=http://localhost:8080/
 356                blueprintprocessor.healthcheck.mapping-service-name-with-service-link=[Execution service,/api/v1/execution-service/health-check],[Resources service,/api/v1/resources/health-check],[Template service,/api/v1/template/health-check]
 357
 358                #BaseUrls for health check Cds Listener services
 359                cdslistener.healthcheck.baseUrl=http://cds-sdc-listener:8080/
 360                cdslistener.healthcheck.mapping-service-name-with-service-link=[SDC Listener service,/api/v1/sdclistener/healthcheck]
 361
 362                #Actuator properties
 363                management.endpoints.web.exposure.include=*
 364                management.endpoint.health.show-details=always
 365                management.info.git.mode=full
 366
 367             In VSC the properties are read from target folder, thats why the following maven command needs to be rerun:
 368
 369             .. code-block:: bash
 370
 371                mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
 372
 373             Click Run in Menu bar.
 374
 375             |imageLogsVSC|
 376
 377
 378 Testing the application
 379 ~~~~~~~~~~~~~~~~~~~~~~~~~
 380
 381 There are two main features of the Blueprints Processor that can be of interest of a developer:
 382 blueprint publish and blueprint process.
 383
 384 To upload custom blueprints,  the endpoint ``api/v1/execution-service/publish`` is used.
 385
 386 To process, the endpoint is ``api/v1/execution-service/process``.
 387
 388 Postman is a software that can be used to send these request, and an example of
 389 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 390
 391 A detailed description of the usage of different APIs of CDS will follow.
 392
 393 Possible Fixes
 394 ~~~~~~~~~~~~~~~~~~~
 395
 396 Imported packages or annotiations are not found, Run Config not available?
 397 *****************************************************************************
 398
 399 1. Rebuild with ``maven install ...`` (see above)
 400 2. Potentially change Maven home directory in Settings
 401 3. Maven reimport in IDE
 402
 403 Compilation error?
 404 ********************
 405
 406 * Change Java Version to 11
 407
 408
 409 .. image alignment inside tabs doesn't work
 410
 411 .. |imageRunConfigJava| image:: media/run_config_java.png
 412    :width: 500pt
 413    :align: middle
 414
 415 .. |imageRunConfigKt| image:: media/run_config_kt.png
 416    :width: 500pt
 417    :align: middle
 418
 419 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 420    :width: 500pt
 421    :align: middle
 422
 423 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 424    :width: 500pt
 425    :align: middle
 426
 427 .. |imageImportProject| image:: media/import_project.png
 428    :width: 300pt
 429    :align: middle
 430
 431 .. |imageReimportMaven| image:: media/reimport_maven.png
 432    :width: 400pt
 433    :align: middle
 434
 435 .. |imageRunDebug| image:: media/run_debug.png
 436    :width: 500pt
 437    :align: middle
 438
 439 .. |imageBuildLogs| image:: media/build_logs.png
 440    :width: 500pt
 441    :align: middle
 442
 443 .. |imageLogsVSC| image:: media/vsc_logs.png
 444    :width: 500pt
 445    :align: middle