docs/developerguide/running-bp-processor-in-ide.rst

   1 .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE
   2 .. This work is licensed under a Creative Commons Attribution 4.0
   3 .. International License. http://creativecommons.org/licenses/by/4.0
   4 .. Copyright (C) 2020 Deutsche Telekom AG.
   5
   6 Running Blueprints Processor Microservice in an IDE
   7 ====================================================
   8
   9 Objective
  10 ~~~~~~~~~~~~
  11
  12 Have the blueprint processor running locally is to use the IDE to run the code, while having the database running in a container.
  13 This way, code changes can be conveniently tested and debugged.
  14
  15
  16 Check out the code
  17 ~~~~~~~~~~~~~~~~~~~
  18
  19 Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds
  20
  21 Build it locally
  22 ~~~~~~~~~~~~~~~~~~
  23
  24 In the checked out directory, type
  25
  26 .. code-block:: bash
  27
  28    mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
  29
  30 Spin up a Docker container with the database
  31 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  32
  33 The Blueprints Processor project uses a database to store information about the blueprints
  34 and therefore it needs to be online before attempting to run it.
  35
  36 One way to create the database is by using the :file:`docker-compose.yaml`
  37 file present on the distribution module. This database will require a local directory to mount a volume, before running docker-compose remember to create following directory:
  38
  39 .. code-block:: bash
  40
  41    mkdir -p -m 755 /opt/app/cds/mysql/data
  42
  43 Navigate to the docker-compose file in the distribution module:
  44
  45 .. tabs::
  46
  47    .. group-tab:: Latest
  48
  49       .. code-block:: bash
  50
  51          cd ms/blueprintsprocessor/application/src/main/dc
  52
  53    .. group-tab:: Frankfurt
  54
  55       .. code-block:: bash
  56
  57          cd ms/blueprintsprocessor/application/src/main/dc
  58
  59    .. group-tab:: El Alto
  60
  61       .. code-block:: bash
  62
  63          ms/blueprintsprocessor/distribution/src/main/dc
  64
  65    .. group-tab:: Dublin
  66
  67       .. code-block:: bash
  68
  69          ms/blueprintsprocessor/distribution/src/main/dc
  70
  71 And run docker-composer:
  72
  73 .. code-block:: bash
  74
  75     docker-compose up -d db
  76
  77 This should spin up a container of the MariaDB image in the background.
  78 To check if it has worked, this command can be used:
  79
  80 .. code-block:: bash
  81
  82     docker-compose logs -f
  83
  84 The phrase ``mysqld: ready for connections`` indicates that the database was started correctly.
  85
  86 From now on, the Docker container will be available on the computer; if it ever gets stopped,
  87 it can be started again by the command:
  88
  89 .. code-block:: bash
  90
  91    docker start <id of mariadb container>
  92
  93
  94 Set permissions on the local file system
  95 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  96
  97 Blueprints processor uses the local file system for some operations and, therefore,
  98 need some existing and accessible paths to run properly.
  99
 100 Execute the following commands to create the needed directories, and grant access to the current user to modify them:
 101
 102 .. code-block:: bash
 103
 104    mkdir -p -m 755 /opt/app/onap/blueprints/archive
 105    mkdir -p -m 755 /opt/app/onap/blueprints/deploy
 106    mkdir -p -m 755 /opt/app/onap/scripts
 107    sudo chown -R $(id -u):$(id -g) /opt/app/onap/
 108
 109 Import the project into the IDE
 110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 111
 112 .. tabs ::
 113
 114    .. group-tab:: IntelliJ IDEA
 115
 116       Go to *File | Open* and choose the :file:`pom.xml` file of the cds directory:
 117
 118       |imageImportProject|
 119
 120       Sometimes it may be necessary to reimport Maven project:
 121
 122       |imageReimportMaven|
 123
 124
 125       Override some application properties
 126       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 127
 128       After the project is compiled, a Run Configuration profile overriding some application properties
 129       with custom values needs to be created, to reflect the local environment characteristics.
 130
 131       .. tabs::
 132
 133          .. group-tab:: Latest
 134
 135             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 136
 137             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 138
 139             Right-click inside it, at any point, to load the context menu and select create
 140             BlueprintProcessorApplication configuration from context:
 141
 142             |imageCreateRunConfigkt|
 143
 144             **The following window will open:**
 145
 146             |imageRunConfigKt|
 147
 148             **Add the following in the field `VM Options`:**
 149
 150             .. code-block:: bash
 151                :caption: **Custom values for properties**
 152
 153                -Dspring.profiles.active=dev
 154
 155             You can override any value from **application-dev.properties** file here. Use the following pattern:
 156
 157             .. code-block:: java
 158
 159                -D<application-dev.properties key>=<application-dev.properties value>
 160
 161          .. group-tab:: Frankfurt
 162
 163             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 164
 165             ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``.
 166
 167             Right-click inside it, at any point, to load the context menu and select create
 168             BlueprintProcessorApplication configuration from context:
 169
 170             |imageCreateRunConfigkt|
 171
 172             **The following window will open:**
 173
 174             |imageRunConfigKt|
 175
 176             **Add the following in the field `VM Options`:**
 177
 178             .. code-block:: bash
 179                :caption: **Custom values for properties**
 180
 181                -Dspring.profiles.active=dev
 182
 183             You can override any value from **application-dev.properties** file here. Use the following pattern:
 184
 185             .. code-block:: java
 186
 187                -D<application-dev.properties key>=<application-dev.properties value>
 188
 189          .. group-tab:: El Alto
 190
 191             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 192
 193             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.``
 194
 195             Right-click inside it, at any point, to load the context menu and select create
 196             BlueprintProcessorApplication configuration from context:
 197
 198             |imageCreateRunConfigJava|
 199
 200             **The following window will open:**
 201
 202             |imageRunConfigJava|
 203
 204             **Add the following in the field `VM Options`:**
 205
 206             .. code-block:: bash
 207                :caption: **Custom values for properties**
 208
 209                -Dspring.profiles.active=dev
 210
 211             You can override any value from **application-dev.properties** file here. Use the following pattern:
 212
 213             .. code-block:: java
 214
 215                -D<application-dev.properties key>=<application-dev.properties value>
 216
 217          .. group-tab:: Dublin
 218
 219             Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class:
 220
 221             ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``.
 222
 223             Right-click inside it, at any point, to load the context menu and select create
 224             BlueprintProcessorApplication configuration from context:
 225
 226             |imageCreateRunConfigJava|
 227
 228             **The following window will open:**
 229
 230             |imageRunConfigJava|
 231
 232             **Add the following in that field:**
 233
 234             .. code-block:: java
 235                :caption: **Custom values for properties**
 236
 237                -DappName=ControllerBluePrints
 238                -Dms_name=org.onap.ccsdk.apps.controllerblueprints
 239                -DappVersion=1.0.0
 240                -Dspring.config.location=opt/app/onap/config/
 241                -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl
 242                -Dspring.datasource.username=sdnctl
 243                -Dspring.datasource.password=sdnctl
 244                -Dcontrollerblueprints.loadInitialData=true
 245                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 246                -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl
 247                -Dblueprintsprocessor.db.primary.username=sdnctl
 248                -Dblueprintsprocessor.db.primary.password=sdnctl
 249                -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver
 250                -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update
 251                -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none
 252                -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy
 253                -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect
 254                -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints
 255                -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf
 256                -Dblueprintsprocessor.restconfEnabled=true
 257                -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth
 258                -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/
 259                -Dblueprintsprocessor.restclient.sdncodl.username=admin
 260                -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
 261                -Dblueprintsprocessor.grpcEnable=false
 262                -Dblueprintsprocessor.grpcPort=9111
 263                -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy
 264                -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive
 265                -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work
 266                -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu
 267                -Dsecurity.user.name=ccsdkapps
 268                -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false
 269                -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t
 270                -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth
 271                -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092
 272                -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t
 273                -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id
 274                -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id
 275                -Dspring.profiles.active=dev
 276                -Dblueprintsprocessor.httpPort=8080
 277                -Dserver.port=55555
 278
 279
 280       Add/replace the following in Blueprint's application-dev.properties file:
 281       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 282
 283       .. code-block:: java
 284
 285          blueprintsprocessor.grpcclient.remote-python.type=token-auth
 286          blueprintsprocessor.grpcclient.remote-python.host=localhost
 287          blueprintsprocessor.grpcclient.remote-python.port=50051
 288          blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw==
 289
 290          blueprintprocessor.remoteScriptCommand.enabled=true
 291
 292
 293       Run the application
 294       ~~~~~~~~~~~~~~~~~~~
 295
 296       Select either run or debug for this Run Configuration to start the Blueprints Processor:
 297
 298       |imageRunDebug|
 299
 300       |imageBuildLogs|
 301
 302    .. group-tab:: Visual Studio Code
 303
 304       * **Step #1** - Make sure your installation of Visual Studio Code is up to date. This guide was writen using version 1.48
 305       * **Step #2** - Install `Kotlin extension from the Visual Studio Code Marketplace <https://marketplace.visualstudio.com/items?itemName=fwcd.kotlin>`_
 306       * **Step #3** - On the top menu click *Run | Open Configurations*
 307
 308         .. warning:: This should open the file called `launch.json` but in some cases you'll need to wait for the Kotlin Language Server to be installed before you can do anything.
 309            Please watch the bottom bar in Visual Studio Code for messages about things getting installed.
 310
 311       * **Step #4** - add configuration shown below to your configurations list.
 312
 313         .. code-block:: json
 314
 315            {
 316                "type": "kotlin",
 317                "request": "launch",
 318                "name": "Blueprint Processor",
 319                "projectRoot": "${workspaceFolder}/ms/blueprintsprocessor/application",
 320                "mainClass": "-Dspring.profiles.active=dev org.onap.ccsdk.cds.blueprintsprocessor.BlueprintProcessorApplicationKt"
 321            }
 322
 323         .. warning:: The `projectRoot` path assumes that you created your Workspace in the main CDS repository folder. If not - please change the path accordingly
 324
 325         .. note:: The `mainClass` contains a spring profile param before the full class name. This is done because `args` is not supported by Kotlin launch.json configuration.
 326            If you have a cleaner idea how to solve this - please let us know.
 327
 328
 329 Testing the application
 330 ~~~~~~~~~~~~~~~~~~~~~~~~~
 331
 332 There are two main features of the Blueprints Processor that can be of interest of a developer:
 333 blueprint upload and blueprint process.
 334
 335 To upload custom blueprints,  the endpoint ``api/v1/execution-service/upload`` is used.
 336
 337 To process, the endpoint is ``api/v1/execution-service/process``.
 338
 339 Postman is a software that can be used to send these request, and an example of
 340 them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc.
 341
 342
 343 Possible Fixes
 344 ~~~~~~~~~~~~~~~~~~~
 345
 346 Imported packages or annotiations are not found, Run Config not available?
 347 *****************************************************************************
 348
 349 1. Rebuild with ``maven install ...`` (see above)
 350 2. Potentially change Maven home directory in Settings
 351 3. Maven reimport in IDE
 352
 353 Compilation error?
 354 ********************
 355
 356 * Change Java Version to 11 or 8
 357
 358 .. |imageRunConfigJava| image:: media/run_config_java.png
 359    :scale: 100 %
 360    :align: middle
 361
 362 .. |imageRunConfigKt| image:: media/run_config_kt.png
 363    :scale: 100 %
 364    :align: middle
 365
 366 .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png
 367    :scale: 100 %
 368    :align: middle
 369
 370 .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png
 371    :scale: 100 %
 372    :align: middle
 373
 374 .. |imageImportProject| image:: media/import_project.png
 375    :scale: 100 %
 376    :align: middle
 377
 378 .. |imageReimportMaven| image:: media/reimport_maven.png
 379    :scale: 70 %
 380    :align: middle
 381
 382 .. |imageRunDebug| image:: media/run_debug.png
 383    :scale: 100 %
 384    :align: middle
 385
 386 .. |imageRunDebug| image:: media/run_debug.png
 387    :align: middle
 388    :scale: 100 %
 389
 390 .. |imageBuildLogs| image:: media/build_logs.png
 391    :align: middle
 392    :scale: 100 %