docs/deployment.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3 .. Copyright (C) 2021-2024 Nordix Foundation
   4 .. Modifications Copyright (C) 2021 Bell Canada.
   5
   6 .. DO NOT CHANGE THIS LABEL FOR RELEASE NOTES - EVEN THOUGH IT GIVES A WARNING
   7 .. _deployment:
   8
   9 CPS Deployment
  10 ##############
  11
  12 .. contents::
  13     :depth: 2
  14
  15 Database configuration
  16 ======================
  17 CPS uses PostgreSQL database. As per the `PostgreSQL documentation on resource consumption
  18 <https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-SHARED-BUFFERS>`_, the *shared_buffers*
  19 parameter should be set between 25% and 40% of total memory. It has a default value of 128 megabytes, so this should be
  20 set appropriately. For example, given a database with 2GB of memory, 512MB is a recommended value.
  21
  22 CPS OOM Charts
  23 ==============
  24 The CPS kubernetes chart is located in the `OOM repository <https://github.com/onap/oom/tree/master/kubernetes/cps>`_.
  25 This chart includes different cps components referred as <cps-component-name> further in the document are listed below:
  26
  27 .. container:: ulist
  28
  29   - `cps-core <https://github.com/onap/oom/tree/master/kubernetes/cps/components/cps-core>`__
  30   - `ncmp-dmi-plugin <https://github.com/onap/oom/tree/master/kubernetes/cps/components/ncmp-dmi-plugin>`__
  31
  32 Please refer to the `OOM documentation <https://docs.onap.org/projects/onap-oom/en/latest/sections/guides/user_guides/oom_user_guide.html>`_ on how to install and deploy ONAP.
  33
  34 Installing or Upgrading CPS Components
  35 ======================================
  36
  37 The assumption is you have cloned the charts from the OOM repository into a local directory.
  38
  39 **Step 1** Go to the cps charts and edit properties in values.yaml files to make any changes to particular cps component if required.
  40
  41 .. code-block:: bash
  42
  43   cd oom/kubernetes/cps/components/<cps-component-name>
  44
  45 **Step 2** Build the charts
  46
  47 .. code-block:: bash
  48
  49   cd oom/kubernetes
  50   make SKIP_LINT=TRUE cps
  51
  52 .. note::
  53    SKIP_LINT is only to reduce the "make" time
  54
  55 **Step 3** Undeploying already deployed cps components
  56
  57 After undeploying cps components, keep monitoring the cps pods until they go away.
  58
  59 .. code-block:: bash
  60
  61   helm del --purge <my-helm-release>-<cps-component-name>
  62   kubectl get pods -n <namespace> | grep <cps-component-name>
  63
  64 **Step 4** Make sure there is no orphan database persistent volume or claim.
  65
  66 First, find if there is an orphan database PV or PVC with the following commands:
  67
  68 .. note::
  69    This step does not apply to ncmp-dmi-plugin.
  70
  71 .. code-block:: bash
  72
  73   kubectl get pvc -n <namespace> | grep <cps-component-name>
  74   kubectl get pv -n <namespace> | grep <cps-component-name>
  75
  76 If there are any orphan resources, delete them with
  77
  78 .. code-block:: bash
  79
  80     kubectl delete pvc <orphan-cps-core-pvc-name>
  81     kubectl delete pv <orphan-cps-core-pv-name>
  82
  83 **Step 5** Delete NFS persisted data for CPS components
  84
  85 Connect to the machine where the file system is persisted and then execute the below command
  86
  87 .. code-block:: bash
  88
  89   rm -fr /dockerdata-nfs/<my-helm-release>/<cps-component-name>
  90
  91 **Step 6** Re-Deploy cps pods
  92
  93 After deploying cps, keep monitoring the cps pods until they come up.
  94
  95 .. code-block:: bash
  96
  97   helm deploy <my-helm-release> local/cps --namespace <namespace>
  98   kubectl get pods -n <namespace> | grep <cps-component-name>
  99
 100 Restarting a faulty component
 101 =============================
 102 Each cps component can be restarted independently by issuing the following command:
 103
 104 .. code-block:: bash
 105
 106     kubectl delete pod <cps-component-pod-name> -n <namespace>
 107
 108 .. Below Label is used by documentation for other CPS components to link here, do not remove even if it gives a warning
 109 .. _cps_common_credentials_retrieval:
 110
 111 Credentials Retrieval
 112 =====================
 113
 114 Application and database credentials are kept in Kubernetes secrets. They are defined as external secrets in the
 115 values.yaml file to be used across different components as :
 116
 117 .. container:: ulist
 118
 119   - `cps-core <https://github.com/onap/oom/blob/master/kubernetes/cps/components/cps-core/values.yaml>`_
 120   - `ncmp-dmi-plugin <https://github.com/onap/oom/blob/master/kubernetes/cps/components/ncmp-dmi-plugin/values.yaml>`_
 121
 122 Below are the list of secrets for different cps components.
 123
 124 +--------------------------+---------------------------------+---------------------------------------------------+
 125 | Component                | Secret type                     | Secret Name                                       |
 126 +==========================+=================================+===================================================+
 127 | cps-core                 | Database authentication         | <my-helm-release>-cps-core-pg-user-creds          |
 128 +--------------------------+---------------------------------+---------------------------------------------------+
 129 | cps-core                 | Rest API Authentication         | <my-helm-release>-cps-core-app-user-creds         |
 130 +--------------------------+---------------------------------+---------------------------------------------------+
 131 | ncmp-dmi-plugin          | Rest API Authentication         | <my-helm-release>-cps-dmi-plugin-user-creds       |
 132 +--------------------------+---------------------------------+---------------------------------------------------+
 133 | ncmp-dmi-plugin          | SDNC authentication             | <my-helm-release>-ncmp-dmi-plugin-sdnc-creds      |
 134 +--------------------------+---------------------------------+---------------------------------------------------+
 135
 136 The credential values from these secrets are configured in running container as environment variables. Eg:
 137 `cps core deployment.yaml <https://github.com/onap/oom/blob/master/kubernetes/cps/components/cps-core/templates/deployment.yaml>`_
 138
 139 If no specific passwords are provided to the chart as override values for deployment, then passwords are automatically
 140 generated when deploying the Helm release. Below command can be used to retrieve application property credentials
 141
 142 .. code::
 143
 144   kubectl get secret <my-helm-release>-<secret-name> -n <namespace> -o json | jq '.data | map_values(@base64d)'
 145
 146 .. note::
 147    base64d works only with jq version 1.6 or above.
 148
 149 CPS Core Pods
 150 =============
 151 To get a listing of the cps-core Pods, run the following command:
 152
 153 .. code-block:: bash
 154
 155   kubectl get pods -n <namespace> | grep cps-core
 156
 157   dev-cps-core-ccd4cc956-r98pv                          1/1     Running            0          24h
 158   dev-cps-core-postgres-primary-f7766d46c-s9d5b         1/1     Running            0          24h
 159   dev-cps-core-postgres-replica-84659d68f9-6qnt4        1/1     Running            0          24h
 160
 161 .. note::
 162     The CPS Service will have to be restarted each time a change is made to a configurable property.
 163
 164 Additional CPS-Core Customizations
 165 ==================================
 166
 167 The following table lists some properties that can be specified as Helm chart
 168 values to configure the application to be deployed. This list is not exhaustive.
 169
 170 Any spring supported property can be configured by providing in ``config.additional.<spring-supported-property-name>: value`` Example: config.additional.spring.datasource.hikari.maximumPoolSize: 30
 171
 172 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 173 | Property                                  | Description                                                                                             | Default Value                 |
 174 +===========================================+=========================================================================================================+===============================+
 175 | config.appUserName                        | User name used by cps-core service to configure the authentication for REST API it exposes.             | ``cpsuser``                   |
 176 |                                           |                                                                                                         |                               |
 177 |                                           | This is the user name to be used by cps-core REST clients to authenticate themselves.                   |                               |
 178 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 179 | config.appUserPassword                    | Password used by cps-core service to configure the authentication for REST API it exposes.              | Not defined                   |
 180 |                                           |                                                                                                         |                               |
 181 |                                           | If not defined, the password is generated when deploying the application.                               |                               |
 182 |                                           |                                                                                                         |                               |
 183 |                                           | See also :ref:`cps_common_credentials_retrieval`.                                                       |                               |
 184 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 185 | postgres.config.pgUserName                | Internal user name used by cps-core to connect to its own database.                                     | ``cps``                       |
 186 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 187 | postgres.config.pgUserPassword            | Internal password used by cps-core to connect to its own database.                                      | Not defined                   |
 188 |                                           |                                                                                                         |                               |
 189 |                                           | If not defined, the password is generated when deploying the application.                               |                               |
 190 |                                           |                                                                                                         |                               |
 191 |                                           | See also :ref:`cps_common_credentials_retrieval`.                                                       |                               |
 192 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 193 | postgres.config.pgDatabase                | Database name used by cps-core                                                                          | ``cpsdb``                     |
 194 |                                           |                                                                                                         |                               |
 195 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 196 | logging.level                             | Logging level set in cps-core                                                                           | info                          |
 197 |                                           |                                                                                                         |                               |
 198 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 199 | config.useStrimziKafka                    | If targeting a custom kafka cluster, i.e. useStrimziKafka: false, the                                   | true                          |
 200 |                                           | config.eventPublisher.spring.kafka values below must be set.                                            |                               |
 201 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 202 | config.eventPublisher.                    | Kafka hostname and port                                                                                 | ``<kafka-bootstrap>:9092``    |
 203 | spring.kafka.bootstrap-servers            |                                                                                                         |                               |
 204 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 205 | config.eventPublisher.                    | Kafka consumer client id                                                                                | ``cps-core``                  |
 206 | spring.kafka.consumer.client-id           |                                                                                                         |                               |
 207 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 208 | config.eventPublisher.                    | Kafka security protocol.                                                                                | ``SASL_PLAINTEXT``            |
 209 | spring.kafka.security.protocol            | Some possible values are:                                                                               |                               |
 210 |                                           |                                                                                                         |                               |
 211 |                                           | * ``PLAINTEXT``                                                                                         |                               |
 212 |                                           | * ``SASL_PLAINTEXT``, for authentication                                                                |                               |
 213 |                                           | * ``SASL_SSL``, for authentication and encryption                                                       |                               |
 214 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 215 | config.eventPublisher.                    | Kafka security SASL mechanism. Required for SASL_PLAINTEXT and SASL_SSL protocols.                      | Not defined                   |
 216 | spring.kafka.properties.                  | Some possible values are:                                                                               |                               |
 217 | sasl.mechanism                            |                                                                                                         |                               |
 218 |                                           | * ``PLAIN``, for PLAINTEXT                                                                              |                               |
 219 |                                           | * ``SCRAM-SHA-512``, for SSL                                                                            |                               |
 220 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 221 | config.eventPublisher.                    | Kafka security SASL JAAS configuration. Required for SASL_PLAINTEXT and SASL_SSL protocols.             | Not defined                   |
 222 | spring.kafka.properties.                  | Some possible values are:                                                                               |                               |
 223 | sasl.jaas.config                          |                                                                                                         |                               |
 224 |                                           | * ``org.apache.kafka.common.security.plain.PlainLoginModule required username="..." password="...";``,  |                               |
 225 |                                           |   for PLAINTEXT                                                                                         |                               |
 226 |                                           | * ``org.apache.kafka.common.security.scram.ScramLoginModule required username="..." password="...";``,  |                               |
 227 |                                           |   for SSL                                                                                               |                               |
 228 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 229 | config.eventPublisher.                    | Kafka security SASL SSL store type. Required for SASL_SSL protocol.                                     | Not defined                   |
 230 | spring.kafka.ssl.trust-store-type         | Some possible values are:                                                                               |                               |
 231 |                                           |                                                                                                         |                               |
 232 |                                           | * ``JKS``                                                                                               |                               |
 233 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 234 | config.eventPublisher.                    | Kafka security SASL SSL store file location. Required for SASL_SSL protocol.                            | Not defined                   |
 235 | spring.kafka.ssl.trust-store-location     |                                                                                                         |                               |
 236 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 237 | config.eventPublisher.                    | Kafka security SASL SSL store password. Required for SASL_SSL protocol.                                 | Not defined                   |
 238 | spring.kafka.ssl.trust-store-password     |                                                                                                         |                               |
 239 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 240 | config.eventPublisher.                    | Kafka security SASL SSL broker hostname identification verification. Required for SASL_SSL protocol.    | Not defined                   |
 241 | spring.kafka.properties.                  | Possible value is:                                                                                      |                               |
 242 | ssl.endpoint.identification.algorithm     |                                                                                                         |                               |
 243 |                                           | * ``""``, empty string to disable                                                                       |                               |
 244 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 245 | config.additional.                        | Core pool size in asynchronous execution of notification.                                               | ``2``                         |
 246 | notification.async.executor.              |                                                                                                         |                               |
 247 | core-pool-size                            |                                                                                                         |                               |
 248 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 249 | config.additional.                        | Max pool size in asynchronous execution of notification.                                                | ``1``                         |
 250 | notification.async.executor.              |                                                                                                         |                               |
 251 | max-pool-size                             |                                                                                                         |                               |
 252 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 253 | config.additional.                        | Queue Capacity in asynchronous execution of notification.                                               | ``500``                       |
 254 | notification.async.executor.              |                                                                                                         |                               |
 255 | queue-capacity                            |                                                                                                         |                               |
 256 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 257 | config.additional.                        | If the executor should wait for the tasks to be completed on shutdown                                   | ``true``                      |
 258 | notification.async.executor.              |                                                                                                         |                               |
 259 | wait-for-tasks-to-complete-on-shutdown    |                                                                                                         |                               |
 260 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 261 | config.additional.                        | Prefix to be added to the thread name in asynchronous execution of notifications.                       | ``Async-``                    |
 262 | notification.async.executor.              |                                                                                                         |                               |
 263 | thread-name-prefix                        |                                                                                                         |                               |
 264 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 265 | config.additional.                        | Maximum time allowed by the thread pool executor for execution of one of the threads in milliseconds.   | ``60000``                     |
 266 | notification.async.executor.              |                                                                                                         |                               |
 267 | time-out-value-in-ms                      |                                                                                                         |                               |
 268 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 269 | config.additional.                        | Specifies number of database connections between database and application.                              | ``10``                        |
 270 | spring.datasource.hikari.                 | This property controls the maximum size that the pool is allowed to reach,                              |                               |
 271 | maximumPoolSize                           | including both idle and in-use connections.                                                             |                               |
 272 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 273
 274 .. _additional-cps-ncmp-customizations:
 275
 276 Additional CPS-NCMP Customizations
 277 ==================================
 278 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 279 | config.dmiPluginUserName                  | User name used by cps-core to authenticate themselves for using ncmp-dmi-plugin service.                | ``dmiuser``                   |
 280 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 281 | config.dmiPluginUserPassword              | Internal password used by cps-core to connect to ncmp-dmi-plugin service.                               | Not defined                   |
 282 |                                           |                                                                                                         |                               |
 283 |                                           | If not defined, the password is generated when deploying the application.                               |                               |
 284 |                                           |                                                                                                         |                               |
 285 |                                           | See also :ref:`cps_common_credentials_retrieval`.                                                       |                               |
 286 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 287 | config.ncmp.timers                        | Specifies the delay in milliseconds in which the module sync watch dog will wake again after finishing. | ``30000``                     |
 288 | .advised-modules-sync.sleep-time-ms       |                                                                                                         |                               |
 289 |                                           |                                                                                                         |                               |
 290 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 291 | config.ncmp.timers                        | Specifies the delay in milliseconds in which the retry mechanism watch dog                              |                               |
 292 | .locked-modules-sync.sleep-time-ms        | will wake again after finishing.                                                                        | ``300000``                    |
 293 |                                           |                                                                                                         |                               |
 294 |                                           |                                                                                                         |                               |
 295 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 296 | config.ncmp.timers                        | Specifies the delay in milliseconds in which the data sync watch dog will wake again after finishing.   | ``30000``                     |
 297 | .cm-handle-data-sync.sleep-time-ms        |                                                                                                         |                               |
 298 |                                           |                                                                                                         |                               |
 299 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 300 | config.additional.ncmp.dmi.httpclient     | Specifies the maximum time in seconds, to wait for establishing a connection for the HTTP Client.       | ``180``                       |
 301 | .connectionTimeoutInSeconds               |                                                                                                         |                               |
 302 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 303 | config.additional.ncmp.dmi.httpclient     | Specifies the maximum number of connections allowed per route in the HTTP client.                       | ``50``                        |
 304 | .maximumConnectionsPerRoute               |                                                                                                         |                               |
 305 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 306 | config.additional.ncmp.dmi.httpclient     | Specifies the maximum total number of connections that can be held by the HTTP client.                  | ``100``                       |
 307 | .maximumConnectionsTotal                  |                                                                                                         |                               |
 308 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 309 | config.additional.ncmp.dmi.httpclient     | Specifies the duration in seconds for the threshold, after which idle connections will be evicted       | ``5``                         |
 310 | .idleConnectionEvictionThresholdInSeconds | from the connection pool by the HTTP client.                                                            |                               |
 311 +-------------------------------------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 312
 313 CPS-Core Docker Installation
 314 ============================
 315
 316 CPS-Core can also be installed in a docker environment. Latest `docker-compose <https://github.com/onap/cps/blob/master/docker-compose/docker-compose.yml>`_ is included in the repo to start all the relevant services.
 317 The latest instructions are covered in the `README <https://github.com/onap/cps/blob/master/docker-compose/README.md>`_.
 318
 319 .. Below Label is used by documentation for other CPS components to link here, do not remove even if it gives a warning
 320 .. _cps_common_distributed_datastructures:
 321
 322 CPS-Core and NCMP Distributed Datastructures
 323 ============================================
 324
 325 CPS-Core and NCMP both internally uses embedded distributed datastructure to replicate the state across various instances for low latency.
 326 These instances require some additional ports to be available. The default range starts from 5701 and based on the number of instances configured they are incremented sequentially.
 327
 328 Below are the list of distributed datastructures that we have.
 329
 330 +--------------+------------------------------------+-----------------------------------------------------------+
 331 | Component    | Datastructure name                 |                 Use                                       |
 332 +==============+====================================+===========================================================+
 333 | cps-core     | anchorDataCache                    | Used to resolve prefix for the container name.            |
 334 +--------------+------------------------------------+-----------------------------------------------------------+
 335 | cps-ncmp     | moduleSyncStartedOnCmHandles       | Watchdog process to register cm handles.                  |
 336 +--------------+------------------------------------+-----------------------------------------------------------+
 337 | cps-ncmp     | dataSyncSemaphores                 | Watchdog process to sync data from the nodes.             |
 338 +--------------+------------------------------------+-----------------------------------------------------------+
 339 | cps-ncmp     | moduleSyncWorkQueue                | Queue used internally for workers to pick the task.       |
 340 +--------------+------------------------------------+-----------------------------------------------------------+
 341 | cps-ncmp     | untrustworthyCmHandlesSet          | Stores untrustworthy cm handles whose trust level is NONE.|
 342 +--------------+------------------------------------+-----------------------------------------------------------+
 343 | cps-ncmp     | trustLevelPerDmiPlugin             | Stores the trust level for the dmi-plugins.               |
 344 +--------------+------------------------------------+-----------------------------------------------------------+
 345 | cps-ncmp     | moduleSetTagCacheMapConfig         | Stores the module set tags for cm handles.                |
 346 +--------------+------------------------------------+-----------------------------------------------------------+
 347 | cps-ncmp     | cmNotificationSubscriptionCache    | Stores and tracks cm notification subscription requests.  |
 348 +--------------+------------------------------------+-----------------------------------------------------------+
 349
 350 Total number of caches : 8