1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Using Helm to deploy DCAE Microservices
5 =======================================
10 Prior to the ONAP Honolulu release, DCAE microservices were deployed
11 using the Cloudify orchestration tool. Each microservice had a Cloudify
12 *blueprint* containing the information needed for Cloudify to deploy the
13 microservice. The DCAE team provided a Cloudify plugin that used the
14 Kubernetes API to create the Kubernetes resources (including a
15 Kubernetes Deployment and a Kubernetes Service) that make up a running
16 instance of the microservice.
18 Beginning with the Honolulu release, DCAE is migrating to a new approach
19 for deploying DCAE microservices. Instead of using Cloudify with a
20 Cloudify blueprint for each microservice, DCAE will use Helm to deploy
21 microservices. Each microservice will have a Helm chart instead of a
22 Cloudify blueprint. In the Honolulu release, four DCAE microservices
23 (the VES and HV-VES collectors, the PNF registration handler, and the
24 TCA Gen2 analytics service) moved to Helm deployment. All four of these
25 are deployed “statically”–that is, they are deployed when DCAE is
26 installed and run continuously.
28 DCAE Service Templates - Introduction
29 -------------------------------------
31 It would be possible to write a Helm chart for each microservice, each
32 completely unrelated. We are taking a different approach. We are
33 providing shared Helm templates that (approximately) create the same
34 Kubernetes resources that the Cloudify plugin created when it processed
35 a blueprint. Creating a Helm chart for a microservice involves setting
36 up a Helm chart directory, which can be done by copying the chart
37 directory for an existing microservice and changing the ``Chart.yaml``
38 file (to set the name, description, and version of the chart) and the
39 ``values.yaml`` file (to customize the templates for the target
42 Once a chart for a microservice has been created, the chart can be used
43 to deploy the microservice, on demand, into a running instance of ONAP
44 and DCAE. This is similar to how we deployed microservices on demand
45 using a Cloudify blueprint and the Cloudify Manager (or the DCAE
48 The bulk of this document describes the different parameters that can be
49 set in the ``values.yaml`` file. There are two sets of parameters. One
50 set comes from the ONAP OOM common templates used by all of the ONAP
51 components that deployed via Helm. The other set consists of parameters
52 that are specific to the DCAE shared templates.
54 DCAE Service Templates - Location and content
55 ---------------------------------------------
56 The DCAE shared Helm charts for microservices are maintained in the
57 OOM repository, in the ``oom/kubernetes/dcaegen2-services/common/dcaegen2-services-common``
58 directory. In this directory subtree are:
60 - ``Chart.yaml``: the usual Helm chart definition file.
61 - ``requirements.yaml``: the dependencies for the chart. Only the OOM "common" chart is needed.
62 - ``values.yaml``: the file is present for completion, but there are no locally-defined values.
63 - ``templates/_configmap.tpl``: a template that sets up a configMap containing the microservices initial configuration and,
64 if needed, a configMap for filebeat logging configuration.
65 - ``templates/_deployment.tpl``: a template that sets up a Kubernetes Deployment for the microservice.
66 - ``templates/_filebeat-config.tpl``: a template containing the standard filebeat configuration for microservices that use filebeat logging.
67 It's used in the ``templates/_configmap.tpl`` template.
68 - ``templates/_job.tpl``: a template that creates a Kubernetes Job that runs when a microservice is deleted. The job brings up a container
69 that removes the microservice configuration information from Consul.
71 Setting variables in ``values.yaml`` for individual microservices
72 -----------------------------------------------------------------
74 Variables used by ONAP OOM common templates
75 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79 Name and tag of the Docker image for the microservice.
80 Required. The image repository is set using the OOM common
81 ``repositoryGenerator.repository`` template. Normally this points to the
82 ONAP image repository, but it can be overridden on a global basis or a
83 per-chart basis. See the OOM documentation for more details.
89 image: onap/org.onap.dcaegen2.services.prh.prh-app-server:1.5.6
91 **imageRepositoryOverride**:
93 Alternative repository for the Docker image for the microservice.
94 Optional. If this value is set, the Docker image for the microservice
95 will be pulled from the repository specified by this value, instead of
96 from the repository set by ``repositoryGenerator.repository``. Note that
97 this alternative repository is used only for the microservice image and
98 not for other images (such as images for initContainers). Note also that
99 the alternative repository must not use any form of authentication,
100 because there is no way to provide credentials for the repository.
106 imageRepositoryOverride: "myrepo.example.org:5000"
108 **global.pullPolicy** and **pullPolicy**:
110 These settings control when
111 the image is pulled from the repository. ``Always`` means the image is
112 always pulled from the repository when a container is created from the
113 image, while ``IfNotPresent`` means that the image is pulled from the
114 repository only if the image is not already present on the host machine
115 where the container is being created. Typical ONAP OOM practice is to
116 set ``pullPolicy`` to ``Always`` in the chart. During development and
117 testing, this can be overriden during the Helm install with
118 ``global.pullPolicy`` set to ``IfNotPresent``, to speed up deployment by
119 reducing the number of times images are pulled from the repository.
129 Many microservices depend on having other services
130 up and running in the DCAE and ONAP environment–services like AAF to get
131 certificates or DMaaP to communicate with other services.
132 ``readinessCheck.wait_for`` is a list of the *containers* that the
133 microservice needs to have available. If this parameter is present, an
134 initContainer will run and wait for all of the listed containers to
135 become ready. (Unfortunately, it’s necessary to know the name of a
136 *container*; it isn’t sufficient to list the name of a service.)
144 - dcae-config-binding-service
149 If this parameter is present, a Kubernetes readiness
150 probe will be configured for the microservice. The template supports
151 either an HTTP(S) readiness probe or a script-based readiness probe. The
152 parameter has the following fields that apply to both types:
154 1. ``initialDelaySeconds``: The number of seconds to wait after container startup before attempting the first readiness probe. *[Optional, default 5]*
155 2. ``periodSeconds``: The number of seconds between readiness probes. *[Optional, default 15]*
156 3. ``timeoutSeconds``: The number of seconds to wait for a connection to the container before timing out. *[Optional, default 1]*
157 4. ``probeType``: The type of readiness probe–``httpGet`` for an HTTP probe or ``exec`` for a script-based probe. *[Optional, default ``httpGet``]*
159 For HTTP(S) readiness probes, the following fields are *required*:
161 1. ``scheme``: ``HTTP`` or ``HTTPS``
162 2. ``path``: the path to the readiness probe API endpoint on the container
163 3. ``port``: the *container port* on which the microservice is listening for readiness probe requests.
164 (This is the *internal* port, not a NodePort or Ingress port.)
166 For script-based readiness probe, the following field is *required*:
167 1. ``command``: an array consisting of the command to be executed to run
168 the readiness probe and any command arguments that are needed.
170 Example (HTTP probe):
175 initialDelaySeconds: 5
181 Example (script-based probe):
187 initialDelaySeconds: 5
191 - /opt/ves-hv-collector/healthcheck.sh
193 Variables used by the DCAE services common templates
194 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
196 **applicationConfig:**
198 *[Optional]*. Initial configuration for
199 microservice. Pushed into Consul for retrieval by config-binding-service
200 and mapped to a file mounted at ``/app-config``. This is a YAML object
201 with keys and values as needed for the specific microservice. It will be
202 converted to JSON before being pushed to Consul or mounted as a file. If
203 not present, defaults to an empty object ({}).
205 *Note: Due to a bug in the Honolulu release (DCAEGEN2-2782), it is
206 necessary to supply an ``applicationConfig`` in the ``values.yaml`` for
207 a microservice even if the microservice does not have any configuration.
208 The workaround is to supply an empty configuration:*
212 applicationConfig: {}
214 *This is being fixed in the Istanbul release.*
218 Microservice-specific environment variables to be
219 set for the microservice’s container. Environment variables can be set
220 to literal string values or a value from a Kubernetes Secret that has
221 been set up using the ONAP OOM common secret template.
223 For a literal string value, use the environment variable name as the
224 key, and the desired string as the value:
229 EXAMPLE_ENV_VAR: "example variable content"
231 For a value taken from a secret, use the environment variable name as
232 the key and set the value to an object with the following fields:
234 1. ``secretUid``: *[Required]* The ``uid`` of the secret (set up with the
235 ONAP OOM common secret template) from which the value will be taken.
236 2. ``key``: *[Required]* The key within the secret that holds the desired value.
237 (A secret can contain several values, each with its own key. One frequently
238 used form of secrets contains login credentials, with keys for username
241 Example of an environment variable set from a secret:
247 secretUid: example-secret
250 The ``applicationEnv`` section of ``values.yaml`` can contain an
251 arbitrary number of environment variables and can contain both literal
252 values and values from secrets. ``applicationEnv`` is optional. If it is
253 not present in the ``values.yaml`` file, no microservice-specific
254 environment variables will be set for the microservice’s container.
256 Note that ``applicationEnv`` is a YAML object (or “dictionary”), not an
261 Controls microservice-specific volumes and volume
262 mounts. Allows a microservice to access an externally-created data
263 store. Currently only configMaps are supported. ``externalVolumes`` is a
264 YAML array of objects. Each object has three required fields and two
267 1. ``name``: *[Required]* The Kubernetes name of the configMap to be mounted.
268 The value is a case sensitive string. Because the names of configMaps are
269 sometimes set at deployment time (for instance, to prefix the Helm release to
270 the name), the string can be a Helm template fragment that will be expanded
272 2. ``type``: *[Required]* For now, this is always ``configMap``. This is a
273 case-insensitive string.
274 3. ``mountPath``: *[Required]* The path to the mount point for the volume
275 in the container file system. The value is a case-sensitive string.
276 4. ``readOnly``: *[Optional]* Boolean flag. Set to ``true`` to mount the volume
277 as read-only. Defaults to ``false``.
278 5. ``optional``: *[Optional]* Boolean flag. Set to ``true`` to make the
279 configMap optional (i.e., to allow the microservice’s pod to start even
280 if the configMap doesn’t exist). If set to ``false``, the configMap must
281 be present in order for the microservice’s pod to start. Defaults to
282 ``true``. *Note that this default is the opposite of the Kubernetes
283 default. We’ve done this to be consistent with the behavior of the DCAE
284 Cloudify plugin for Kubernetes (``k8splugin``), which always set
285 ``optional`` to ``true`` and did not allow for overriding this value.*
287 Example of an ``externalVolumes`` section:
292 - name: my-example-configmap
294 mountPath: /opt/app/config
295 - name: '{{ include "common.release" . }}-another-example'
297 mountPath: /opt/app/otherconfig
299 The dcaegen2-services-common deployment template will set up a volume
300 pointing to the specific configMap in the microservice’s pod and a
301 volume mount (mounted at ``mountPath`` on the microservice’s container.)
303 The ``externalVolumes`` section is optional. If it is not present, no
304 external volumes will be set up for the microservice.
308 Path to the directory in the microservice’s
309 container file system where TLS-certificate information from AAF should
310 be mounted. This is an optional field. When it is present, the
311 dcaegen2-services-common deployment template will set up an
312 initContainer that retrieves the certificate information into a shared
313 volume, which will then be mounted at the path specified by
320 certDirectory: /etc/ves-hv/ssl
324 Boolean flag. If set to ``true``, the
325 dcaegen2-services-common deployment will configure the initContainer
326 described above to fetch a server certificate for the microservice. If
327 set to ``false``, the initContainer will fetch only a CA certificate for
328 the AAF certificate authority. ``tlsServer`` is optional. The value
329 defaults to ``false``. ``tlsServer`` is ignored if ``certDirectory`` is
334 Path to the directory where the microservice writes
335 its log files. ``logDirectory`` is optional. If ``logDirectory`` is
336 present, the dcaegen2-services-common deployment template will deploy a
337 sidecar container that forwards the log file content to a log server.
343 logDirectory: /var/log/ONAP/dcae-hv-ves-collector
345 Note that ONAP is moving away from the sidecar approach and encouraging
346 applications (including DCAE microservices) to write log information to
347 ``stdout`` and ``stderr``.
351 If present, the dcaegen2-services-common deployment
352 template will deploy a sidecar container that polls the ONAP policy
353 subsystem for policy-driven configuration information.
355 ``policies`` is a YAML object (“dictionary”) that can contain the
358 1. ``policyID``: *[Optional]* A string representation of a JSON array of policy ID
359 values that the sidecar should monitor. Default ‘[]’.
360 2. ``filter``: *[Optional]* A string representation of a JSON array of regular
361 expressions that match policy IDs that the sidecar should monitory. Default ‘[]’.
362 3. ``duration``: *[Optional]* The interval (in seconds) between polling requests
363 made by the sidecar to the policy subsystem. Default: 2600.
371 '["onap.vfirewall.tca","abc"]'
373 '["DCAE.Config_vfirewall_.*"]'
376 **dcaePolicySyncImage:**
378 Name and tag of the policy sidecar image to be
379 used. Required if the policy sidecar is being used. The image repository
380 is set using the OOM common ``repositoryGenerator.repository`` template.
381 Normally this points to the ONAP image repository, but it can be
382 overridden on a global basis or a per-chart basis. See the OOM
383 documentation for more details.
389 dcaePolicySyncImage: onap/org.onap.dcaegen2.deployments.dcae-services-policy-sync:1.0.1
391 **consulLoaderImage:**
393 Name and tag of the consul loader image to be
394 used. Required. The consul loader image runs in an initContainer that
395 loads application configuration information into Consul. The image
396 repository is set using the OOM common
397 ``repositoryGenerator.repository`` template. Normally this points to the
398 ONAP image repository, but it can be overridden on a global basis or a
399 per-chart basis. See the OOM documentation for more details.
405 consulLoaderImage: onap/org.onap.dcaegen2.deployments.consul-loader-container:1.1.0
409 Name and tag of the TLS initialization image to be used.
410 Required if the microservice is configured to act as a TLS client and/or
411 server using AAF certificates. The TLS initialization image runs in an
412 initContainer and pulls TLS certificate information from AAF and stores
413 it in a volume on the microservice’s pod. The image repository is set
414 using the OOM common ``repositoryGenerator.repository`` template.
415 Normally this points to the ONAP image repository, but it can be
416 overridden on a global basis or a per-chart basis. See the OOM
417 documentation for more details.
423 tlsImage: onap/org.onap.dcaegen2.deployments.tls-init-container:2.1.0
425 **certProcessorImage:**
427 Name and tag of the CMPv2 certificate
428 initialization image to be used. Required if the microservice is
429 configured to act as a TLS client and/or server using CMPv2
430 certificates. This image runs in an initContainer and sets up trust
431 stores and keystores for CMPv2 use. The image repository is set using
432 the OOM common ``repositoryGenerator.repository`` template. Normally
433 this points to the ONAP image repository, but it can be overridden on a
434 global basis or a per-chart basis. See the OOM documentation for more
441 onap/org.onap.oom.platform.cert-service.oom-certservice-post-processor:2.1.0
444 Deploying multiple instances of a microservice
445 ----------------------------------------------
446 The dcaegen2-services-common charts can be used to deploy multiple instances of the same microservice. To do this successfully,
447 it's necessary to make sure that any Kubernetes service that the microservice exposes has different service names for each instance and,
448 if the service is exposed outside the Kubernetes cluster, a different external port assignment. This can be done by overriding the default
449 settings in the ``values.yaml`` file.
451 As an example, consider the DCAE VES collector (``dcae-ves-collector``). One instance of the VES collector is deployed by default when DCAE is installed using the ONAP installation
452 process. It exposes a service with the name ``dcae-ves-collector`` which is also exposed outside the Kubernetes cluster on NodePort 30417.
454 To deploy a second instance of the VES collector, we can create a YAML override file to define the service exposed by the second instance. The following
455 override file (``ves2.yaml``) will name the service as ``dcae-ves-collector-2`` and expose it on port 30499:
460 name: dcae-ves-collector-2
469 In the directory containing the ``dcae-ves-collector`` chart and the file ``ves.yaml``, running the following command will deploy a second instance
470 of the VES collector:
472 ``helm install -n onap --set global.masterPassword=whatever --set pullPolicy=IfNotPresent -f ves2.yaml ves2 .``
474 This creates a new Helm release called ``ves2``. The instance can be removed with:
476 ``helm delete -n onap ves2``
478 Note that if a component is using TLS with an AAF certificate, the DCAE certificate would need to include the new service name.
479 If a component is using an external certificate (CMPv2), the override file would need to supply the proper parameters to get a certificate with
480 correct common name/SAN.
482 Also note that if the chart for ``dcae-ves-collector`` has been pushed into a Helm repository, the ``helm install`` command can refer to the
483 repository (for instance, ``local/dcae-ves-collector``) instead of using the chart on the local filesystem.
486 Dynamic Topic and Feed Provisioning
487 -----------------------------------
488 This section introduces details on creation of dynamic Dmaap Topics in Message Router and Feeds in Data Router via DCAE helm charts.
490 Provisioning support through DCAE common-service template
491 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
493 When using DCAE common-service template in microservice chart ``deployment.yaml`` file it is required to include ``dcaegen2-services-common.microserviceDeployment`` template.
494 The dcaegen2-services-common include necessary ``common.dmaap.provisioning.initContainer`` template which provisions topics and feeds on Dmaap Message Router and Data Router.
496 Example : Including ``dcaegen2-services-common.microserviceDeployment`` template in ``deployment.yaml``.
500 {{ include "dcaegen2-services-common.microserviceDeployment" . }}
502 The ``common.dmaap.provisioning.initContainer`` template included in DCAE ``dcaegen2-services-common.microserviceDeployment`` makes use of
503 dmaap-bc client image to create Topics on Message Router and Feeds on Data Router microservice, with the help of ``dbc-client.sh`` script,
504 it uses Bus Controller REST API to create resources.
506 If the resource creation is successful via script, the response is logged in file with appropriate naming convention.
509 The configuration provided via ``values.yaml`` file, is consumed by ``common.dmaap.provisioning.initContainer`` template which runs two
510 init-container, First named init-dmaap-provisioning for creating resources on Dmaap, Second named init-merge-config which updates application config
511 with response generated as an outcome of operation by init-dmaap-provisioning container.
513 The figure below shows Dmaap Topics, Feeds Provisioning architecture via dcae-service-common helm charts.
516 The following diagram has been created on https://app.diagrams.net/. There is an editable version of the diagram
517 in repository under path docs/sections/images/dmaap_provisioning_architecture_diagram.drawio, import file to update diagram.
519 .. image:: images/dmaap_provisioning.png
521 Configuration to be added in ``values.yaml`` file.
524 For more information on attributes that are set in ``values.yaml`` for Data Router Feed, Publisher and Subscriber, Message Router Topic creation,
525 you can refer DMaaP Bus Controller API documentation at: https://docs.onap.org/projects/onap-dmaap-buscontroller/en/latest/apis/api.html
527 Dmaap Data Router Feeds creation input can be provided in below format. It consumes list of Feeds.
530 For DR Feed creation except ``feedName``, ``feedDescription``, ``feedVersion`` avoid update on other attributes.
531 All other attributes are mandatory, contains required default values.
536 - feedName: bulk_pm_feed
539 asprClassification: unclassified
540 feedDescription: DFC Feed Creation
542 Once the Feeds creation is successful we can attach Publisher and Subscriber to Feeds.
544 Dmaap Data Router Publisher config:
547 For DR Publisher creation except ``feedName`` avoid update on other attribute.
548 All other attributes are mandatory, contains required default values.
553 - feedName: bulk_pm_feed
554 dcaeLocationName: loc00
556 Dmaap Data Router Subscriber config:
559 For DR Subscriber creation except ``feedName`` avoid update on other attributes.
560 Attribute username, userpwd will be updated via init-merge-config init-container of ``common.dmaap.provisioning.initContainer`` template.
561 In case dcae-pm-mapper microservice is not the Subscriber, attribute deliveryURL need to be updated and privilegedSubscriber can be updated to False.
562 All other attributes are mandatory, contains required default values.
567 - feedName: bulk_pm_feed
569 username: ${DR_USERNAME}
570 userpwd: ${DR_PASSWORD}
571 dcaeLocationName: loc00
572 privilegedSubscriber: True
573 deliveryURL: https://dcae-pm-mapper:8443/delivery
575 Dmaap Message Router Topics creation input can be provided in below format. It consumes list of Topics.
576 Also we can attach Message Router Publisher and Subscriber at same time while creation of Topic.
579 For Message Router Topic creation except ``topicName`` and ``topicDescription`` avoid update on other attributes.
580 All other attributes are mandatory, contains required default values.
585 - topicName: PERFORMANCE_MEASUREMENTS
586 topicDescription: Description about Topic
590 - dcaeLocationName: san-francisco
591 clientRole: org.onap.dcae.pmPublisher
596 Volume configuration for configMap to be provided in ``values.yaml`` file.
602 path: /opt/app/config/feeds
604 path: /opt/app/config/dr_pubs
606 path: /opt/app/config/dr_subs
607 - name: topics-config
608 path: /opt/app/config/topics
611 For example directory containing ``dcae-datafile-collector``, ``dcae-pm-mapper`` chart under
612 dcaegen2-services in OOM repository we can find examples for Feed and Topic creation.
614 Provisioning support through DCAE When using custom deployment.yaml
615 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
617 When using custom ``deployment.yaml`` it is required explicitly to include ``common.dmaap.provisioning.initContainer`` template in
618 initContainer specs of ``deployment.yaml`` file.
620 Example : Including ``common.dmaap.provisioning.initContainer`` template in ``deployment.yaml`` file.
624 {{- include "common.dmaap.provisioning.initContainer" . | nindent XX }}
626 Note also need to take care of the ``Volumes`` that are required to be mounted on Application Pod in ``deployment.yaml``.
630 {{- include "common.dmaap.provisioning._volumes" . | nindent XX -}}
632 Configuration to be added in ``values.yaml`` file is similar to described in ``Provisioning support through DCAE common-service template``.
634 Removal of Data Router Feed, Publisher and Subscriber Or Message Router Topic
635 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
637 DCAE does not support automatic removal of Feed, Publisher and Subscriber from Data Router or Topic from Message Router at present.
638 So it is the responsibility of operator to manually remove Feeds and associated Publisher or Subscriber from Data Router and Topics
639 from Message Router after uninstalling microservice charts which created resources on installation.
641 Reference to DMAAP Bus Controller API documentation to figure out steps for manual removal of resources.
642 https://docs.onap.org/projects/onap-dmaap-buscontroller/en/latest/apis/api.html