1 .. This work is licensed under a Creative Commons Attribution 4.0
2 .. International License.
3 .. http://creativecommons.org/licenses/by/4.0
4 .. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung
5 .. Modification copyright (C) 2022 Nordix Foundation
9 .. _oom_dev_config_management:
12 Configuration Management
13 ########################
15 ONAP is a large system composed of many components - each of which are complex
16 systems in themselves - that needs to be deployed in a number of different
17 ways. For example, within a single operator's network there may be R&D
18 deployments under active development, pre-production versions undergoing system
19 testing and production systems that are operating live networks. Each of these
20 deployments will differ in significant ways, such as the version of the
21 software images deployed. In addition, there may be a number of application
22 specific configuration differences, such as operating system environment
23 variables. The following describes how the Helm configuration management
24 system is used within the OOM project to manage both ONAP infrastructure
25 configuration as well as ONAP components configuration.
27 One of the artifacts that OOM/Kubernetes uses to deploy ONAP components is the
28 deployment specification, yet another yaml file. Within these deployment specs
29 are a number of parameters as shown in the following example:
37 app.kubernetes.io/name: zookeeper
38 helm.sh/chart: zookeeper
39 app.kubernetes.io/component: server
40 app.kubernetes.io/managed-by: Tiller
41 app.kubernetes.io/instance: onap-oof
42 name: onap-oof-zookeeper
49 app.kubernetes.io/name: zookeeper
50 app.kubernetes.io/component: server
51 app.kubernetes.io/instance: onap-oof
52 serviceName: onap-oof-zookeeper-headless
56 app.kubernetes.io/name: zookeeper
57 helm.sh/chart: zookeeper
58 app.kubernetes.io/component: server
59 app.kubernetes.io/managed-by: Tiller
60 app.kubernetes.io/instance: onap-oof
67 image: gcr.io/google_samples/k8szk:v3
68 imagePullPolicy: Always
82 Note that within the statefulset specification, one of the container arguments
83 is the key/value pair image: gcr.io/google_samples/k8szk:v3 which
84 specifies the version of the zookeeper software to deploy. Although the
85 statefulset specifications greatly simplify statefulset, maintenance of the
86 statefulset specifications themselves become problematic as software versions
87 change over time or as different versions are required for different
88 statefulsets. For example, if the R&D team needs to deploy a newer version of
89 mariadb than what is currently used in the production environment, they would
90 need to clone the statefulset specification and change this value. Fortunately,
91 this problem has been solved with the templating capabilities of Helm.
93 The following example shows how the statefulset specifications are modified to
94 incorporate Helm templates such that key/value pairs can be defined outside of
95 the statefulset specifications and passed during instantiation of the component.
102 name: {{ include "common.fullname" . }}
103 namespace: {{ include "common.namespace" . }}
104 labels: {{- include "common.labels" . | nindent 4 }}
106 replicas: {{ .Values.replicaCount }}
108 matchLabels: {{- include "common.matchLabels" . | nindent 6 }}
109 # serviceName is only needed for StatefulSet
110 # put the postfix part only if you have add a postfix on the service name
111 serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }}
115 labels: {{- include "common.labels" . | nindent 8 }}
116 annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }}
117 name: {{ include "common.name" . }}
121 - name: {{ include "common.name" . }}
122 image: {{ .Values.image }}
123 imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
125 {{- range $index, $port := .Values.service.ports }}
126 - containerPort: {{ $port.port }}
127 name: {{ $port.name }}
129 {{- range $index, $port := .Values.service.headlessPorts }}
130 - containerPort: {{ $port.port }}
131 name: {{ $port.name }}
135 This version of the statefulset specification has gone through the process of
136 templating values that are likely to change between statefulsets. Note that the
137 image is now specified as: image: {{ .Values.image }} instead of a
138 string used previously. During the statefulset phase, Helm (actually the Helm
139 sub-component Tiller) substitutes the {{ .. }} entries with a variable defined
140 in a values.yaml file. The content of this file is as follows:
145 image: gcr.io/google_samples/k8szk:v3
150 Within the values.yaml file there is an image key with the value
151 `gcr.io/google_samples/k8szk:v3` which is the same value used in
152 the non-templated version. Once all of the substitutions are complete, the
153 resulting statefulset specification ready to be used by Kubernetes.
155 When creating a template consider the use of default values if appropriate.
156 Helm templating has built in support for DEFAULT values, here is
162 - name: "{{ .Values.nsPrefix | default "onap" }}-docker-registry-key"
164 The pipeline operator ("|") used here hints at that power of Helm templates in
165 that much like an operating system command line the pipeline operator allow
166 over 60 Helm functions to be embedded directly into the template (note that the
167 Helm template language is a superset of the Go template language). These
168 functions include simple string operations like upper and more complex flow
169 control operations like if/else.
171 OOM is mainly helm templating. In order to have consistent deployment of the
172 different components of ONAP, some rules must be followed.
174 Templates are provided in order to create Kubernetes resources (Secrets,
175 Ingress, Services, ...) or part of Kubernetes resources (names, labels,
176 resources requests and limits, ...).
178 a full list and simple description is done in
179 `kubernetes/common/common/documentation.rst`.
184 In order to create a Service for a component, you have to create a file (with
185 `service` in the name.
186 For normal service, just put the following line:
190 {{ include "common.service" . }}
192 For headless service, the line to put is the following:
196 {{ include "common.headlessService" . }}
198 The configuration of the service is done in component `values.yaml`:
203 name: NAME-OF-THE-SERVICE
207 someAnnotationsKey: value
219 `annotations` and `postfix` keys are optional.
220 if `service.type` is `NodePort`, then you have to give `nodePort` value for your
221 service ports (which is the end of the computed nodePort, see example).
223 It would render the following Service Resource (for a component named
224 `name-of-my-component`, with version `x.y.z`, helm deployment name
225 `my-deployment` and `global.nodePortPrefix` `302`):
233 someAnnotationsKey: value
234 name: NAME-OF-THE-SERVICE-MY-POSTFIX
236 app.kubernetes.io/name: name-of-my-component
237 helm.sh/chart: name-of-my-component-x.y.z
238 app.kubernetes.io/instance: my-deployment-name-of-my-component
239 app.kubernetes.io/managed-by: Tiller
243 targetPort: tcp-MyPort
249 targetPort: https-api
252 app.kubernetes.io/name: name-of-my-component
253 app.kubernetes.io/instance: my-deployment-name-of-my-component
256 In the deployment or statefulSet file, you needs to set the good labels in
257 order for the service to match the pods.
259 here's an example to be sure it matches (for a statefulSet):
266 name: {{ include "common.fullname" . }}
267 namespace: {{ include "common.namespace" . }}
268 labels: {{- include "common.labels" . | nindent 4 }}
271 matchLabels: {{- include "common.matchLabels" . | nindent 6 }}
272 # serviceName is only needed for StatefulSet
273 # put the postfix part only if you have add a postfix on the service name
274 serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }}
278 labels: {{- include "common.labels" . | nindent 8 }}
279 annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }}
280 name: {{ include "common.name" . }}
284 - name: {{ include "common.name" . }}
286 {{- range $index, $port := .Values.service.ports }}
287 - containerPort: {{ $port.port }}
288 name: {{ $port.name }}
290 {{- range $index, $port := .Values.service.headlessPorts }}
291 - containerPort: {{ $port.port }}
292 name: {{ $port.name }}
296 The configuration of the service is done in component `values.yaml`:
301 name: NAME-OF-THE-SERVICE
305 anotherAnnotationsKey : value
306 publishNotReadyAddresses: true
315 `headless.annotations`, `headless.postfix` and
316 `headless.publishNotReadyAddresses` keys are optional.
318 If `headless.postfix` is not set, then we'll add `-headless` at the end of the
321 If it set to `NONE`, there will be not postfix.
323 And if set to something, it will add `-something` at the end of the service
326 It would render the following Service Resource (for a component named
327 `name-of-my-component`, with version `x.y.z`, helm deployment name
328 `my-deployment` and `global.nodePortPrefix` `302`):
336 anotherAnnotationsKey: value
337 name: NAME-OF-THE-SERVICE
339 app.kubernetes.io/name: name-of-my-component
340 helm.sh/chart: name-of-my-component-x.y.z
341 app.kubernetes.io/instance: my-deployment-name-of-my-component
342 app.kubernetes.io/managed-by: Tiller
347 targetPort: tcp-MyPort
353 targetPort: https-api
355 publishNotReadyAddresses: true
357 app.kubernetes.io/name: name-of-my-component
358 app.kubernetes.io/instance: my-deployment-name-of-my-component
361 Previous example of StatefulSet would also match (except for the `postfix` part
364 Creating Deployment or StatefulSet
365 ----------------------------------
367 Deployment and StatefulSet should use the `apps/v1` (which has appeared in
369 As seen on the service part, the following parts are mandatory:
376 name: {{ include "common.fullname" . }}
377 namespace: {{ include "common.namespace" . }}
378 labels: {{- include "common.labels" . | nindent 4 }}
381 matchLabels: {{- include "common.matchLabels" . | nindent 6 }}
382 # serviceName is only needed for StatefulSet
383 # put the postfix part only if you have add a postfix on the service name
384 serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }}
388 labels: {{- include "common.labels" . | nindent 8 }}
389 annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }}
390 name: {{ include "common.name" . }}
394 - name: {{ include "common.name" . }}
396 Dependency Management
397 ---------------------
398 These Helm charts describe the desired state
399 of an ONAP deployment and instruct the Kubernetes container manager as to how
400 to maintain the deployment in this state. These dependencies dictate the order
401 in-which the containers are started for the first time such that such
402 dependencies are always met without arbitrary sleep times between container
403 startups. For example, the SDC back-end container requires the Elastic-Search,
404 Cassandra and Kibana containers within SDC to be ready and is also dependent on
405 DMaaP (or the message-router) to be ready - where ready implies the built-in
406 "readiness" probes succeeded - before becoming fully operational. When an
407 initial deployment of ONAP is requested the current state of the system is NULL
408 so ONAP is deployed by the Kubernetes manager as a set of Docker containers on
409 one or more predetermined hosts. The hosts could be physical machines or
410 virtual machines. When deploying on virtual machines the resulting system will
411 be very similar to "Heat" based deployments, i.e. Docker containers running
412 within a set of VMs, the primary difference being that the allocation of
413 containers to VMs is done dynamically with OOM and statically with "Heat".
414 Example SO deployment descriptor file shows SO's dependency on its mariadb
417 SO deployment specification excerpt:
424 name: {{ include "common.fullname" . }}
425 namespace: {{ include "common.namespace" . }}
426 labels: {{- include "common.labels" . | nindent 4 }}
428 replicas: {{ .Values.replicaCount }}
430 matchLabels: {{- include "common.matchLabels" . | nindent 6 }}
434 app: {{ include "common.name" . }}
435 release: {{ .Release.Name }}