X-Git-Url: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blobdiff_plain;f=docs%2Foom_developer_guide.rst;h=8df8e74acd3e9e7f88d8a02dd8d1470f81ac5409;hp=a2ccc1ba60244c2cba7476470c9efa1b1c2b049b;hb=1660a8c5518c45b8a424b15b8c63b71ae0b39c69;hpb=fd737d1ad0b44eeceab5629928805ca2407d1921 diff --git a/docs/oom_developer_guide.rst b/docs/oom_developer_guide.rst index a2ccc1ba60..8df8e74acd 100644 --- a/docs/oom_developer_guide.rst +++ b/docs/oom_developer_guide.rst @@ -1,13 +1,14 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. This work is licensed under a Creative Commons Attribution 4.0 +.. International License. .. http://creativecommons.org/licenses/by/4.0 -.. Copyright 2018 Amdocs, Bell Canada +.. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung .. Links .. _Helm: https://docs.helm.sh/ .. _Helm Charts: https://github.com/kubernetes/charts .. _Kubernetes: https://Kubernetes.io/ .. _Docker: https://www.docker.com/ -.. _Nexus: https://nexus.onap.org/#welcome +.. _Nexus: https://nexus.onap.org/ .. _AWS Elastic Block Store: https://aws.amazon.com/ebs/ .. _Azure File: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction .. _GCE Persistent Disk: https://cloud.google.com/compute/docs/disks/ @@ -21,7 +22,7 @@ OOM Developer Guide ################### -.. figure:: oomLogoV2-medium.png +.. figure:: images/oom_logo/oomLogoV2-medium.png :align: right ONAP consists of a large number of components, each of which are substantial @@ -79,159 +80,122 @@ have been created following the guidelines provided. The top level of the ONAP charts is shown below: -.. graphviz:: +.. code-block:: bash + + common + ├── cassandra + │   ├── Chart.yaml + │   ├── resources + │   │   ├── config + │   │   │   └── docker-entrypoint.sh + │   │   ├── exec.py + │   │   └── restore.sh + │   ├── templates + │   │   ├── backup + │   │   │   ├── configmap.yaml + │   │   │   ├── cronjob.yaml + │   │   │   ├── pv.yaml + │   │   │   └── pvc.yaml + │   │   ├── configmap.yaml + │   │   ├── pv.yaml + │   │   ├── service.yaml + │   │   └── statefulset.yaml + │   └── values.yaml + ├── common + │   ├── Chart.yaml + │   ├── templates + │   │   ├── _createPassword.tpl + │   │   ├── _ingress.tpl + │   │   ├── _labels.tpl + │   │   ├── _mariadb.tpl + │   │   ├── _name.tpl + │   │   ├── _namespace.tpl + │   │   ├── _repository.tpl + │   │   ├── _resources.tpl + │   │   ├── _secret.yaml + │   │   ├── _service.tpl + │   │   ├── _storage.tpl + │   │   └── _tplValue.tpl + │   └── values.yaml + ├── ... + └── postgres-legacy +    ├── Chart.yaml + ├── charts + └── configs - digraph onap_top_chart { - rankdir="LR"; - { - node [shape=folder] - oValues [label="values.yaml"] - oChart [label="Chart.yaml"] - dev [label="dev.yaml"] - prod [label="prod.yaml"] - crb [label="clusterrolebindings.yaml"] - secrets [label="secrets.yaml"] - } - { - node [style=dashed] - vCom [label="component"] - } +The common section of charts consists of a set of templates that assist with +parameter substitution (`_name.tpl`, `_namespace.tpl` and others) and a set of +charts for components used throughout ONAP. When the common components are used +by other charts they are instantiated each time or we can deploy a shared +instances for several components. - onap -> oValues - onap -> oChart - onap -> templates - onap -> resources - oValues -> vCom - resources -> environments - environments -> dev - environments -> prod - templates -> crb - templates -> secrets - } +All of the ONAP components have charts that follow the pattern shown below: -Within the `values.yaml` file at the `onap` level, one will find a set of -boolean values that control which of the ONAP components get deployed as shown -below: +.. code-block:: bash + + name-of-my-component + ├── Chart.yaml + ├── component + │   └── subcomponent-folder + ├── charts + │   └── subchart-folder + ├── resources + │   ├── folder1 + │   │   ├── file1 + │   │   └── file2 + │   └── folder1 + │   ├── file3 + │   └── folder3 + │      └── file4 + ├── templates + │   ├── NOTES.txt + │   ├── configmap.yaml + │   ├── deployment.yaml + │   ├── ingress.yaml + │   ├── job.yaml + │   ├── secrets.yaml + │   └── service.yaml + └── values.yaml + +Note that the component charts / components may include a hierarchy of sub +components and in themselves can be quite complex. + +You can use either `charts` or `components` folder for your subcomponents. +`charts` folder means that the subcomponent will always been deployed. + +`components` folders means we can choose if we want to deploy the +subcomponent. + +This choice is done in root `values.yaml`: .. code-block:: yaml - aaf: # Application Authorization Framework - enabled: false - <...> - so: # Service Orchestrator - enabled: true - -By setting these flags a custom deployment can be created and used during -deployment by using the `-f` Helm option as follows:: - - > helm install local/onap -name development -f dev.yaml - -Note that there are one or more example deployment files in the -`onap/resources/environments/` directory. It is best practice to create a -unique deployment file for each environment used to ensure consistent -behaviour. - -To aid in the long term supportability of ONAP, a set of common charts have -been created (and will be expanded in subsequent releases of ONAP) that can be -used by any of the ONAP components by including the common component in its -`requirements.yaml` file. The common components are arranged as follows: + --- + global: + key: value -.. graphviz:: - - digraph onap_common_chart { - rankdir="LR"; - { - node [shape=folder] - mValues [label="values.yaml"] - ccValues [label="values.yaml"] - comValues [label="values.yaml"] - comChart [label="Chart.yaml"] - ccChart [label="Chart.yaml"] - mChart [label="Chart.yaml"] - - mReq [label="requirements.yaml"] - mService [label="service.yaml"] - mMap [label="configmap.yaml"] - ccName [label="_name.tpl"] - ccNS [label="_namespace.tpl"] - } - { - cCom [label="common"] - mTemp [label="templates"] - ccTemp [label="templates"] - } - { - more [label="...",style=dashed] - } - - common -> comValues - common -> comChart - common -> cCom - common -> mysql - common -> more - - cCom -> ccChart - cCom -> ccValues - cCom -> ccTemp - ccTemp -> ccName - ccTemp -> ccNS - - mysql -> mValues - mysql -> mChart - mysql -> mReq - mysql -> mTemp - mTemp -> mService - mTemp -> mMap - } - -The common section of charts consists of a set of templates that assist with -parameter substitution (`_name.tpl` and `_namespace.tpl`) and a set of charts -for components used throughout ONAP. Initially `mysql` is in the common area -but this will expand to include other databases like `mariadb-galera`, -`postgres`, and `cassandra`. Other candidates for common components include -`redis` and`kafka`. When the common components are used by other charts they -are instantiated each time. In subsequent ONAP releases some of the common -components could be a setup as services that are used by multiple ONAP -components thus minimizing the deployment and operational costs. - -All of the ONAP components have charts that follow the pattern shown below: - -.. graphviz:: - - digraph onap_component_chart { - rankdir="LR"; - { - node [shape=folder] - cValues [label="values.yaml"] - cChart [label="Chart.yaml"] - cService [label="service.yaml"] - cMap [label="configmap.yaml"] - cFiles [label="config file(s)"] - } - { - cCharts [label="charts"] - cTemp [label="templates"] - cRes [label="resources"] + component1: + enabled: true + component2: + enabled: true - } - { - sCom [label="component",style=dashed] - } +Then in `Chart.yaml` dependencies section, you'll use these values: - component -> cValues - component -> cChart - component -> cCharts - component -> cTemp - component -> cRes - cTemp -> cService - cTemp -> cMap - cRes -> config - config -> cFiles - cCharts -> sCom - } +.. code-block:: yaml -Note that the component charts may include a hierarchy of components and in -themselves can be quite complex. + --- + dependencies: + - name: common + version: ~x.y-0 + repository: '@local' + - name: component1 + version: ~x.y-0 + repository: 'file://components/component1' + condition: component1.enabled + - name: component2 + version: ~x.y-0 + repository: 'file://components/component2' + condition: component2.enabled Configuration of the components varies somewhat from component to component but generally follows the pattern of one or more `configmap.yaml` files which can @@ -260,126 +224,134 @@ configuration as well as ONAP components configuration. One of the artifacts that OOM/Kubernetes uses to deploy ONAP components is the deployment specification, yet another yaml file. Within these deployment specs -are a number of parameters as shown in the following mariadb example: +are a number of parameters as shown in the following example: .. code-block:: yaml - apiVersion: extensions/v1beta1 - kind: Deployment + apiVersion: apps/v1 + kind: StatefulSet metadata: - name: mariadb + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof + name: onap-oof-zookeeper + namespace: onap spec: - <...> + <...> + replicas: 3 + selector: + matchLabels: + app.kubernetes.io/name: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/instance: onap-oof + serviceName: onap-oof-zookeeper-headless template: - <...> + metadata: + labels: + app.kubernetes.io/name: zookeeper + helm.sh/chart: zookeeper + app.kubernetes.io/component: server + app.kubernetes.io/managed-by: Tiller + app.kubernetes.io/instance: onap-oof spec: - hostname: mariadb + <...> + affinity: containers: - - args: - image: nexus3.onap.org:10001/mariadb:10.1.11 - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" + - name: zookeeper <...> - imagePullSecrets: - - name: onap-docker-registry-key - -Note that within the deployment specification, one of the container arguments -is the key/value pair image: nexus3.onap.org:10001/mariadb:10.1.11 which -specifies the version of the mariadb software to deploy. Although the -deployment specifications greatly simplify deployment, maintenance of the -deployment specifications themselves become problematic as software versions + image: gcr.io/google_samples/k8szk:v3 + imagePullPolicy: Always + <...> + ports: + - containerPort: 2181 + name: client + protocol: TCP + - containerPort: 3888 + name: election + protocol: TCP + - containerPort: 2888 + name: server + protocol: TCP + <...> + +Note that within the statefulset specification, one of the container arguments +is the key/value pair image: gcr.io/google_samples/k8szk:v3 which +specifies the version of the zookeeper software to deploy. Although the +statefulset specifications greatly simplify statefulset, maintenance of the +statefulset specifications themselves become problematic as software versions change over time or as different versions are required for different -deployments. For example, if the R&D team needs to deploy a newer version of +statefulsets. For example, if the R&D team needs to deploy a newer version of mariadb than what is currently used in the production environment, they would -need to clone the deployment specification and change this value. Fortunately, +need to clone the statefulset specification and change this value. Fortunately, this problem has been solved with the templating capabilities of Helm. -The following example shows how the deployment specifications are modified to +The following example shows how the statefulset specifications are modified to incorporate Helm templates such that key/value pairs can be defined outside of -the deployment specifications and passed during instantiation of the component. +the statefulset specifications and passed during instantiation of the component. .. code-block:: yaml - apiVersion: extensions/v1beta1 - kind: Deployment + apiVersion: apps/v1 + kind: StatefulSet metadata: - name: mariadb - namespace: "{{ .Values.nsPrefix }}-mso" + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} <...> template: - <...> + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} spec: - hostname: mariadb - containers: - - args: - image: {{ .Values.image.mariadb }} - imagePullPolicy: {{ .Values.pullPolicy }} - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" <...> - imagePullSecrets: - - name: "{{ .Values.nsPrefix }}-docker-registry-key"apiVersion: extensions/v1beta1 - kind: Deployment - metadata: - name: mariadb - namespace: "{{ .Values.nsPrefix }}-mso" - spec: - <...> - template: - <...> - spec: - hostname: mariadb containers: - - args: - image: {{ .Values.image.mariadb }} - imagePullPolicy: {{ .Values.pullPolicy }} - name: "mariadb" - env: - - name: MYSQL_ROOT_PASSWORD - value: password - - name: MARIADB_MAJOR - value: "10.1" - <...> - imagePullSecrets: - - name: "{{ .Values.nsPrefix }}-docker-registry-key" - -This version of the deployment specification has gone through the process of -templating values that are likely to change between deployments. Note that the -image is now specified as: image: {{ .Values.image.mariadb }} instead of a -string used previously. During the deployment phase, Helm (actually the Helm + - name: {{ include "common.name" . }} + image: {{ .Values.image }} + imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +This version of the statefulset specification has gone through the process of +templating values that are likely to change between statefulsets. Note that the +image is now specified as: image: {{ .Values.image }} instead of a +string used previously. During the statefulset phase, Helm (actually the Helm sub-component Tiller) substitutes the {{ .. }} entries with a variable defined in a values.yaml file. The content of this file is as follows: .. code-block:: yaml - nsPrefix: onap - pullPolicy: IfNotPresent - image: - readiness: oomk8s/readiness-check:2.0.0 - mso: nexus3.onap.org:10001/openecomp/mso:1.0-STAGING-latest - mariadb: nexus3.onap.org:10001/mariadb:10.1.11 + <...> + image: gcr.io/google_samples/k8szk:v3 + replicaCount: 3 + <...> -Within the values.yaml file there is an image section with the key/value pair -mariadb: nexus3.onap.org:10001/mariadb:10.1.11 which is the same value used in -the non-templated version. Once all of the substitutions are complete, the -resulting deployment specification ready to be used by Kubernetes. -Also note that in this example, the namespace key/value pair is specified in -the values.yaml file. This key/value pair will be global across the entire -ONAP deployment and is therefore a prime example of where configuration -hierarchy can be very useful. +Within the values.yaml file there is an image key with the value +`gcr.io/google_samples/k8szk:v3` which is the same value used in +the non-templated version. Once all of the substitutions are complete, the +resulting statefulset specification ready to be used by Kubernetes. -When creating a deployment template consider the use of default values if -appropriate. Helm templating has built in support for DEFAULT values, here is +When creating a template consider the use of default values if appropriate. +Helm templating has built in support for DEFAULT values, here is an example: .. code-block:: yaml @@ -394,6 +366,230 @@ Helm template language is a superset of the Go template language). These functions include simple string operations like upper and more complex flow control operations like if/else. +OOM is mainly helm templating. In order to have consistent deployment of the +different components of ONAP, some rules must be followed. + +Templates are provided in order to create Kubernetes resources (Secrets, +Ingress, Services, ...) or part of Kubernetes resources (names, labels, +resources requests and limits, ...). + +a full list and simple description is done in +`kubernetes/common/common/documentation.rst`. + +Service template +---------------- + +In order to create a Service for a component, you have to create a file (with +`service` in the name. +For normal service, just put the following line: + +.. code-block:: yaml + + {{ include "common.service" . }} + +For headless service, the line to put is the following: + +.. code-block:: yaml + + {{ include "common.headlessService" . }} + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + postfix: MY-POSTFIX + type: NodePort + annotations: + someAnnotationsKey: value + ports: + - name: tcp-MyPort + port: 5432 + nodePort: 88 + - name: http-api + port: 8080 + nodePort: 89 + - name: https-api + port: 9443 + nodePort: 90 + +`annotations` and `postfix` keys are optional. +if `service.type` is `NodePort`, then you have to give `nodePort` value for your +service ports (which is the end of the computed nodePort, see example). + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + someAnnotationsKey: value + name: NAME-OF-THE-SERVICE-MY-POSTFIX + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: NodePort + +In the deployment or statefulSet file, you needs to set the good labels in +order for the service to match the pods. + +here's an example to be sure it matches (for a statefulSet): + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} + ports: + {{- range $index, $port := .Values.service.ports }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + {{- range $index, $port := .Values.service.headlessPorts }} + - containerPort: {{ $port.port }} + name: {{ $port.name }} + {{- end }} + <...> + +The configuration of the service is done in component `values.yaml`: + +.. code-block:: yaml + + service: + name: NAME-OF-THE-SERVICE + headless: + postfix: NONE + annotations: + anotherAnnotationsKey : value + publishNotReadyAddresses: true + headlessPorts: + - name: tcp-MyPort + port: 5432 + - name: http-api + port: 8080 + - name: https-api + port: 9443 + +`headless.annotations`, `headless.postfix` and +`headless.publishNotReadyAddresses` keys are optional. + +If `headless.postfix` is not set, then we'll add `-headless` at the end of the +service name. + +If it set to `NONE`, there will be not postfix. + +And if set to something, it will add `-something` at the end of the service +name. + +It would render the following Service Resource (for a component named +`name-of-my-component`, with version `x.y.z`, helm deployment name +`my-deployment` and `global.nodePortPrefix` `302`): + +.. code-block:: yaml + + apiVersion: v1 + kind: Service + metadata: + annotations: + anotherAnnotationsKey: value + name: NAME-OF-THE-SERVICE + labels: + app.kubernetes.io/name: name-of-my-component + helm.sh/chart: name-of-my-component-x.y.z + app.kubernetes.io/instance: my-deployment-name-of-my-component + app.kubernetes.io/managed-by: Tiller + spec: + clusterIP: None + ports: + - port: 5432 + targetPort: tcp-MyPort + nodePort: 30288 + - port: 8080 + targetPort: http-api + nodePort: 30289 + - port: 9443 + targetPort: https-api + nodePort: 30290 + publishNotReadyAddresses: true + selector: + app.kubernetes.io/name: name-of-my-component + app.kubernetes.io/instance: my-deployment-name-of-my-component + type: ClusterIP + +Previous example of StatefulSet would also match (except for the `postfix` part +obviously). + +Creating Deployment or StatefulSet +---------------------------------- + +Deployment and StatefulSet should use the `apps/v1` (which has appeared in +v1.9). +As seen on the service part, the following parts are mandatory: + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: {{ include "common.fullname" . }} + namespace: {{ include "common.namespace" . }} + labels: {{- include "common.labels" . | nindent 4 }} + spec: + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} + # serviceName is only needed for StatefulSet + # put the postfix part only if you have add a postfix on the service name + serviceName: {{ include "common.servicename" . }}-{{ .Values.service.postfix }} + <...> + template: + metadata: + labels: {{- include "common.labels" . | nindent 8 }} + annotations: {{- include "common.tplValue" (dict "value" .Values.podAnnotations "context" $) | nindent 8 }} + name: {{ include "common.name" . }} + spec: + <...> + containers: + - name: {{ include "common.name" . }} ONAP Application Configuration ------------------------------ @@ -423,18 +619,16 @@ SO deployment specification excerpt: .. code-block:: yaml - apiVersion: extensions/v1beta1 + apiVersion: apps/v1 kind: Deployment metadata: - name: {{ include "common.name" . }} + name: {{ include "common.fullname" . }} namespace: {{ include "common.namespace" . }} - labels: - app: {{ include "common.name" . }} - chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} - release: {{ .Release.Name }} - heritage: {{ .Release.Service }} + labels: {{- include "common.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {{- include "common.matchLabels" . | nindent 6 }} template: metadata: labels: @@ -443,7 +637,7 @@ SO deployment specification excerpt: spec: initContainers: - command: - - /root/ready.py + - /app/ready.py args: - --container-name - so-mariadb @@ -493,7 +687,7 @@ The relationship between these objects is shown in the following figure: .. database PV .. @enduml -.. figure:: kubernetes_objects.png +.. figure:: images/k8s/kubernetes_objects.png OOM uses these Kubernetes objects as described in the following sections. @@ -816,7 +1010,7 @@ MSB service discovery. The following is a brief description of how this integration will be done: A registrator to push the service endpoint info to MSB service -discovery.  +discovery. - The needed service endpoint info is put into the kubernetes yaml file as annotation, including service name, Protocol,version, visual @@ -880,7 +1074,7 @@ MSB Integration with OOM ------------------------ A preliminary view of the OOM-MSB integration is as follows: -.. figure:: MSB-OOM-Diagram.png +.. figure:: images/msb/MSB-OOM-Diagram.png A message sequence chart of the registration process: @@ -935,7 +1129,7 @@ access to watch the kubernetes events and get service annotation by Kubernetes APIs. The token can be found in the kubectl configuration file *~/.kube/config* -More details can be found here `MSB installation `__. +More details can be found here `MSB installation `_. .. MISC .. ====