1 .. This work is licensed under a Creative Commons Attribution 4.0
2 .. International License.
3 .. http://creativecommons.org/licenses/by/4.0
4 .. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung
7 .. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements
8 .. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
9 .. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
10 .. _Docker DevOps: https://wiki.onap.org/display/DW/Docker+DevOps#DockerDevOps-DockerBuild
11 .. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug
12 .. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474
13 .. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md
15 .. figure:: oomLogoV2-medium.png
18 .. _onap-on-kubernetes-with-rancher:
20 ONAP on HA Kubernetes Cluster
21 #############################
23 This guide provides instructions on how to setup a Highly-Available Kubernetes
24 Cluster. For this, we are hosting our cluster on OpenStack VMs and using the
25 Rancher Kubernetes Engine (RKE) to deploy and manage our Kubernetes Cluster.
32 The result at the end of this tutorial will be:
34 #. Creation of a Key Pair to use with Open Stack and RKE
36 #. Creation of OpenStack VMs to host Kubernetes Control Plane
38 #. Creation of OpenStack VMs to host Kubernetes Workers
40 #. Installation and configuration of RKE to setup an HA Kubernetes
42 #. Installation and configuration of kubectl
44 #. Installation and configuration of Helm
46 #. Creation of an NFS Server to be used by ONAP as shared persistance
48 There are many ways one can execute the above steps. Including automation
49 through the use of HEAT to setup the OpenStack VMs. To better illustrate the
50 steps involved, we have captured the manual creation of such an environment
51 using the ONAP Wind River Open Lab.
55 A Key Pair is required to access the created OpenStack VMs and will be used by
56 RKE to configure the VMs for Kubernetes.
58 Use an existing key pair, import one or create a new one to assign.
60 .. image:: images/keys/key_pair_1.png
63 If you're creating a new Key Pair, ensure to create a local copy of the
64 Private Key through the use of "Copy Private Key to Clipboard".
66 For the purpose of this guide, we will assume a new local key called "onap-key"
67 has been downloaded and is copied into **~/.ssh/**, from which it can be
74 > chmod 600 ~/.ssh/onap-key
80 An internal network is required in order to deploy our VMs that will host
83 .. image:: images/network/network_1.png
85 .. image:: images/network/network_2.png
87 .. image:: images/network/network_3.png
90 It's better to have one network per deployment and obviously the name of this
91 network should be unique.
93 Now we need to create a router to attach this network to outside:
95 .. image:: images/network/network_4.png
100 A specific security group is also required
102 .. image:: images/sg/sg_1.png
104 then click on `manage rules` of the newly created security group.
105 And finally click on `Add Rule` and create the following one:
107 .. image:: images/sg/sg_2.png
110 the security is clearly not good here and the right SG will be proposed in a
113 Create Kubernetes Control Plane VMs
114 ===================================
116 The following instructions describe how to create 3 OpenStack VMs to host the
117 Highly-Available Kubernetes Control Plane.
118 ONAP workloads will not be scheduled on these Control Plane nodes.
120 Launch new VM instances
121 -----------------------
123 .. image:: images/cp_vms/control_plane_1.png
125 Select Ubuntu 18.04 as base image
126 ---------------------------------
127 Select "No" for "Create New Volume"
129 .. image:: images/cp_vms/control_plane_2.png
133 The recommended flavor is at least 4 vCPU and 8GB ram.
135 .. image:: images/cp_vms/control_plane_3.png
140 Use the created network:
142 .. image:: images/cp_vms/control_plane_4.png
147 Use the created security group:
149 .. image:: images/cp_vms/control_plane_5.png
153 Assign the key pair that was created/selected previously (e.g. onap_key).
155 .. image:: images/cp_vms/control_plane_6.png
157 Apply customization script for Control Plane VMs
158 ------------------------------------------------
160 Click :download:`openstack-k8s-controlnode.sh <openstack-k8s-controlnode.sh>`
161 to download the script.
163 .. literalinclude:: openstack-k8s-controlnode.sh
166 This customization script will:
171 .. image:: images/cp_vms/control_plane_7.png
176 .. image:: images/cp_vms/control_plane_8.png
180 Create Kubernetes Worker VMs
181 ============================
182 The following instructions describe how to create OpenStack VMs to host the
183 Highly-Available Kubernetes Workers. ONAP workloads will only be scheduled on
186 Launch new VM instances
187 -----------------------
189 The number and size of Worker VMs is dependent on the size of the ONAP
190 deployment. By default, all ONAP applications are deployed. It's possible to
191 customize the deployment and enable a subset of the ONAP applications. For the
192 purpose of this guide, however, we will deploy 12 Kubernetes Workers that have
193 been sized to handle the entire ONAP application workload.
195 .. image:: images/wk_vms/worker_1.png
197 Select Ubuntu 18.04 as base image
198 ---------------------------------
199 Select "No" on "Create New Volume"
201 .. image:: images/wk_vms/worker_2.png
205 The size of Kubernetes hosts depend on the size of the ONAP deployment
208 If a small subset of ONAP applications are being deployed
209 (i.e. for testing purposes), then 16GB or 32GB may be sufficient.
211 .. image:: images/wk_vms/worker_3.png
216 .. image:: images/wk_vms/worker_4.png
221 .. image:: images/wk_vms/worker_5.png
225 Assign the key pair that was created/selected previously (e.g. onap_key).
227 .. image:: images/wk_vms/worker_6.png
229 Apply customization script for Kubernetes VM(s)
230 -----------------------------------------------
232 Click :download:`openstack-k8s-workernode.sh <openstack-k8s-workernode.sh>` to
235 .. literalinclude:: openstack-k8s-workernode.sh
238 This customization script will:
248 .. image:: images/wk_vms/worker_7.png
253 Assign Floating IP addresses
254 ----------------------------
255 Assign Floating IPs to all Control Plane and Worker VMs.
256 These addresses provide external access to the VMs and will be used by RKE
257 to configure kubernetes on to the VMs.
259 Repeat the following for each VM previously created:
261 .. image:: images/floating_ips/floating_1.png
263 Resulting floating IP assignments in this example.
265 .. image:: images/floating_ips/floating_2.png
270 Configure Rancher Kubernetes Engine (RKE)
271 =========================================
275 Download and install RKE on a VM, desktop or laptop.
276 Binaries can be found here for Linux and Mac: https://github.com/rancher/rke/releases/tag/v1.0.6
278 RKE requires a *cluster.yml* as input. An example file is show below that
279 describes a Kubernetes cluster that will be mapped onto the OpenStack VMs
280 created earlier in this guide.
282 Click :download:`cluster.yml <cluster.yml>` to download the
285 .. literalinclude:: cluster.yml
290 Before this configuration file can be used the external **address**
291 and the **internal_address** must be mapped for each control and worker node
296 From within the same directory as the cluster.yml file, simply execute::
300 The output will look something like::
302 INFO[0000] Initiating Kubernetes cluster
303 INFO[0000] [certificates] Generating admin certificates and kubeconfig
304 INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
305 INFO[0000] Building Kubernetes cluster
306 INFO[0000] [dialer] Setup tunnel for host [10.12.6.82]
307 INFO[0000] [dialer] Setup tunnel for host [10.12.6.249]
308 INFO[0000] [dialer] Setup tunnel for host [10.12.6.74]
309 INFO[0000] [dialer] Setup tunnel for host [10.12.6.85]
310 INFO[0000] [dialer] Setup tunnel for host [10.12.6.238]
311 INFO[0000] [dialer] Setup tunnel for host [10.12.6.89]
312 INFO[0000] [dialer] Setup tunnel for host [10.12.5.11]
313 INFO[0000] [dialer] Setup tunnel for host [10.12.6.90]
314 INFO[0000] [dialer] Setup tunnel for host [10.12.6.244]
315 INFO[0000] [dialer] Setup tunnel for host [10.12.5.165]
316 INFO[0000] [dialer] Setup tunnel for host [10.12.6.126]
317 INFO[0000] [dialer] Setup tunnel for host [10.12.6.111]
318 INFO[0000] [dialer] Setup tunnel for host [10.12.5.160]
319 INFO[0000] [dialer] Setup tunnel for host [10.12.5.191]
320 INFO[0000] [dialer] Setup tunnel for host [10.12.6.195]
321 INFO[0002] [network] Deploying port listener containers
322 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.85]
323 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.89]
324 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.90]
325 INFO[0011] [network] Successfully pulled image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.89]
327 INFO[0309] [addons] Setting up Metrics Server
328 INFO[0309] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
329 INFO[0309] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
330 INFO[0309] [addons] Executing deploy job rke-metrics-addon
331 INFO[0315] [addons] Metrics Server deployed successfully
332 INFO[0315] [ingress] Setting up nginx ingress controller
333 INFO[0315] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
334 INFO[0316] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
335 INFO[0316] [addons] Executing deploy job rke-ingress-controller
336 INFO[0322] [ingress] ingress controller nginx deployed successfully
337 INFO[0322] [addons] Setting up user addons
338 INFO[0322] [addons] no user addons defined
339 INFO[0322] Finished building Kubernetes cluster successfully
344 Download and install kubectl. Binaries can be found here for Linux and Mac:
346 https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/linux/amd64/kubectl
347 https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/darwin/amd64/kubectl
349 You only need to install kubectl where you'll launch Kubernetes command. This
350 can be any machines of the Kubernetes cluster or a machine that has IP access
352 Usually, we use the first controller as it has also access to internal
353 Kubernetes services, which can be convenient.
360 > cp kube_config_cluster.yml ~/.kube/config.onap
362 > export KUBECONFIG=~/.kube/config.onap
364 > kubectl config use-context onap
366 > kubectl get nodes -o=wide
370 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
371 onap-control-1 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.8 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
372 onap-control-2 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.11 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
373 onap-control-3 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.12 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
374 onap-k8s-1 Ready worker 3h53m v1.15.2 10.0.0.14 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
375 onap-k8s-10 Ready worker 3h53m v1.15.2 10.0.0.16 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
376 onap-k8s-11 Ready worker 3h53m v1.15.2 10.0.0.18 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
377 onap-k8s-12 Ready worker 3h53m v1.15.2 10.0.0.7 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
378 onap-k8s-2 Ready worker 3h53m v1.15.2 10.0.0.26 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
379 onap-k8s-3 Ready worker 3h53m v1.15.2 10.0.0.5 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
380 onap-k8s-4 Ready worker 3h53m v1.15.2 10.0.0.6 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
381 onap-k8s-5 Ready worker 3h53m v1.15.2 10.0.0.9 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
382 onap-k8s-6 Ready worker 3h53m v1.15.2 10.0.0.17 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
383 onap-k8s-7 Ready worker 3h53m v1.15.2 10.0.0.20 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
384 onap-k8s-8 Ready worker 3h53m v1.15.2 10.0.0.10 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
385 onap-k8s-9 Ready worker 3h53m v1.15.2 10.0.0.4 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
391 Example Helm client install on Linux::
393 > wget https://get.helm.sh/helm-v2.16.6-linux-amd64.tar.gz
395 > tar -zxvf helm-v2.16.6-linux-amd64.tar.gz
397 > sudo mv linux-amd64/helm /usr/local/bin/helm
399 Initialize Kubernetes Cluster for use by Helm
400 ---------------------------------------------
404 > kubectl -n kube-system create serviceaccount tiller
406 > kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
408 > helm init --service-account tiller
410 > kubectl -n kube-system rollout status deploy/tiller-deploy
414 Setting up an NFS share for Multinode Kubernetes Clusters
415 =========================================================
416 Deploying applications to a Kubernetes cluster requires Kubernetes nodes to
417 share a common, distributed filesystem. In this tutorial, we will setup an
418 NFS Master, and configure all Worker nodes a Kubernetes cluster to play
419 the role of NFS slaves.
421 It is recommneded that a separate VM, outside of the kubernetes
422 cluster, be used. This is to ensure that the NFS Master does not compete for
423 resources with Kubernetes Control Plane or Worker Nodes.
426 Launch new NFS Server VM instance
427 ---------------------------------
428 .. image:: images/nfs_server/nfs_server_1.png
430 Select Ubuntu 18.04 as base image
431 ---------------------------------
432 Select "No" on "Create New Volume"
434 .. image:: images/nfs_server/nfs_server_2.png
439 .. image:: images/nfs_server/nfs_server_3.png
444 .. image:: images/nfs_server/nfs_server_4.png
449 .. image:: images/nfs_server/nfs_server_5.png
453 Assign the key pair that was created/selected previously (e.g. onap_key).
455 .. image:: images/nfs_server/nfs_server_6.png
457 Apply customization script for NFS Server VM
458 --------------------------------------------
460 Click :download:`openstack-nfs-server.sh <openstack-nfs-server.sh>` to download
463 .. literalinclude:: openstack-nfs-server.sh
466 This customization script will:
475 .. image:: images/nfs_server/nfs_server_7.png
479 Assign Floating IP addresses
480 ----------------------------
482 .. image:: images/nfs_server/nfs_server_8.png
484 Resulting floating IP assignments in this example.
486 .. image:: images/nfs_server/nfs_server_9.png
489 To properly set up an NFS share on Master and Slave nodes, the user can run the
492 Click :download:`master_nfs_node.sh <master_nfs_node.sh>` to download the
495 .. literalinclude:: master_nfs_node.sh
498 Click :download:`slave_nfs_node.sh <slave_nfs_node.sh>` to download the script.
500 .. literalinclude:: slave_nfs_node.sh
503 The master_nfs_node.sh script runs in the NFS Master node and needs the list of
504 NFS Slave nodes as input, e.g.::
506 > sudo ./master_nfs_node.sh node1_ip node2_ip ... nodeN_ip
508 The slave_nfs_node.sh script runs in each NFS Slave node and needs the IP of
509 the NFS Master node as input, e.g.::
511 > sudo ./slave_nfs_node.sh master_node_ip
514 ONAP Deployment via OOM
515 =======================
516 Now that Kubernetes and Helm are installed and configured you can prepare to
517 deploy ONAP. Follow the instructions in the README.md_ or look at the official
518 documentation to get started:
520 - :ref:`quick-start-label` - deploy ONAP on an existing cloud
521 - :ref:`user-guide-label` - a guide for operators of an ONAP instance