1 .. This work is licensed under a Creative Commons Attribution 4.0
2 .. International License.
3 .. http://creativecommons.org/licenses/by/4.0
4 .. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung
7 .. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements
8 .. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
9 .. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
10 .. _Docker DevOps: https://wiki.onap.org/display/DW/Docker+DevOps#DockerDevOps-DockerBuild
11 .. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug
12 .. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474
13 .. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md
15 .. figure:: oomLogoV2-medium.png
18 .. _onap-on-kubernetes-with-rancher:
20 ONAP on HA Kubernetes Cluster
21 #############################
23 This guide provides instructions on how to setup a Highly-Available Kubernetes
24 Cluster. For this, we are hosting our cluster on OpenStack VMs and using the
25 Rancher Kubernetes Engine (RKE) to deploy and manage our Kubernetes Cluster.
32 The result at the end of this tutorial will be:
34 #. Creation of a Key Pair to use with Open Stack and RKE
36 #. Creation of OpenStack VMs to host Kubernetes Control Plane
38 #. Creation of OpenStack VMs to host Kubernetes Workers
40 #. Installation and configuration of RKE to setup an HA Kubernetes
42 #. Installation and configuration of kubectl
44 #. Installation and configuration of Helm
46 #. Creation of an NFS Server to be used by ONAP as shared persistance
48 There are many ways one can execute the above steps. Including automation
49 through the use of HEAT to setup the OpenStack VMs. To better illustrate the
50 steps involved, we have captured the manual creation of such an environment
51 using the ONAP Wind River Open Lab.
55 A Key Pair is required to access the created OpenStack VMs and will be used by
56 RKE to configure the VMs for Kubernetes.
58 Use an existing key pair, import one or create a new one to assign.
60 .. image:: images/keys/key_pair_1.png
63 If you're creating a new Key Pair, ensure to create a local copy of the
64 Private Key through the use of "Copy Private Key to Clipboard".
66 For the purpose of this guide, we will assume a new local key called "onap-key"
67 has been downloaded and is copied into **~/.ssh/**, from which it can be
74 > chmod 600 ~/.ssh/onap-key
80 An internal network is required in order to deploy our VMs that will host
83 .. image:: images/network/network_1.png
85 .. image:: images/network/network_2.png
87 .. image:: images/network/network_3.png
90 It's better to have one network per deployment and obviously the name of this
91 network should be unique.
93 Now we need to create a router to attach this network to outside:
95 .. image:: images/network/network_4.png
100 A specific security group is also required
102 .. image:: images/sg/sg_1.png
104 then click on `manage rules` of the newly created security group.
105 And finally click on `Add Rule` and create the following one:
107 .. image:: images/sg/sg_2.png
110 the security is clearly not good here and the right SG will be proposed in a
113 Create Kubernetes Control Plane VMs
114 ===================================
116 The following instructions describe how to create 3 OpenStack VMs to host the
117 Highly-Available Kubernetes Control Plane.
118 ONAP workloads will not be scheduled on these Control Plane nodes.
120 Launch new VM instances
121 -----------------------
123 .. image:: images/cp_vms/control_plane_1.png
125 Select Ubuntu 18.04 as base image
126 ---------------------------------
127 Select "No" for "Create New Volume"
129 .. image:: images/cp_vms/control_plane_2.png
133 The recommended flavor is at least 4 vCPU and 8GB ram.
135 .. image:: images/cp_vms/control_plane_3.png
140 Use the created network:
142 .. image:: images/cp_vms/control_plane_4.png
147 Use the created security group:
149 .. image:: images/cp_vms/control_plane_5.png
153 Assign the key pair that was created/selected previously (e.g. onap_key).
155 .. image:: images/cp_vms/control_plane_6.png
157 Apply customization script for Control Plane VMs
158 ------------------------------------------------
160 Click :download:`openstack-k8s-controlnode.sh <openstack-k8s-controlnode.sh>`
161 to download the script.
163 .. literalinclude:: openstack-k8s-controlnode.sh
166 This customization script will:
171 .. image:: images/cp_vms/control_plane_7.png
176 .. image:: images/cp_vms/control_plane_8.png
180 Create Kubernetes Worker VMs
181 ============================
182 The following instructions describe how to create OpenStack VMs to host the
183 Highly-Available Kubernetes Workers. ONAP workloads will only be scheduled on
186 Launch new VM instances
187 -----------------------
189 The number and size of Worker VMs is dependent on the size of the ONAP
190 deployment. By default, all ONAP applications are deployed. It's possible to
191 customize the deployment and enable a subset of the ONAP applications. For the
192 purpose of this guide, however, we will deploy 12 Kubernetes Workers that have
193 been sized to handle the entire ONAP application workload.
195 .. image:: images/wk_vms/worker_1.png
197 Select Ubuntu 18.04 as base image
198 ---------------------------------
199 Select "No" on "Create New Volume"
201 .. image:: images/wk_vms/worker_2.png
205 The size of Kubernetes hosts depend on the size of the ONAP deployment
208 If a small subset of ONAP applications are being deployed
209 (i.e. for testing purposes), then 16GB or 32GB may be sufficient.
211 .. image:: images/wk_vms/worker_3.png
216 .. image:: images/wk_vms/worker_4.png
221 .. image:: images/wk_vms/worker_5.png
225 Assign the key pair that was created/selected previously (e.g. onap_key).
227 .. image:: images/wk_vms/worker_6.png
229 Apply customization script for Kubernetes VM(s)
230 -----------------------------------------------
232 Click :download:`openstack-k8s-workernode.sh <openstack-k8s-workernode.sh>` to
235 .. literalinclude:: openstack-k8s-workernode.sh
238 This customization script will:
248 .. image:: images/wk_vms/worker_7.png
253 Assign Floating IP addresses
254 ----------------------------
255 Assign Floating IPs to all Control Plane and Worker VMs.
256 These addresses provide external access to the VMs and will be used by RKE
257 to configure kubernetes on to the VMs.
259 Repeat the following for each VM previously created:
261 .. image:: images/floating_ips/floating_1.png
263 Resulting floating IP assignments in this example.
265 .. image:: images/floating_ips/floating_2.png
270 Configure Rancher Kubernetes Engine (RKE)
271 =========================================
275 Download and install RKE on a VM, desktop or laptop.
276 Binaries can be found here for Linux and Mac: https://github.com/rancher/rke/releases/tag/v1.0.6
279 There are several ways to install RKE. Further parts of this documentation
280 assumes that you have rke command available.
281 If you don't know how to install RKE you may follow the below steps:
283 * chmod +x ./rke_linux-amd64
284 * sudo mv ./rke_linux-amd64 /user/local/bin/rke
286 RKE requires a *cluster.yml* as input. An example file is show below that
287 describes a Kubernetes cluster that will be mapped onto the OpenStack VMs
288 created earlier in this guide.
290 Click :download:`cluster.yml <cluster.yml>` to download the
293 .. literalinclude:: cluster.yml
298 Before this configuration file can be used the external **address**
299 and the **internal_address** must be mapped for each control and worker node
304 From within the same directory as the cluster.yml file, simply execute::
308 The output will look something like::
310 INFO[0000] Initiating Kubernetes cluster
311 INFO[0000] [certificates] Generating admin certificates and kubeconfig
312 INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
313 INFO[0000] Building Kubernetes cluster
314 INFO[0000] [dialer] Setup tunnel for host [10.12.6.82]
315 INFO[0000] [dialer] Setup tunnel for host [10.12.6.249]
316 INFO[0000] [dialer] Setup tunnel for host [10.12.6.74]
317 INFO[0000] [dialer] Setup tunnel for host [10.12.6.85]
318 INFO[0000] [dialer] Setup tunnel for host [10.12.6.238]
319 INFO[0000] [dialer] Setup tunnel for host [10.12.6.89]
320 INFO[0000] [dialer] Setup tunnel for host [10.12.5.11]
321 INFO[0000] [dialer] Setup tunnel for host [10.12.6.90]
322 INFO[0000] [dialer] Setup tunnel for host [10.12.6.244]
323 INFO[0000] [dialer] Setup tunnel for host [10.12.5.165]
324 INFO[0000] [dialer] Setup tunnel for host [10.12.6.126]
325 INFO[0000] [dialer] Setup tunnel for host [10.12.6.111]
326 INFO[0000] [dialer] Setup tunnel for host [10.12.5.160]
327 INFO[0000] [dialer] Setup tunnel for host [10.12.5.191]
328 INFO[0000] [dialer] Setup tunnel for host [10.12.6.195]
329 INFO[0002] [network] Deploying port listener containers
330 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.85]
331 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.89]
332 INFO[0002] [network] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.90]
333 INFO[0011] [network] Successfully pulled image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.27] on host [10.12.6.89]
335 INFO[0309] [addons] Setting up Metrics Server
336 INFO[0309] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
337 INFO[0309] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
338 INFO[0309] [addons] Executing deploy job rke-metrics-addon
339 INFO[0315] [addons] Metrics Server deployed successfully
340 INFO[0315] [ingress] Setting up nginx ingress controller
341 INFO[0315] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
342 INFO[0316] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
343 INFO[0316] [addons] Executing deploy job rke-ingress-controller
344 INFO[0322] [ingress] ingress controller nginx deployed successfully
345 INFO[0322] [addons] Setting up user addons
346 INFO[0322] [addons] no user addons defined
347 INFO[0322] Finished building Kubernetes cluster successfully
352 Download and install kubectl. Binaries can be found here for Linux and Mac:
354 https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/linux/amd64/kubectl
355 https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/darwin/amd64/kubectl
357 You only need to install kubectl where you'll launch Kubernetes command. This
358 can be any machines of the Kubernetes cluster or a machine that has IP access
360 Usually, we use the first controller as it has also access to internal
361 Kubernetes services, which can be convenient.
370 > cp kube_config_cluster.yml ~/.kube/config.onap
372 > export KUBECONFIG=~/.kube/config.onap
374 > kubectl config use-context onap
376 > kubectl get nodes -o=wide
380 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
381 onap-control-1 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.8 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
382 onap-control-2 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.11 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
383 onap-control-3 Ready controlplane,etcd 3h53m v1.15.2 10.0.0.12 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
384 onap-k8s-1 Ready worker 3h53m v1.15.2 10.0.0.14 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
385 onap-k8s-10 Ready worker 3h53m v1.15.2 10.0.0.16 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
386 onap-k8s-11 Ready worker 3h53m v1.15.2 10.0.0.18 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
387 onap-k8s-12 Ready worker 3h53m v1.15.2 10.0.0.7 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
388 onap-k8s-2 Ready worker 3h53m v1.15.2 10.0.0.26 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
389 onap-k8s-3 Ready worker 3h53m v1.15.2 10.0.0.5 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
390 onap-k8s-4 Ready worker 3h53m v1.15.2 10.0.0.6 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
391 onap-k8s-5 Ready worker 3h53m v1.15.2 10.0.0.9 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
392 onap-k8s-6 Ready worker 3h53m v1.15.2 10.0.0.17 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
393 onap-k8s-7 Ready worker 3h53m v1.15.2 10.0.0.20 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
394 onap-k8s-8 Ready worker 3h53m v1.15.2 10.0.0.10 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
395 onap-k8s-9 Ready worker 3h53m v1.15.2 10.0.0.4 <none> Ubuntu 18.04 LTS 4.15.0-22-generic docker://18.9.5
401 Example Helm client install on Linux::
403 > wget https://get.helm.sh/helm-v2.16.6-linux-amd64.tar.gz
405 > tar -zxvf helm-v2.16.6-linux-amd64.tar.gz
407 > sudo mv linux-amd64/helm /usr/local/bin/helm
409 Initialize Kubernetes Cluster for use by Helm
410 ---------------------------------------------
414 > kubectl -n kube-system create serviceaccount tiller
416 > kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
418 > helm init --service-account tiller
420 > kubectl -n kube-system rollout status deploy/tiller-deploy
424 Setting up an NFS share for Multinode Kubernetes Clusters
425 =========================================================
426 Deploying applications to a Kubernetes cluster requires Kubernetes nodes to
427 share a common, distributed filesystem. In this tutorial, we will setup an
428 NFS Master, and configure all Worker nodes a Kubernetes cluster to play
429 the role of NFS slaves.
431 It is recommended that a separate VM, outside of the kubernetes
432 cluster, be used. This is to ensure that the NFS Master does not compete for
433 resources with Kubernetes Control Plane or Worker Nodes.
436 Launch new NFS Server VM instance
437 ---------------------------------
438 .. image:: images/nfs_server/nfs_server_1.png
440 Select Ubuntu 18.04 as base image
441 ---------------------------------
442 Select "No" on "Create New Volume"
444 .. image:: images/nfs_server/nfs_server_2.png
449 .. image:: images/nfs_server/nfs_server_3.png
454 .. image:: images/nfs_server/nfs_server_4.png
459 .. image:: images/nfs_server/nfs_server_5.png
463 Assign the key pair that was created/selected previously (e.g. onap_key).
465 .. image:: images/nfs_server/nfs_server_6.png
467 Apply customization script for NFS Server VM
468 --------------------------------------------
470 Click :download:`openstack-nfs-server.sh <openstack-nfs-server.sh>` to download
473 .. literalinclude:: openstack-nfs-server.sh
476 This customization script will:
485 .. image:: images/nfs_server/nfs_server_7.png
489 Assign Floating IP addresses
490 ----------------------------
492 .. image:: images/nfs_server/nfs_server_8.png
494 Resulting floating IP assignments in this example.
496 .. image:: images/nfs_server/nfs_server_9.png
499 To properly set up an NFS share on Master and Slave nodes, the user can run the
502 Click :download:`master_nfs_node.sh <master_nfs_node.sh>` to download the
505 .. literalinclude:: master_nfs_node.sh
508 Click :download:`slave_nfs_node.sh <slave_nfs_node.sh>` to download the script.
510 .. literalinclude:: slave_nfs_node.sh
513 The master_nfs_node.sh script runs in the NFS Master node and needs the list of
514 NFS Slave nodes as input, e.g.::
516 > sudo ./master_nfs_node.sh node1_ip node2_ip ... nodeN_ip
518 The slave_nfs_node.sh script runs in each NFS Slave node and needs the IP of
519 the NFS Master node as input, e.g.::
521 > sudo ./slave_nfs_node.sh master_node_ip
524 ONAP Deployment via OOM
525 =======================
526 Now that Kubernetes and Helm are installed and configured you can prepare to
527 deploy ONAP. Follow the instructions in the README.md_ or look at the official
528 documentation to get started:
530 - :ref:`quick-start-label` - deploy ONAP on an existing cloud
531 - :ref:`user-guide-label` - a guide for operators of an ONAP instance