1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. Copyright 2018 Amdocs, Bell Canada
6 .. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements
7 .. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
8 .. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
9 .. _Docker DevOps: https://wiki.onap.org/display/DW/Docker+DevOps#DockerDevOps-DockerBuild
10 .. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug
11 .. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474
12 .. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md
14 .. figure:: oomLogoV2-medium.png
17 .. _onap-on-kubernetes-with-rancher:
19 ONAP on Kubernetes with Rancher
20 ###############################
22 The following instructions will step you through the installation of Kubernetes
23 on an OpenStack environment with Rancher. The development lab used for this
24 installation is the ONAP Windriver lab.
26 This guide does not cover all of the steps required to setup your OpenStack
27 environment: e.g. OAM networks and security groups but there is a wealth of
28 OpenStack information on the web.
33 The following instructions describe how to create an Openstack VM running
34 Rancher. This node will not be used to host ONAP itself, it will be used
35 exclusively by Rancher.
37 Launch new VM instance to host the Rancher Server
38 -------------------------------------------------
40 .. image:: Rancher-Launch_new_VM_instance_to_host_the_Rancher_Server.jpeg
42 Select Ubuntu 16.04 as base image
43 ---------------------------------
44 Select "No" on "Create New Volume"
46 .. image:: Rancher-Select_Ubuntu_16.04_as_base_image.jpeg
50 Known issues exist if flavor is too small for Rancher. Please select a flavor
51 with at least 4 vCPU and 8GB ram. A size of 8 vCPU and 16GB ram is recommended.
53 .. image:: Rancher-Select_Flavor.jpeg
58 .. image:: Rancher-Networking.jpeg
63 .. image:: Rancher-Security_Groups.jpeg
67 Use an existing key pair (e.g. onap_key), import an existing one or create a
70 .. image:: Rancher-Key_Pair.jpeg
72 Apply customization script for the Rancher VM
73 ---------------------------------------------
75 Click :download:`openstack-rancher.sh <openstack-rancher.sh>` to download the
78 .. literalinclude:: openstack-rancher.sh
81 This customization script will:
83 * setup root access to the VM (comment out if you wish to disable this
84 capability and restrict access to ssh access only)
92 The Casablanca release of OOM only supports Helm 2.9.1 not the 2.7.2 shown in
93 the screen capture below. The supported versions of all the software components
94 are listed in the :ref:`cloud-setup-guide-label`.
96 .. image:: Apply_customization_script_for_the_Rancher_VM.jpeg
101 .. image:: Rancher-Launch_Instance.jpeg
103 Assign Floating IP for external access
104 --------------------------------------
106 .. image:: Rancher-Allocate_Floating_IP.jpeg
108 .. image:: Rancher-Manage_Floating_IP_Associations.jpeg
110 .. image:: Rancher-Launch_Instance.jpeg
112 Kubernetes Installation
113 =======================
115 Launch new VM instance(s) to create a Kubernetes single host or cluster
116 -----------------------------------------------------------------------
121 #. do not append a '-1' suffix (e.g. sb4-k8s)
122 #. increase count to the # of of kubernetes worker nodes you want (eg. 3)
124 .. image:: K8s-Launch_new_VM_instance_to_create_a_Kubernetes_single_host_or_cluster.jpeg
126 Select Ubuntu 16.04 as base image
127 ---------------------------------
128 Select "No" on "Create New Volume"
130 .. image:: K8s-Select_Ubuntu_16.04_as_base_image.jpeg
134 The size of a Kubernetes host depends on the size of the ONAP deployment that
137 As of the Casablanca release a minimum 224GB will be needed to run a
138 full ONAP deployment (all components). It is recommended that more hosts are
139 used with fewer resources instead of only a few large hosts. For example 14 x
142 If a small subset of ONAP components are being deployed for testing purposes,
143 then a single 16GB or 32GB host should suffice.
145 .. image:: K8s-Select_Flavor.jpeg
150 .. image:: K8s-Networking.jpeg
155 .. image:: K8s-Security_Group.jpeg
159 Use an existing key pair (e.g. onap_key), import an existing one or create a
162 .. image:: K8s-Key_Pair.jpeg
164 Apply customization script for Kubernetes VM(s)
165 -----------------------------------------------
167 Click :download:`openstack-k8s-node.sh <openstack-k8s-node.sh>` to
170 .. literalinclude:: openstack-k8s-node.sh
173 This customization script will:
175 * setup root access to the VM (comment out if you wish to disable this
176 capability and restrict access to ssh access only)
180 * install nfs common (see configuration step here)
183 Ensure you are using the correct versions as described in the
184 :ref:`cloud-setup-guide-label`
189 .. image:: K8s-Launch_Instance.jpeg
191 Assign Floating IP for external access
192 --------------------------------------
194 .. image:: K8s-Manage_Floating_IP_Associations.jpeg
196 .. image:: K8s-Launch_Instance.jpeg
198 Setting up an NFS share for Multinode Kubernetes Clusters
199 =========================================================
200 The figure below illustrates a possible topology of a multinode Kubernetes
203 .. image:: k8s-topology.jpg
205 One node, the Master Node, runs Rancher and Helm clients and connects to all
206 the Kubernetes nodes in the cluster. Kubernetes nodes, in turn, run Rancher,
207 Kubernetes and Tiller (Helm) agents, which receive, execute, and respond to
208 commands issued by the Master Node (e.g. kubectl or helm operations). Note that
209 the Master Node can be either a remote machine that the user can log in to or a
210 local machine (e.g. laptop, desktop) that has access to the Kubernetes cluster.
212 Deploying applications to a Kubernetes cluster requires Kubernetes nodes to
213 share a common, distributed filesystem. One node in the cluster plays the role
214 of NFS Master (not to confuse with the Master Node that runs Rancher and Helm
215 clients, which is located outside the cluster), while all the other cluster
216 nodes play the role of NFS slaves. In the figure above, the left-most cluster
217 node plays the role of NFS Master (indicated by the crown symbol). To properly
218 set up an NFS share on Master and Slave nodes, the user can run the scripts
221 Click :download:`master_nfs_node.sh <master_nfs_node.sh>` to download the
224 .. literalinclude:: master_nfs_node.sh
227 Click :download:`slave_nfs_node.sh <slave_nfs_node.sh>` to download the script.
229 .. literalinclude:: slave_nfs_node.sh
232 The master_nfs_node.sh script runs in the NFS Master node and needs the list of
233 NFS Slave nodes as input, e.g.::
235 > sudo ./master_nfs_node.sh node1_ip node2_ip ... nodeN_ip
237 The slave_nfs_node.sh script runs in each NFS Slave node and needs the IP of
238 the NFS Master node as input, e.g.::
240 > sudo ./slave_nfs_node.sh master_node_ip
242 Configuration (Rancher and Kubernetes)
243 ======================================
245 Access Rancher server via web browser
246 -------------------------------------
247 (e.g. http://10.12.6.16:8080/env/1a5/apps/stacks)
249 .. image:: Access_Rancher_server_via_web_browser.jpeg
251 Add Kubernetes Environment to Rancher
252 -------------------------------------
254 1. Select “Manage Environments”
256 .. image:: Add_Kubernetes_Environment_to_Rancher.png
258 2. Select “Add Environment”
260 .. image:: Select_Add_Environment.png
262 3. Add unique name for your new Rancher environment
264 4. Select the Kubernetes template
268 .. image:: Click_create.jpeg
270 6. Select the new named environment (ie. SB4) from the dropdown list (top
273 Rancher is now waiting for a Kubernetes Host to be added.
275 .. image:: K8s-Assign_Floating_IP_for_external_access.jpeg
280 1. If this is the first (or only) host being added - click on the "Add a host"
283 .. image:: K8s-Assign_Floating_IP_for_external_access.jpeg
285 and click on "Save" (accept defaults).
287 .. image:: and_click_on_Save_accept_defaults.jpeg
289 otherwise select INFRASTRUCTURE→ Hosts and click on "Add Host"
291 .. image:: otherwise_select_INFRASTRUCTURE_Hosts_and_click_on_Add_Host.jpg
293 2. Enter the management IP for the k8s VM (e.g. 10.0.0.4) that was just
296 3. Click on “Copy to Clipboard” button
298 4. Click on “Close” button
300 .. image:: Click_on_Close_button.jpeg
302 Without the 10.0.0.4 IP - the CATTLE_AGENT will be derived on the host - but it
303 may not be a routable IP.
305 Configure Kubernetes Host
306 -------------------------
308 1. Login to the new Kubernetes Host::
310 > ssh -i ~/oom-key.pem ubuntu@10.12.5.1
311 The authenticity of host '10.12.5.172 (10.12.5.172)' can't be established.
312 ECDSA key fingerprint is SHA256:tqxayN58nCJKOJcWrEZzImkc0qKQHDDfUTHqk4WMcEI.
313 Are you sure you want to continue connecting (yes/no)? yes
314 Warning: Permanently added '10.12.5.172' (ECDSA) to the list of known hosts.
315 Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-64-generic x86_64)
317 * Documentation: https://help.ubuntu.com
318 * Management: https://landscape.canonical.com
319 * Support: https://ubuntu.com/advantage
321 Get cloud support with Ubuntu Advantage Cloud Guest:
322 http://www.ubuntu.com/business/services/cloud
324 180 packages can be updated.
325 100 updates are security updates.
327 The programs included with the Ubuntu system are free software;
328 the exact distribution terms for each program are described in the
329 individual files in /usr/share/doc/*/copyright.
331 Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
334 To run a command as administrator (user "root"), use "sudo <command>".
335 See "man sudo_root" for details.
340 2. Paste Clipboard content and hit enter to install Rancher Agent::
342 ubuntu@sb4-k8s-1:~$ sudo docker run -e CATTLE_AGENT_IP="10.0.0.4“ --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.9 http://10.12.6.16:8080/v1/scripts/5D757C68BD0A2125602A:1514678400000:yKW9xHGJDLvq6drz2eDzR2mjato
343 Unable to find image 'rancher/agent:v1.2.9' locally
344 v1.2.9: Pulling From rancher/agent
345 b3e1c725a85f: Pull complete
346 6071086409fc: Pull complete
347 d0ac3b234321: Pull complete
348 87f567b5cf58: Pull complete
349 a63e24b217c4: Pull complete
350 d0a3f58caef0: Pull complete
351 16914729cfd3: Pull complete
352 dc5c21984c5b: Pull complete
353 d7e8f9784b20: Pull complete
354 Digest: sha256:c21255ac4d94ffbc7b523F870F20ea5189b68Fa3d642800adb4774aab4748e66
355 Status: Downloaded newer image for rancher/agent:v1.2.9
357 INFO: Running Agent Registration Process, CATTLE_URL=http://10.12.6.16:8080/v1
358 INFO: Attempting to connect to: http://10.12.6.16:8080/v1
359 INFO: http://10.12.6.16:8080/v1 is accessible
360 INFO: Inspecting host capabilities
361 INFO: Boot2Docker: false
362 INFO: Host writable: true
363 INFO: Token: xxxxxxxx
364 INFO: Running registration
365 INFO: Printing Environment
366 INFO: ENV: CATTLE_ACCESS_KEY=98B35AC484FBF820E0AD
367 INFO: ENV: CATTLE_AGENT_IP=10.0.9.4
368 INFO: ENV: CATTLE_HOME=/var/lib/cattle
369 INFO: ENV: CATTLE_REGISTRATION_ACCESS_KEY=registrationToken
370 INFO: ENV: CATTLE_REGISTRATION_SECRET_KEY=xxxxxxx
371 INFO: ENV: CATTLE_SECRET_KEY=xxxxxxx
372 INFO: ENV: CATTLE_URL=http://10.12.6.16:8080/v1
373 INFO: ENV: DETECTED_CATTLE_AGENT_IP=10.12.5.172
374 INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.2.9
375 INFO: Launched Rancher Agent: c27ee0f3dc4c783b0db647ea1f73c35b3843a4b8d60b96375b1a05aa77d83136
378 3. Return to Rancher environment (e.g. SB4) and wait for services to complete
381 .. image:: Return_to_Rancher_environment_eg_SB4_and_wait_for_services_to_complete_10-15_mins.jpeg
383 Configure kubectl and helm
384 ==========================
385 In this example we are configuring kubectl and helm that have been installed
386 (as a convenience) onto the rancher and kubernetes hosts. Typically you would
387 install them both on your PC and remotely connect to the cluster. The following
388 procedure would remain the same.
390 1. Click on CLI and then click on “Generate Config”
392 .. image:: Click_on_CLI_and_then_click_on_Generate_Config.jpeg
394 2. Click on “Copy to Clipboard” - wait until you see a "token" - do not copy
395 user+password - the server is not ready at that point
397 .. image:: Click_on_Copy_to_Clipboard-wait_until_you_see_a_token-do_not_copy_user+password-the_server_is_not_ready_at_that_point.jpeg
399 3. Create a .kube directory in user directory (if one does not exist)::
401 ubuntu@sb4-kSs-1:~$ mkdir .kube
402 ubuntu@sb4-kSs-1:~$ vi .kube/config
404 4. Paste contents of Clipboard into a file called “config” and save the file::
411 insecure-skip-tls-verify: true
412 server: "https://10.12.6.16:8080/r/projects/1a7/kubernetes:6443"
419 current-context: "SB4"
423 token: "QmFzaWMgTlRBd01qZzBOemc)TkRrMk1UWkNOMFpDTlVFNlExcHdSa1JhVZreE5XSm1TRGhWU2t0Vk1sQjVhalZaY0dWaFVtZGFVMHQzWW1WWVJtVmpSQT09"
425 5. Validate that kubectl is able to connect to the kubernetes cluster::
427 ubuntu@sb4-k8s-1:~$ kubectl config get-contexts
428 CURRENT NAME CLUSTER AUTHINFO NAMESPACE
432 and show running pods::
434 ubuntu@sb4-k8s-1:~$ kubectl get pods --all-namespaces -o=wide
435 NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
436 kube-system heapster—7Gb8cd7b5 -q7p42 1/1 Running 0 13m 10.42.213.49 sb4-k8s-1
437 kube-system kube-dns-5d7bM87c9-c6f67 3/3 Running 0 13m 10.42.181.110 sb4-k8s-1
438 kube-system kubernetes-dashboard-f9577fffd-kswjg 1/1 Running 0 13m 10.42.105.113 sb4-k8s-1
439 kube-system monitoring-grafana-997796fcf-vg9h9 1/1 Running 0 13m 10.42,141.58 sb4-k8s-1
440 kube-system monitoring-influxdb-56chd96b-hk66b 1/1 Running 0 13m 10.4Z.246.90 sb4-k8s-1
441 kube-system tiller-deploy-cc96d4f6b-v29k9 1/1 Running 0 13m 10.42.147.248 sb4-k8s-1
444 6. Validate helm is running at the right version. If not, an error like this
447 ubuntu@sb4-k8s-1:~$ helm list
448 Error: incompatible versions c1ient[v2.9.1] server[v2.6.1]
451 7. Upgrade the server-side component of helm (tiller) via `helm init --upgrade`::
453 ubuntu@sb4-k8s-1:~$ helm init --upgrade
454 Creating /home/ubuntu/.helm
455 Creating /home/ubuntu/.helm/repository
456 Creating /home/ubuntu/.helm/repository/cache
457 Creating /home/ubuntu/.helm/repository/local
458 Creating /home/ubuntu/.helm/plugins
459 Creating /home/ubuntu/.helm/starters
460 Creating /home/ubuntu/.helm/cache/archive
461 Creating /home/ubuntu/.helm/repository/repositories.yaml
462 Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
463 Adding local repo with URL: http://127.0.0.1:8879/charts
464 $HELM_HOME has been configured at /home/ubuntu/.helm.
466 Tiller (the Helm server-side component) has been upgraded to the current version.
470 ONAP Deployment via OOM
471 =======================
472 Now that kubernetes and Helm are installed and configured you can prepare to
473 deploy ONAP. Follow the instructions in the README.md_ or look at the official
474 documentation to get started:
476 - :ref:`quick-start-label` - deploy ONAP on an existing cloud
477 - :ref:`user-guide-label` - a guide for operators of an ONAP instance