.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 .. Copyright 2018 Amdocs, Bell Canada .. Links .. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements .. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ .. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir .. _Docker DevOps: https://wiki.onap.org/display/DW/Docker+DevOps#DockerDevOps-DockerBuild .. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug .. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474 .. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md .. figure:: oomLogoV2-medium.png :align: right .. _onap-on-kubernetes-with-rancher: ONAP on Kubernetes with Rancher ############################### The following instructions will step you through the installation of Kubernetes on an OpenStack environment with Rancher. The development lab used for this installation is the ONAP Windriver lab. This guide does not cover all of the steps required to setup your OpenStack environment: e.g. OAM networks and security groups but there is a wealth of OpenStack information on the web. Rancher Installation ==================== The following instructions describe how to create an Openstack VM running Rancher. This node will not be used to host ONAP itself, it will be used exclusively by Rancher. Launch new VM instance to host the Rancher Server ------------------------------------------------- .. image:: Rancher-Launch_new_VM_instance_to_host_the_Rancher_Server.jpeg Select Ubuntu 16.04 as base image --------------------------------- Select "No" on "Create New Volume" .. image:: Rancher-Select_Ubuntu_16.04_as_base_image.jpeg Select Flavor ------------- Known issues exist if flavor is too small for Rancher. Please select a flavor with at least 4 vCPU and 8GB ram. A size of 8 vCPU and 16GB ram is recommended. .. image:: Rancher-Select_Flavor.jpeg Networking ---------- .. image:: Rancher-Networking.jpeg Security Groups --------------- .. image:: Rancher-Security_Groups.jpeg Key Pair -------- Use an existing key pair (e.g. onap_key), import an existing one or create a new one to assign. .. image:: Rancher-Key_Pair.jpeg Apply customization script for the Rancher VM --------------------------------------------- Click :download:`openstack-rancher.sh ` to download the script. .. literalinclude:: openstack-rancher.sh :language: bash This customization script will: * setup root access to the VM (comment out if you wish to disable this capability and restrict access to ssh access only) * install docker * install rancher * install kubectl * install helm * install nfs server .. note:: The Casablanca release of OOM only supports Helm 2.9.1 not the 2.7.2 shown in the screen capture below. The supported versions of all the software components are listed in the :ref:`cloud-setup-guide-label`. .. image:: Apply_customization_script_for_the_Rancher_VM.jpeg Launch Instance --------------- .. image:: Rancher-Launch_Instance.jpeg Assign Floating IP for external access -------------------------------------- .. image:: Rancher-Allocate_Floating_IP.jpeg .. image:: Rancher-Manage_Floating_IP_Associations.jpeg .. image:: Rancher-Launch_Instance.jpeg Kubernetes Installation ======================= Launch new VM instance(s) to create a Kubernetes single host or cluster ----------------------------------------------------------------------- To create a cluster: .. note:: #. do not append a '-1' suffix (e.g. sb4-k8s) #. increase count to the # of of kubernetes worker nodes you want (eg. 3) .. image:: K8s-Launch_new_VM_instance_to_create_a_Kubernetes_single_host_or_cluster.jpeg Select Ubuntu 16.04 as base image --------------------------------- Select "No" on "Create New Volume" .. image:: K8s-Select_Ubuntu_16.04_as_base_image.jpeg Select Flavor ------------- The size of a Kubernetes host depends on the size of the ONAP deployment that will be installed. As of the Casablanca release a minimum 224GB will be needed to run a full ONAP deployment (all components). It is recommended that more hosts are used with fewer resources instead of only a few large hosts. For example 14 x 16GB hosts. If a small subset of ONAP components are being deployed for testing purposes, then a single 16GB or 32GB host should suffice. .. image:: K8s-Select_Flavor.jpeg Networking ----------- .. image:: K8s-Networking.jpeg Security Group --------------- .. image:: K8s-Security_Group.jpeg Key Pair -------- Use an existing key pair (e.g. onap_key), import an existing one or create a new one to assign. .. image:: K8s-Key_Pair.jpeg Apply customization script for Kubernetes VM(s) ----------------------------------------------- Click :download:`openstack-k8s-node.sh ` to download the script. .. literalinclude:: openstack-k8s-node.sh :language: bash This customization script will: * setup root access to the VM (comment out if you wish to disable this capability and restrict access to ssh access only) * install docker * install kubectl * install helm * install nfs common (see configuration step here) .. note:: Ensure you are using the correct versions as described in the :ref:`cloud-setup-guide-label` Launch Instance --------------- .. image:: K8s-Launch_Instance.jpeg Assign Floating IP for external access -------------------------------------- .. image:: K8s-Manage_Floating_IP_Associations.jpeg .. image:: K8s-Launch_Instance.jpeg Setting up an NFS share for Multinode Kubernetes Clusters ========================================================= The figure below illustrates a possible topology of a multinode Kubernetes cluster. .. image:: k8s-topology.jpg One node, the Master Node, runs Rancher and Helm clients and connects to all the Kubernetes nodes in the cluster. Kubernetes nodes, in turn, run Rancher, Kubernetes and Tiller (Helm) agents, which receive, execute, and respond to commands issued by the Master Node (e.g. kubectl or helm operations). Note that the Master Node can be either a remote machine that the user can log in to or a local machine (e.g. laptop, desktop) that has access to the Kubernetes cluster. Deploying applications to a Kubernetes cluster requires Kubernetes nodes to share a common, distributed filesystem. One node in the cluster plays the role of NFS Master (not to confuse with the Master Node that runs Rancher and Helm clients, which is located outside the cluster), while all the other cluster nodes play the role of NFS slaves. In the figure above, the left-most cluster node plays the role of NFS Master (indicated by the crown symbol). To properly set up an NFS share on Master and Slave nodes, the user can run the scripts below. Click :download:`master_nfs_node.sh ` to download the script. .. literalinclude:: master_nfs_node.sh :language: bash Click :download:`slave_nfs_node.sh ` to download the script. .. literalinclude:: slave_nfs_node.sh :language: bash The master_nfs_node.sh script runs in the NFS Master node and needs the list of NFS Slave nodes as input, e.g.:: > sudo ./master_nfs_node.sh node1_ip node2_ip ... nodeN_ip The slave_nfs_node.sh script runs in each NFS Slave node and needs the IP of the NFS Master node as input, e.g.:: > sudo ./slave_nfs_node.sh master_node_ip Configuration (Rancher and Kubernetes) ====================================== Access Rancher server via web browser ------------------------------------- (e.g. http://10.12.6.16:8080/env/1a5/apps/stacks) .. image:: Access_Rancher_server_via_web_browser.jpeg Add Kubernetes Environment to Rancher ------------------------------------- 1. Select “Manage Environments” .. image:: Add_Kubernetes_Environment_to_Rancher.png 2. Select “Add Environment” .. image:: Select_Add_Environment.png 3. Add unique name for your new Rancher environment 4. Select the Kubernetes template 5. Click "create" .. image:: Click_create.jpeg 6. Select the new named environment (ie. SB4) from the dropdown list (top left). Rancher is now waiting for a Kubernetes Host to be added. .. image:: K8s-Assign_Floating_IP_for_external_access.jpeg Add Kubernetes Host ------------------- 1. If this is the first (or only) host being added - click on the "Add a host" link .. image:: K8s-Assign_Floating_IP_for_external_access.jpeg and click on "Save" (accept defaults). .. image:: and_click_on_Save_accept_defaults.jpeg otherwise select INFRASTRUCTURE→ Hosts and click on "Add Host" .. image:: otherwise_select_INFRASTRUCTURE_Hosts_and_click_on_Add_Host.jpg 2. Enter the management IP for the k8s VM (e.g. 10.0.0.4) that was just created. 3. Click on “Copy to Clipboard” button 4. Click on “Close” button .. image:: Click_on_Close_button.jpeg Without the 10.0.0.4 IP - the CATTLE_AGENT will be derived on the host - but it may not be a routable IP. Configure Kubernetes Host ------------------------- 1. Login to the new Kubernetes Host:: > ssh -i ~/oom-key.pem ubuntu@10.12.5.1 The authenticity of host '10.12.5.172 (10.12.5.172)' can't be established. ECDSA key fingerprint is SHA256:tqxayN58nCJKOJcWrEZzImkc0qKQHDDfUTHqk4WMcEI. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.12.5.172' (ECDSA) to the list of known hosts. Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-64-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 180 packages can be updated. 100 updates are security updates. The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details. ubuntu@sb4-k8s-1:~$ 2. Paste Clipboard content and hit enter to install Rancher Agent:: ubuntu@sb4-k8s-1:~$ sudo docker run -e CATTLE_AGENT_IP="10.0.0.4“ --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.9 http://10.12.6.16:8080/v1/scripts/5D757C68BD0A2125602A:1514678400000:yKW9xHGJDLvq6drz2eDzR2mjato Unable to find image 'rancher/agent:v1.2.9' locally v1.2.9: Pulling From rancher/agent b3e1c725a85f: Pull complete 6071086409fc: Pull complete d0ac3b234321: Pull complete 87f567b5cf58: Pull complete a63e24b217c4: Pull complete d0a3f58caef0: Pull complete 16914729cfd3: Pull complete dc5c21984c5b: Pull complete d7e8f9784b20: Pull complete Digest: sha256:c21255ac4d94ffbc7b523F870F20ea5189b68Fa3d642800adb4774aab4748e66 Status: Downloaded newer image for rancher/agent:v1.2.9 INFO: Running Agent Registration Process, CATTLE_URL=http://10.12.6.16:8080/v1 INFO: Attempting to connect to: http://10.12.6.16:8080/v1 INFO: http://10.12.6.16:8080/v1 is accessible INFO: Inspecting host capabilities INFO: Boot2Docker: false INFO: Host writable: true INFO: Token: xxxxxxxx INFO: Running registration INFO: Printing Environment INFO: ENV: CATTLE_ACCESS_KEY=98B35AC484FBF820E0AD INFO: ENV: CATTLE_AGENT_IP=10.0.9.4 INFO: ENV: CATTLE_HOME=/var/lib/cattle INFO: ENV: CATTLE_REGISTRATION_ACCESS_KEY=registrationToken INFO: ENV: CATTLE_REGISTRATION_SECRET_KEY=xxxxxxx INFO: ENV: CATTLE_SECRET_KEY=xxxxxxx INFO: ENV: CATTLE_URL=http://10.12.6.16:8080/v1 INFO: ENV: DETECTED_CATTLE_AGENT_IP=10.12.5.172 INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.2.9 INFO: Launched Rancher Agent: c27ee0f3dc4c783b0db647ea1f73c35b3843a4b8d60b96375b1a05aa77d83136 ubuntu@sb4-k8s-1:~$ 3. Return to Rancher environment (e.g. SB4) and wait for services to complete (~ 10-15 mins) .. image:: Return_to_Rancher_environment_eg_SB4_and_wait_for_services_to_complete_10-15_mins.jpeg Configure kubectl and helm ========================== In this example we are configuring kubectl and helm that have been installed (as a convenience) onto the rancher and kubernetes hosts. Typically you would install them both on your PC and remotely connect to the cluster. The following procedure would remain the same. 1. Click on CLI and then click on “Generate Config” .. image:: Click_on_CLI_and_then_click_on_Generate_Config.jpeg 2. Click on “Copy to Clipboard” - wait until you see a "token" - do not copy user+password - the server is not ready at that point .. image:: Click_on_Copy_to_Clipboard-wait_until_you_see_a_token-do_not_copy_user+password-the_server_is_not_ready_at_that_point.jpeg 3. Create a .kube directory in user directory (if one does not exist):: ubuntu@sb4-kSs-1:~$ mkdir .kube ubuntu@sb4-kSs-1:~$ vi .kube/config 4. Paste contents of Clipboard into a file called “config” and save the file:: apiVersion: v1 kind : Config clusters: - cluster: api-version: v1 insecure-skip-tls-verify: true server: "https://10.12.6.16:8080/r/projects/1a7/kubernetes:6443" name: "SB4" contexts: - context: cluster: "SB4" user: "SB4" name: "SB4" current-context: "SB4" users: - name: "SB4" user: token: "QmFzaWMgTlRBd01qZzBOemc)TkRrMk1UWkNOMFpDTlVFNlExcHdSa1JhVZreE5XSm1TRGhWU2t0Vk1sQjVhalZaY0dWaFVtZGFVMHQzWW1WWVJtVmpSQT09" 5. Validate that kubectl is able to connect to the kubernetes cluster:: ubuntu@sb4-k8s-1:~$ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * SB4 SB4 SB4 ubuntu@sb4-kSs-1:~$ and show running pods:: ubuntu@sb4-k8s-1:~$ kubectl get pods --all-namespaces -o=wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system heapster—7Gb8cd7b5 -q7p42 1/1 Running 0 13m 10.42.213.49 sb4-k8s-1 kube-system kube-dns-5d7bM87c9-c6f67 3/3 Running 0 13m 10.42.181.110 sb4-k8s-1 kube-system kubernetes-dashboard-f9577fffd-kswjg 1/1 Running 0 13m 10.42.105.113 sb4-k8s-1 kube-system monitoring-grafana-997796fcf-vg9h9 1/1 Running 0 13m 10.42,141.58 sb4-k8s-1 kube-system monitoring-influxdb-56chd96b-hk66b 1/1 Running 0 13m 10.4Z.246.90 sb4-k8s-1 kube-system tiller-deploy-cc96d4f6b-v29k9 1/1 Running 0 13m 10.42.147.248 sb4-k8s-1 ubuntu@sb4-k8s-1:~$ 6. Validate helm is running at the right version. If not, an error like this will be displayed:: ubuntu@sb4-k8s-1:~$ helm list Error: incompatible versions c1ient[v2.9.1] server[v2.6.1] ubuntu@sb4-k8s-1:~$ 7. Upgrade the server-side component of helm (tiller) via `helm init --upgrade`:: ubuntu@sb4-k8s-1:~$ helm init --upgrade Creating /home/ubuntu/.helm Creating /home/ubuntu/.helm/repository Creating /home/ubuntu/.helm/repository/cache Creating /home/ubuntu/.helm/repository/local Creating /home/ubuntu/.helm/plugins Creating /home/ubuntu/.helm/starters Creating /home/ubuntu/.helm/cache/archive Creating /home/ubuntu/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /home/ubuntu/.helm. Tiller (the Helm server-side component) has been upgraded to the current version. Happy Helming! ubuntu@sb4-k8s-1:~$ ONAP Deployment via OOM ======================= Now that kubernetes and Helm are installed and configured you can prepare to deploy ONAP. Follow the instructions in the README.md_ or look at the official documentation to get started: - :ref:`quick-start-label` - deploy ONAP on an existing cloud - :ref:`user-guide-label` - a guide for operators of an ONAP instance