Before you dive into the installation you should prepare the offline installer itself - the installer consists of at least two packages/resources. You can read about it in the `Build Guide`_, which provides the instructions for creating them.
-This current version of the *Installation Guide* supports `Casablanca release`_.
+This current version of the *Installation Guide* supports `El Alto release`_.
-----
Part 1. Prerequisites
---------------------
-OOM ONAP deployment has certain hardware resource requirements - `Casablanca requirements`_:
+OOM ONAP deployment has certain hardware resource requirements - `El Alto requirements`_:
-- 14 VM (1 Rancher, 13 K8s nodes) - 8 vCPU - 16 GB RAM
-- 160 GB Storage
-
-That means the full deployment footprint is about ``224 GB RAM`` and ``112 vCPUs``. We will not follow strictly this setup due to such demanding resource consumption and so we will deploy our installation across four nodes (VMs) instead of fourteen. Our simplified setup is definitively not supported or recommended - you are free to diverge - you can follow the official guidelines or make completely different layout, but the minimal count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion <https://lists.onap.org/g/onap-discuss/topic/oom_110_kubernetes_pod/25213556>`_).
+Community recommended footprint from `El Alto requirements`_ page is 16 VMs ``224 GB RAM`` and ``112 vCPUs``. We will not follow strictly this setup due to such demanding resource consumption and so we will deploy our installation across four nodes (VMs) instead of sixteen. Our simplified setup is definitively not supported or recommended - you are free to diverge - you can follow the official guidelines or make completely different layout, but the minimal count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion <https://lists.onap.org/g/onap-discuss/topic/oom_110_kubernetes_pod/25213556>`_).
.. _oooi_installguide_preparations_k8s_cluster:
- nexus
- nginx proxy
- dns
- - rancher server
+ - kubernetes-etcd
+ - kubernetes-control-plane
+
+**NOTE:** kubernetes-* roles can be collocated directly with kubernetes nodes and not necessarily on infra node.
- **kubernetes node 1-3**::
- - rancher agent
+ - kubernetes worker
You don't need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in this table (or according to your needs as we hinted above):
Kubernetes cluster overview
^^^^^^^^^^^^^^^^^^^^^^^^^^^
-=================== ========= ==================== ============== ============ ===============
-KUBERNETES NODE OS NETWORK CPU RAM STORAGE
-=================== ========= ==================== ============== ============ ===============
-**infra-node** RHEL 7 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB``
-**kube-node1** RHEL 7 ``10.8.8.101/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
-**kube-node2** RHEL 7 ``10.8.8.102/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
-**kube-node3** RHEL 7 ``10.8.8.103/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
-SUM ``56 vCPUs`` ``152+ GB`` ``400 GB``
-================================================== ============== ============ ===============
+In El Alto we are using RKE as k8s orchestrator method, however everyone is free to diverge from this example and can set it up in own way omitting our rke playbook execution.
+
+=================== ================== ==================== ============== ============ ===============
+KUBERNETES NODE OS NETWORK CPU RAM STORAGE
+=================== ================== ==================== ============== ============ ===============
+**infra-node** RHEL/CentOS 7.6 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB``
+**kube-node1** RHEL/CentOS 7.6 ``10.8.8.101/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
+**kube-node2** RHEL/CentOS 7.6 ``10.8.8.102/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
+**kube-node3** RHEL/CentOS 7.6 ``10.8.8.103/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
+SUM ``56 vCPUs`` ``176+ GB`` ``400 GB``
+=========================================================== ============== ============ ===============
-Unfortunately, the offline installer supports only **RHEL 7.x** distribution as of now. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing. It is also worth knowing that the exact RHEL version (major and minor number - 7.6 for example) should match for the package build procedure and the target installation. That means: if you are building packages on RHEL 7.6 release your VMs should be RHEL 7.6 too.
+As of now, the offline installer supports only **RHEL 7.x** and **CentOS 7.6** distributions, with at least *@core* and *@base* package groups installed including *Mandatory* and *Default* package sets. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing.
We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from our *install-server* (below), which can be the hypervisor, *infra-node* or completely different machine. But in either of these cases the *install-server* must be able to connect over ssh to all of these nodes.
As was stated above you must have prepared the installer packages (names will differ - check out the `Build Guide`_):
-- offline-onap-3.0.1-resources.tar
-- offline-onap-3.0.1-aux-resources.tar
-- offline-onap-3.0.1-sw.tar
+- sw_package.tar
+- resources_package.tar
+- aux_package.tar
-**NOTE:** ``'offline-onap-3.0.1-aux-resources.tar'`` is optional and if you don't have use for it, you can ignore it.
+**NOTE:** ``'aux_package.tar'`` is optional and if you don't have use for it, you can ignore it.
We will store them in the ``/data`` directory on the *install-server* and then we will unpack the ``'sw'`` package to your home directory for example::
$ mkdir ~/onap-offline-installer
- $ tar -C ~/onap-offline-installer -xf /data/offline-onap-3.0.1-sw.tar
+ $ tar -C ~/onap-offline-installer -xf /data/sw_package.tar
.. _oooi_installguide_config_app:
You can see multiple files and directories inside - this is the *offline-installer*. It is implemented as a set of ansible playbooks.
-If you created the ``'sw'`` package according to the *Build Guide* then you should had have the ``'application'`` directory populated with at least the following files:
+If you created the ``'sw'`` package according to the *Build Guide* then you should have had the *offline-installer* populated with at least the following files:
-- ``application_configuration.yml``
-- ``hosts.yml``
+- ``application/application_configuration.yml``
+- ``inventory/hosts.yml``
-**NOTE:** The following paragraph describes a way how to create or fine-tune your own ``'application_configuration.yml'`` - we are discouraging you from executing this step. The recommended way is to use the packaged files inside the ``'application'`` directory.
-
-**NOT RECOMMENDED:** If for some reason you don't have these files inside the ``'application'`` directory or you simply want to do things the hard way then you can recreate them from their templates. It is better to keep the originals (templates) intact - so we will copy them to the ``'application'`` directory::
-
- $ cp ../config/application_configuration.yml application/
- $ cp inventory/hosts.yml application/
+Following paragraphs describe fine-tuning of ``'inventory.yml'`` and ``'application_configuration.yml'`` to reflect your VMs setup.
.. _oooi_installguide_config_hosts:
# This is group of hosts which are/will be part of Kubernetes cluster.
kubernetes:
- hosts:
- kubernetes-node-1:
- ansible_host: 10.8.8.19
- #ip of the node that it uses for communication with k8s cluster.
- cluster_ip: 10.8.8.19
+ children:
+ # This is a group of hosts containing kubernetes worker nodes.
+ kubernetes-node:
+ hosts:
+ kubernetes-node-1:
+ ansible_host: 10.8.8.19
+ #ip of the node that it uses for communication with k8s cluster.
+ cluster_ip: 10.8.8.19
+
+ # Group of hosts containing etcd cluster nodes.
+ # Defaults to infra.
+ kubernetes-etcd:
+ hosts:
+ infrastructure-server
+
+ # This is a group of hosts that are to be used as kubernetes control plane nodes.
+ # This means they host kubernetes api server, controller manager and scheduler.
+ # This example uses infra for this purpose, however note that any
+ # other host could be used including kubernetes nodes.
+ # cluster_ip needs to be set for hosts used as control planes.
+ kubernetes-control-plane:
+ hosts:
+ infrastructure-server
nfs-server:
hosts:
# This is group of hosts which are/will be part of Kubernetes cluster.
kubernetes:
- hosts:
- kubernetes-node-1:
- ansible_host: 10.8.8.101
- #ip of the node that it uses for communication with k8s cluster.
- cluster_ip: 10.8.8.101
- kubernetes-node-2:
- ansible_host: 10.8.8.102
- #ip of the node that it uses for communication with k8s cluster.
- cluster_ip: 10.8.8.102
- kubernetes-node-3:
- ansible_host: 10.8.8.103
- #ip of the node that it uses for communication with k8s cluster.
- cluster_ip: 10.8.8.103
+ children:
+ # This is a group of hosts containing kubernetes worker nodes.
+ kubernetes-node:
+ hosts:
+ kubernetes-node-1:
+ ansible_host: 10.8.8.101
+ #ip of the node that it uses for communication with k8s cluster.
+ cluster_ip: 10.8.8.101
+ kubernetes-node-2:
+ ansible_host: 10.8.8.102
+ #ip of the node that it uses for communication with k8s cluster.
+ cluster_ip: 10.8.8.102
+ kubernetes-node-3:
+ ansible_host: 10.8.8.103
+ #ip of the node that it uses for communication with k8s cluster.
+ cluster_ip: 10.8.8.103
+
+ # Group of hosts containing etcd cluster nodes.
+ # Defaults to infra.
+ kubernetes-etcd:
+ hosts:
+ infrastructure-server
+
+ # This is a group of hosts that are to be used as kubernetes control plane nodes.
+ # This means they host kubernetes api server, controller manager and scheduler.
+ # This example uses infra for this purpose, however note that any
+ # other host could be used including kubernetes nodes.
+ # cluster_ip needs to be set for hosts used as control planes.
+ kubernetes-control-plane:
+ hosts:
+ infrastructure-server
nfs-server:
hosts:
- ``app_data_path``
- ``aux_data_path``
- ``app_name``
+- ``timesync``
``'resource_dir'``, ``'resources_filename'`` and ``'aux_resources_filename'`` must correspond to the file paths on the *resource-host* (variable ``'resource_host'``), which is in our case the *install-server*.
-The ``'resource_dir'`` should be set to ``'/data'``, ``'resources_filename'`` to ``'offline-onap-3.0.1-resources.tar'`` and ``'aux_resources_filename'`` to ``'offline-onap-3.0.1-aux-resources.tar'``. The values should be the same as are in the `Installer packages`_ section.
+The ``'resource_dir'`` should be set to ``'/data'``, ``'resources_filename'`` to ``'resources_package.tar'`` and ``'aux_resources_filename'`` to ``'aux_package.tar'``. The values should be the same as are in the `Installer packages`_ section.
-``'app_data_path'`` is the absolute path on the *infra-node* to where the package ``'offline-onap-3.0.1-resources.tar'`` will be extracted and similarly ``'aux_data_path'`` is another absolute path for ``'offline-onap-3.0.1-aux-resources.tar'``. Both the paths are fully arbitrary, but they should point to the filesystem with enough space - the storage requirement in `Overview table of the kubernetes cluster`_.
+``'app_data_path'`` is the absolute path on the *infra-node* to where the package ``'resources_package.tar'`` will be extracted and similarly ``'aux_data_path'`` is another absolute path for ``'aux_package.tar'``. Both the paths are fully arbitrary, but they should point to the filesystem with enough space - the storage requirement in `Overview table of the kubernetes cluster`_.
**NOTE:** As we mentioned in `Installer packages`_ - the auxiliary package is not mandatory and we will not utilize it in here either.
-The last variable ``'app_name'`` should be short and descriptive. We will set it simply to: ``onap``.
+The ``'app_name'`` variable should be short and descriptive. We will set it simply to: ``onap``.
+
+The ``'timesync'`` variable is optional and controls synchronisation of the system clock on hosts. It should be configured only if a custom NTP server is available and needed. Such a time authority should be on a host reachable from all installation nodes. If this setting is not provided then the default behavior is to setup NTP daemon on infra-node and sync all kube-nodes' time with it.
+
+If you wish to provide your own NTP servers configure their IPs as follows::
+
+ timesync:
+ servers:
+ - <ip address of NTP_1>
+ - <...>
+ - <ip address of NTP_N>
+
+Another time adjustment related variables are ``'timesync.slewclock'`` and ``'timesync.timezone'`` .
+First one can have value of ``'true'`` or ``'false'`` (default). It controls whether (in case of big time difference compared to server) time should be adjusted gradually by slowing down or speeding up the clock as required (``'true'``) or in one step (``'false'``)::
+
+ timesync:
+ slewclock: true
-It can look all together something like this::
+Second one controls time zone setting on host. It's value should be time zone name according to tz database names with ``'Universal'`` being the default one::
+
+ timesync.
+ timezone: UTC
+
+``'timesync.servers'``, ``'timesync.slewclock'`` and ``'timesync.timezone'`` settings can be used independently.
+
+Final configuration can resemble the following::
resources_dir: /data
- resources_filename: offline-onap-3.0.1-resources.tar
+ resources_filename: resources_package.tar
app_data_path: /opt/onap
app_name: onap
+ timesync:
+ servers:
+ - 192.168.0.1
+ - 192.168.0.2
+ slewclock: true
+ timezone: UTC
.. _oooi_installguide_config_appconfig_overrides:
Helm chart value overrides
^^^^^^^^^^^^^^^^^^^^^^^^^^
-If there is a need to change onap settings such as managed openstack credentials, service ports, or even docker image versions used, you can do this by putting settings under the ``overrides`` key in ``application_configuration.yml``.
+In El Alto OOM charts are coming with all ONAP components disabled, this setting is also prepackaged within our sw_package.tar. Luckily there are multiple ways supported how to override this setting. It's also necessary for setting-up VIM specific entries and basically to configure any stuff with non default values.
+
+First option is to use ``overrides`` key in ``application_configuration.yml``.
These settings will override helm values originally stored in ``values.yaml`` files in helm chart directories.
For example, the following lines could be appended to ``application_configuration.yml`` to set up managed openstack credentials for onap's so component::
openStackKeyStoneUrl: "keystone_url"
openStackEncryptedPasswordHere: "encrypted_password"
+In addition or alternatively to that one can configure ``helm_override_files`` key, which is new feature implemented in Change-Id: I8b8ded38b39aa9a75e55fc63fa0e11b986556cb8.
+
.. _oooi_installguide_config_ssh:
SSH authentication
You can use the ansible playbook ``'setup.yml'`` like this::
- $ ./run_playbook.sh -i application/hosts.yml setup.yml -u root --ask-pass
+ $ ./run_playbook.sh -i inventory/hosts.yml setup.yml -u root --ask-pass
You will be asked for password per each node and the playbook will generate a unprotected ssh key-pair ``'~/.ssh/offline_ssh_key'``, which will be distributed to the nodes.
If you generated the ssh key manually then you can now run the ``'setup.yml'`` playbook like this and achieve the same result as in the first execution::
- $ ./run_playbook.sh -i application/hosts.yml setup.yml
+ $ ./run_playbook.sh -i inventory/hosts.yml setup.yml
This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason.
Installation is actually very straightforward now::
- $ ./run_playbook.sh -i application/hosts.yml -e @application/application_configuration.yml site.yml
+ $ ./run_playbook.sh -i inventory/hosts.yml -e @application/application_configuration.yml site.yml
This will take a while so be patient.
- ``upload_resources.yml``
- ``infrastructure.yml``
-- ``rancher_kubernetes.yml``
+- ``rke.yml``
- ``application.yml``
----
.. _oooi_installguide_postinstall:
-Part 4. Postinstallation and troubleshooting
---------------------------------------------
+Part 4. Post-installation and troubleshooting
+---------------------------------------------
-After all the playbooks are finished, it will still take a lot of time until all pods will be up and running. You can monitor your newly created kubernetes cluster for example like this::
+After all of the playbooks are run successfully, it will still take a lot of time until all pods are up and running. You can monitor your newly created kubernetes cluster for example like this::
- $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.4 # tailor this command to connect to your infra-node
+ $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.100 # tailor this command to connect to your infra-node
$ watch -d -n 5 'kubectl get pods --all-namespaces'
+Alternatively you can monitor progress with ``helm_deployment_status.py`` script located in offline-installer directory. Transfer it to infra-node and run::
+
+ $ python helm_deployment_status.py -n <namespace_name> # namespace defaults to onap
+
+To automatically verify functionality with healthchecks after deployment becomes ready or after timeout period expires, append ``-hp`` switch followed by the full path to the healthcheck script and ``--health-mode`` optional switch with appropriate mode supported by that script (``health`` by default, ``--help`` displays available modes)::
+
+ $ python helm_deployment_status.py -hp <app_data_path>/<app_name>/helm_charts/robot/ete-k8s.sh --health-mode <healthcheck mode>
+
+It is strongly recommended to tailor ``helm_deployment_status.py`` to your needs since default values might not be what you'd expect. The defaults can be displayed with ``--help`` switch.
Final result of installation varies based on number of k8s nodes used and distribution of pods. In some dev envs we quite frequently hit problems with not all pods properly deployed. In successful deployments all jobs should be in successful state.
This can be verified using ::
$ cd <app_data_path>/<app_name>/helm_charts/robot
$ ./ete-k8s.sh onap health
+For better work with terminal screen and jq packages were added . It can be installed from resources directory.
+
+Screen is a terminal multiplexer. With screen it is possible to have more terminal instances active. Screen as well keeps active SSH connections even terminal is closed.
+
+Jq can be used for editing json data format as output of kubectl. For example jq was used to troubleshoot `SDNC-739 (UEB - Listener in Crashloopback) <https://jira.onap.org/browse/SDNC-739/>`_ ::
+
+ $ kubectl -n onap get job onap-sdc-sdc-be-config-backend -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl -n onap replace --force -f -
-----
-----
.. _Build Guide: ./BuildGuide.rst
-.. _Casablanca requirements: https://onap.readthedocs.io/en/casablanca/guides/onap-developer/settingup/index.html#installing-onap
-.. _Casablanca release: https://docs.onap.org/en/casablanca/release/
+.. _El Alto requirements: https://onap.readthedocs.io/en/elalto/guides/onap-developer/settingup/index.html#installing-onap
+.. _El Alto release: https://docs.onap.org/en/elalto/release/
.. _OOM ONAP: https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project
Code Review / oom / offline-installer.git / blobdiff