1 # Offline Kubernetes deployment with Ansible
3 This solution is generic offline deployment for Kubernetes which can host any
4 Kubernetes application in completely offline manner without internet connection.
6 It's developed for the purpose of hosting [ONAP](https://www.onap.org/) and proven
7 to work with that. ONAP is a very big Kubernetes application deployed by using
8 [Helm](https://helm.sh/) charts.
10 By default this solution supports deploying Kubernetes application with Helm charts.
12 ## Ansible execution/bootstrap
14 There are two ways how to easily run our ansible playbooks.
16 If you already have or can install a docker then you can build the provided `Dockerfile` for the ansible and run playbooks in the docker container.
18 Another way to deploy ansible is via chroot environment which is bundled together within this directory.
20 ### (Re)build docker image and/or chroot archive
22 Inside the docker directory is the `Dockerfile` and `build_ansible_image.sh` script. You can run `build_ansible_image.sh` script somewhere with internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into flat chroot environment.
24 Note: git must be installed before this script is called
26 Built image is exported into `ansible_chroot.tgz` archive in the same (./docker) directory.
28 This script has two optional arguments:
33 Note: if optional arguments are not used, docker image name will be set to `ansible` by default.
35 ### Launching ansible playbook using chroot environment
37 This is our default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive `ansible_chroot.tgz`.
39 It should be available in ./docker directory as the end-result of our packaging script or after manual running of `build_ansible_image.sh` referenced above.
41 All playbooks can be executed via `./run_playbook.sh` wrapper script.
43 To get more info about how the `./run_playbook.sh` wrapper script should be used, run:
48 The main purpose of this wrapper script is to provide ansible to machine where it was bootstrapped w/o need of installing additional packages, for displaying ansible-playbook command help one can run this following help command:
50 ./run_playbook.sh --help
53 #### Developers notes from chroot approach
55 * There are two scripts which work in tandem for creating and running chroot
56 * First can convert docker image into chroot directory
57 * Second will automate chrooting (necessary steps for chroot to work and cleanup)
58 * Both of them have help - just run:
62 $ ./create_docker_chroot.sh help
63 $ ./run_chroot.sh help
70 $ docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot
71 $ cat ./new_name_for_chroot/README.md
72 $ docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null
75 ### Launching ansible playbook using docker container (ALTERNATIVE APPROACH)
77 This option is here just to keep support for the older method which rely on a running docker service. For the offline deployment use the chroot option as indicated above.
79 You will not need `ansible_chroot.tgz` archive anymore, but the new requirement is a prebuilt docker image of ansible (based on our `Dockerfile`). It should be available in your local docker repository (otherwise the default name `ansible` may fetch unwanted image from default registry).
81 To trigger this functionality and run `ansible-playbook` inside a docker container instead of the chroot environment, you must first set the `ANSIBLE_DOCKER_IMAGE` variable. The value must be a name of the built ansible docker image - built using our provided `Dockerfile`.
83 Image name can be obtained again from `docker images` output.
87 REPOSITORY TAG IMAGE ID CREATED SIZE
88 ansible latest c44633617bfb 2 hours ago 127 MB
91 Usage is basically the same as with the default chroot way - the only difference is the existence of the env variable:
94 export ANSIBLE_DOCKER_IMAGE=ansible
95 ./run_playbook.sh <args>
98 or in a single command
101 ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh <args>
108 This chapter introduces all playbooks that each can be run independently, however
109 the order of running playbooks is implicitly fixed.
111 The complete offline deployment is handled by `site.yml` Ansible playbook.
112 This playbook contains imports for all the other playbooks needed to deploy
113 wanted Kubernetes application:
114 - `upload_resources.yml`
115 - `infrastructure.yml`
116 - `rancher_kubernetes.yml`
117 - `application.yml` - this is an application related playbook
121 Resource upload(nexus data, docker images, packages, etc.) is handled by playbook `upload_resources.yml`.
124 - setup nfs server for folder with resources (`resources.tar`) so it can be mounted on infrastructure server.
126 ### Infrastructure setup
128 Infrastructure deployment consists of:
129 - docker installed on all nodes
130 - dnsmasq docker container setup and start
131 - vncserver docker container setup and start
132 - nexus3 docker container - consists of docker images for application, Rancher and other resources
133 - nginx docker container - serves mainly as proxy for nexus, and also for providing other resources via HTTP/HTTPS
135 Infrastructure setup is handled by `infrastructure.yml` playbook.
138 - resources uploaded to infrastructure server (usually handled by `upload_resources.yml` playbook). All folders from resources should be located in directory defined in `{{ app_data_path }}` variable (e.g. `/opt/onap`).
140 ### Kubernetes cluster deployment
142 Kubernetes cluster is deployment is handled by `rancher_kubernetes.yml` playbook.
145 - infrastructure deployed by running `infrastructure.yml` playbook
147 ### Kubernetes application installation
149 Application installation is handled by `application.yml` playbook.
151 Application is dependent on resources:
152 - images, files, etc. included in resources.tar handled by upload_resources.yml playbook.
153 - Helm charts pointing to those images and files.
154 - Application specific configuration
156 Binary resources must be handled to be in place already during packaging and that
157 is instructed in **HERE/TODO**.
159 Application Helm charts and configuration can also be packaged into the package itself and is then available on ./application folder. However it can also be
160 copied there after installer package is deployed and before installing the application.
162 Application Helm charts and configuration is better described in [application/README.md](./application/README.md)
165 - Kubernetes cluster must be up and running i.e. `rancher_kubernetes.yml` playbook has been run.
168 To run ansible playbook call `run_playbook.sh` with same arguments as you would
169 call ansible-playbook.
171 Script `run_playbook.sh` must be executed from ansible directory to work correctly.
175 ./run_playbook.sh -i application/hosts.yml playbook.yml
180 This chapter describes how to start whole deployment of offline solution from scratch
181 after offline software package is installed (untarred) to target machine from which
182 Ansible will be run towards the target servers. Target servers are already in place
185 After offline software package is untarred, locate the ansible directory in there
186 having this README file you are reading.
192 In our playbooks we expects to have passwordless login to all ansible administrated nodes.
193 There is a helping playbook created to arrange that. Before setup playbook is executed
194 please insert IPs to inventory file as described below/
198 File **./inventory/hosts.yml** is an example inventory file that can be used as a
199 baseline. Copy inventory/hosts.yml file to ./application folder and modify needed
202 - `./application/hosts.yml`:
204 # This group contains hosts with all resources (binaries, packages, etc.)
208 # this key is supposed to be generated during setup.yml playbook execution
209 # change it just when you have better one working for all nodes
210 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
211 ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
217 ansible_host: 10.8.8.5
219 # This is group of hosts where nexus, nginx, dns and all other required
220 # services are running.
223 infrastructure-server:
224 ansible_host: 10.8.8.9
226 # This is group of hosts which are/will be part of Kubernetes cluster.
230 ansible_host: 10.8.8.13
239 after setting up inventory file helping setup playbook can be executed to arrange passwordless
240 login to all nodes. It make sense to launch it just when root user can't access all nodes in
241 passwordless way using ssh key. If root user has access already with ssh key, please override
242 just ansible_ssh_private_key_file in inventory file accordingly and there is no need to launch
245 If there is non-root user with ssh key access configured launch setup.yml in this way:
248 ./run_playbook.sh -i application/hosts.yml setup.yml -u <user> -e ansible_ssh_private_key_file=<key>
251 ./run_playbook.sh -i application/hosts.yml setup.yml -u cloud-user -e ansible_ssh_private_key_file=/root/.ssh/id_rsa
253 in case of password login to nodes, run playbook with --ask-pass param
256 ./run_playbook.sh -i application/hosts.yml setup.yml -u <user> --ask-pass
259 After running that playbook, passwordless login should be arranged for root user
260 using key ~/.ssh/offline_ssh_key which is preconfigured in inventory file.
263 **Deployment related configuration**:
265 Please check and review all options in `group_vars/all.yml` and `group_vars/infrastructure.yml`.
267 After inventory is setup and configuration is fine, all other playbooks (except application.yml) could be already be run and empty Kubernetes cluster (without application) could be setup.
269 **Application configuration**:
271 To deploy also application into Kubernetes, configure application configuration
272 needed by `application.yml` playbook.
273 See instructions on separate README.md file on [application/README.md](./application/README.md).
279 ./run_playbook.sh -i application/hosts.yml site.yml -e @application/application_configuration.yml