.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. Copyright 2019 Samsung Electronics Co., Ltd.
+.. Copyright 2021 Samsung Electronics Co., Ltd.
-OOM ONAP Offline Installer Package Build Guide
-=============================================================
+Offline Installer Package Build Guide
+=====================================
-This document is describing procedure for building offline installer packages. It is supposed to be triggered on server with internet connectivity and will download all artifacts required for ONAP deployment based on our static lists. The server used for the procedure in this guide is preferred to be separate build server.
+This document describes how to build offline installer packages. The build process should be triggered on a host with internet connectivity. It will retrieve all artifacts required for ONAP deployment based on both - static data list files and dynamically assembled ones. The host used for the procedure in this guide should be preferably a separate build server.
-Procedure was completely tested on RHEL 7.6 as it’s tested target platform, however with small adaptations it should be applicable also for other platforms.
+Procedure was completely tested on RHEL 7.6 as it’s the default target installation platform, however with small adaptations it should be applicable also for other platforms.
Some discrepancies when Centos 7.6 is used are described below as well.
-Part 1. Preparations
---------------------
-We assume that procedure is executed on RHEL 7.6 server with \~300G disc space, 16G+ RAM and internet connectivity
+Part 1. Prerequisites
+---------------------
-Some additional sw packages are required by ONAP Offline platform building tooling. in order to install them
-following repos has to be configured for RHEL 7.6 platform.
+We assume that procedure is executed on RHEL 7.6 server with \~300G disc space, 16G+ RAM and internet connectivity.
+Some additional software packages are required by ONAP Offline platform building tooling. In order to install them following repos have to be configured for RHEL 7.6 platform.
+
+.. note::
+ All commands stated in this guide are meant to be run in root shell.
+
::
############
# Register server
subscription-manager register --username <rhel licence name> --password <password> --auto-attach
- # required by special centos docker recommended by ONAP
+ # required by custom docker version recommended by ONAP
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# required by docker dependencies i.e. docker-selinux
::
- ToDo: newer download scripts needs to be verified on Centos with ONAP Dublin
-
##############
# Centos 7.6 #
##############
- # required by special centos docker recommended by ONAP
+ # required by custom docker version recommended by ONAP
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# enable epel repo for npm and jq
::
# install following packages
- yum install -y docker-ce-18.09.5 python-pip git createrepo expect nodejs npm jq
+ yum install -y docker-ce-19.03.15 git createrepo expect nodejs npm jq
- # twine package is needed by nexus blob build script
- pip install twine
+ # install Python 3
+ yum install -y python36 python36-pip
- # docker daemon must be running on host
+ # ensure docker daemon is running
service docker start
Then it is necessary to clone all installer and build related repositories and prepare the directory structure.
git clone https://gerrit.onap.org/r/oom/offline-installer onap-offline
cd onap-offline
- # install required pip packages for download scripts
- pip install -r ./build/download/requirements.txt
+ # install required pip packages for build and download scripts
+ pip3 install -r ./build/requirements.txt
+ pip3 install -r ./build/download/requirements.txt
Part 2. Download artifacts for offline installer
------------------------------------------------
-.. note:: Skip this step if you have already all necessary resources and continue with Part 3. Populate local nexus
+Generate the actual list of docker images that are defined within OOM helm charts. Run the docker-images-collector.sh script (check script for runtime dependencies) from cloned OOM repository.
-It's possible to download all artifacts in single ./download.py execution. Recently we improved reliability of download scripts
-so one might try following command to download most of the required artifacts in single shot.
+At the beginning of the generated list file there is the OOM repo commit sha from which it was created - the same commit reference
+should be used in **Part 4. Packages preparation** as *--application-repository_reference* option value.
-**Step1 - download wrapper script execution**
+Following example will create the list to the default path (*build/data_lists/onap_docker_images.list*):
::
- # following arguments are provided
- # all data lists are taken in ./build/data_lists/ folder
- # all resources will be stored in expected folder structure within ../resources folder
- # for more details refer to Appendix 1.
+ # clone the OOM repository
+ git clone https://gerrit.onap.org/r/oom -b <branch> --recurse-submodules /tmp/oom
+ #run the collector providing path the the project
+ ./build/creating_data/docker-images-collector.sh /tmp/oom/kubernetes/onap
- ./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra \
- --docker ./build/data_lists/rke_docker_images.list ../resources/offline_data/docker_images_for_nexus \
- --docker ./build/data_lists/onap_docker_images.list ../resources/offline_data/docker_images_for_nexus \
- --git ./build/data_lists/onap_git_repos.list ../resources/git-repo \
- --npm ./build/data_lists/onap_npm.list ../resources/offline_data/npm_tar \
- --rpm ./build/data_lists/onap_rpm.list ../resources/pkg/rhel \
- --pypi ./build/data_lists/onap_pip_packages.list ../resources/offline_data/pypi \
- --http ./build/data_lists/infra_bin_utils.list ../resources/downloads
+For the list of all available options check script usage info.
+.. note:: replace <branch> with OOM branch you want to build
-Alternatively, step-by-step procedure is described in Appendix 1.
+.. note:: docker-images-collector.sh script uses oom/kubernetes/onap/resources/overrides/onap-all.yaml file to find what subsystems are enabled. By default all subsystems are enabled there. Modify the file to disable some of them if needed.
-Following steps are still required and are not supported by current version of download.py script.
+.. note:: Skip this step if you have already all necessary resources and continue with **Part 3. Populate local nexus**
-**Step 2 - Building own dns image**
+Create repository containing OS packages to be installed on all nodes:
::
- # We are building our own dns image within our offline infrastructure
- ./build/creating_data/create_nginx_image/01create-image.sh /tmp/resources/offline_data/docker_images_infra
-
+ # run create_repo.sh script to download all required packages with their dependencies
+ # set destination directory for packages with '-d' parameter
+ # optionally use '-t' parameter to set target platform (host platform by default)
+ ./offline-installer/build/create_repo.sh -d $(pwd) -t centos|rhel|ubuntu
-**Step 3 - Http files**
+.. note:: If script fails due to permissions issue, it could be a problem with SeLinux. It can be fixed by running:
+ ::
-ToDo: complete and verified list of http files will come just during/after vFWCL testcase
+ # Change security context of directory
+ chcon -Rt svirt_sandbox_file_t $(pwd)
-
-**Step 4 - Binaries**
+Download all required binaries and docker images. Run download.py twice (as shown below) as it does not support mixing downloading docker images to a local directory and pulling them to local docker engine cache in one run. Docker images from *infra_docker_images.list* need to be saved to resources directory while the rest of the images need to be just pulled locally:
::
- # Binaries are downloaded in step one but some post processing is still needed.
- # This will be improved in future in installer itself
-
- tar -xf ../resources/downloads/helm-v2.12.3-linux-amd64.tar.gz linux-amd64/helm -O > ../resources/downloads/helm
- rm -f ../resources/downloads/helm-v2.12.3-linux-amd64.tar.gz
- mv ../resources/downloads/rke_linux-amd64 rke
-
-
-**Step 5 - Create repo**
-
-::
+ # all data lists are taken from ./build/data_lists/ folder by default
+ # all resources will be stored in expected folder structure within "../resources" folder
+ ./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra \
+ --http ./build/data_lists/infra_bin_utils.list ../resources/downloads
- createrepo ../resources/pkg/rhel
+ # second argument for --docker is not present, images are just pulled and cached
+ ./build/download/download.py --docker ./build/data_lists/rke_docker_images.list \
+ --docker ./build/data_lists/k8s_docker_images.list \
+ --docker ./build/data_lists/onap_docker_images.list
-This concludes SW download part required for ONAP offline platform creating.
Part 3. Populate local nexus
----------------------------
-Prerequisites:
-
-- All data lists and resources which are pushed to local nexus repository are available
-- Following ports are not occupied buy another service: 80, 8081, 8082, 10001
-- There's no docker container called "nexus"
-
-.. note:: In case you skipped the Part 2 for the artifacts download, please ensure that the copy of resources data are untarred in *./onap-offline/../resources/*
+In order to build nexus blob all docker images required for ONAP offline platform should be available locally (see Part 2).
-Whole nexus blob data will be created by running script build_nexus_blob.sh.
-It will load the listed docker images, run the Nexus, configure it as npm, pypi
-and docker repositories. Then it will push all listed npm and pypi packages and
-docker images to the repositories. After all is done the repository container
-is stopped.
+.. note:: In case you skipped the Part 2 for the artifacts download, please ensure that the onap docker images are cached and copy of resources data are untarred in *./onap-offline/../resources/*
-.. note:: build_nexus_blob.sh script is using docker, npm and pip data lists for building nexus blob. Unfortunatelly we now have 2 different docker data lists (RKE & ONAP). So we need to merge them as visible from following snippet. This problem will be fixed in OOM-1890
-
-You can run the script as following example:
-
-::
-
- # merge RKE and ONAP app data lists
- cat ./build/data_lists/rke_docker_images.list >> ./build/data_lists/onap_docker_images.list
-
- ./build/build_nexus_blob.sh
-
-.. note:: in current release scope we aim to maintain just single example data lists set, tags used in previous releases are not needed. Datalists are also covering latest versions verified by us despite user is allowed to build data lists on his own.
-
-Once the Nexus data blob is created, the docker images and npm and pypi
-packages can be deleted to reduce the package size as they won't be needed in
-the installation time:
-
-E.g.
+*build_nexus_blob.sh* script will run the Nexus container and configure it as docker repository. Then it will push all docker images from previously generated list to it. After that the repository container is stopped and its filesystem gets saved to resources directory.
::
- rm -f /tmp/resources/offline_data/docker_images_for_nexus/*
- rm -rf /tmp/resources/offline_data/npm_tar
- rm -rf /tmp/resources/offline_data/pypi
+ ./onap-offline/build/build_nexus_blob.sh
-Part 4. Application helm charts preparation and patching
---------------------------------------------------------
+It will load the listed docker images, run the Nexus, configure it as npm, pypi and docker repositories. Then it will push all listed docker images to the repositories. After all is done the repository container is stopped.
-This is about to clone oom repository and patch it to be able to use it
-offline. Use the following command:
+.. note:: By default the script uses data lists from ./build/data_lists/ directory and saves the blob to ../resources/nexus_data.
-::
-
- ./build/fetch_and_patch_charts.sh <helm charts repo> <commit/tag/branch> <patchfile> <target\_dir>
-
-For example:
+.. note:: By default the script uses "nexus" for the container name and binds 8081 and 8082 ports. Should those names/ports be already used please check the script options on how to customize them.
-::
-
- ./build/fetch_and_patch_charts.sh https://gerrit.onap.org/r/oom 0b904977dde761d189874d6dc6c527cd45928 /tmp/onap-offline/patches/onap.patch /tmp/oom-clone
-
-Part 5. Creating offline installation package
----------------------------------------------
-
-For the packagin itself it's necessary to prepare configuration. You can
-use ./build/package.conf as template or
-directly modify it.
-
-There are some parameters needs to be set in configuration file.
-Example values below are setup according to steps done in this guide to package ONAP.
-
-+---------------------------------------+------------------------------------------------------------------------------+
-| Parameter | Description |
-+=======================================+==============================================================================+
-| HELM_CHARTS_DIR | directory with Helm charts for the application |
-| | |
-| | Example: /tmp/oom-clone/kubernetes |
-+---------------------------------------+------------------------------------------------------------------------------+
-| APP_CONFIGURATION | application install configuration (application_configuration.yml) for |
-| | ansible installer and custom ansible role code directories if any. |
-| | |
-| | Example:: |
-| | |
-| | APP_CONFIGURATION=( |
-| | /tmp/onap-offline/config/application_configuration.yml |
-| | /tmp/onap-offline/patches/onap-patch-role |
-| | ) |
-| | |
-+---------------------------------------+------------------------------------------------------------------------------+
-| APP_BINARY_RESOURCES_DIR | directory with all (binary) resources for offline infra and application |
-| | |
-| | Example: /tmp/resources |
-+---------------------------------------+------------------------------------------------------------------------------+
-| APP_AUX_BINARIES | additional binaries such as docker images loaded during runtime [optional] |
-+---------------------------------------+------------------------------------------------------------------------------+
-
-Offline installer packages are created with prepopulated data via
-following command run from onap-offline directory
-
-::
- ./build/package.sh <project> <version> <packaging target directory>
-
-E.g.
-
-::
-
- ./build/package.sh onap 4.0.0 /tmp/package
-
-
-So in the target directory you should find tar files with
-
-::
-
- offline-<PROJECT_NAME>-<PROJECT_VERSION>-sw.tar
- offline-<PROJECT_NAME>-<PROJECT_VERSION>-resources.tar
- offline-<PROJECT_NAME>-<PROJECT_VERSION>-aux-resources.tar
-
-
-Appendix 1. Step-by-step download procedure
--------------------------------------------
-
-**Step 1 - docker images**
-
-::
-
- # This step will parse all 3 docker datalists (offline infrastructure images, rke k8s images & onap images)
- # and start building onap offline platform in /tmp/resources folder
-
- ./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra \
- --docker ./build/data_lists/rke_docker_images.list ../resources/offline_data/docker_images_for_nexus \
- --docker ./build/data_lists/onap_docker_images.list ../resources/offline_data/docker_images_for_nexus
-
-
-**Step 2 - building own dns image**
-
-::
-
- # We are building our own dns image within our offline infrastructure
- ./build/creating_data/create_nginx_image/01create-image.sh /tmp/resources/offline_data/docker_images_infra
-
-**Step 3 - git repos**
-
-::
-
- # Following step will download all git repos
- ./build/download/download.py --git ./build/data_lists/onap_git_repos.list ../resources/git-repo
-
-**Step 4 - http files**
-
-ToDo: complete and verified list of http files will come just during/after vFWCL testcase
+Part 4. Packages preparation
+----------------------------
-**Step 5 - npm packages**
+ONAP offline deliverable consist of 3 packages:
-::
++---------------------------------------+------------------------------------------------------------------------------------+
+| Package | Description |
++=======================================+====================================================================================+
+| sw_package.tar | Contains provisioning software and configuration for infrastructure and ONAP |
++---------------------------------------+------------------------------------------------------------------------------------+
+| resources_package.tar | Contains all binary data and config files needed to deploy infrastructure and ONAP |
++---------------------------------------+------------------------------------------------------------------------------------+
+| aux_package.tar | Contains auxiliary input files that can be added to ONAP |
++---------------------------------------+------------------------------------------------------------------------------------+
- # Following step will download all npm packages
- ./build/download/download.py --npm ./build/data_lists/onap_npm.list ../resources/offline_data/npm_tar
+All packages can be created using build/package.py script. Beside of archiving files gathered in the previous steps, script also builds docker images used on infra server.
-**Step 6 - binaries**
+From onap-offline directory run:
::
- # Following step will download and prepare rke, kubectl and helm binaries
- ./build/download/download.py --http ./build/data_lists/infra_bin_utils.sh ../resources/downloads
- tar -xf ../resources/downloads/helm-v2.12.3-linux-amd64.tar.gz linux-amd64/helm -O > ../resources/downloads/helm
- rm -f ../resources/downloads/helm-v2.12.3-linux-amd64.tar.gz
- mv ../resources/downloads/rke_linux-amd64 rke
+ ./build/package.py <helm charts repo> --build-version <version> --application-repository_reference <commit/tag/branch> --output-dir <target\_dir> --resources-directory <target\_dir>
-**Step 7 - rpms**
+For example:
::
- # Following step will download all rpms and create repo
- ./build/download/download.py --rpm ./build/data_lists/onap_rpm.list ../resources/pkg/rhel
+ ./build/package.py https://gerrit.onap.org/r/oom --application-repository_reference <branch> --output-dir /tmp/packages --resources-directory /tmp/resources
- createrepo ../resources/pkg/rhel
+.. note:: replace <branch> by branch you want to build
-**Step 8 - pip packages**
+Above command should produce following tar files in the target directory:
::
- # Following step will download all pip packages
- ./build/download/download.py --pypi ./build/data_lists/onap_pip_packages.list ../resources/offline_data/pypi
+ sw_package.tar
+ resources_package.tar
+ aux_package.tar
Code Review / oom / offline-installer.git / blobdiff