docs/BuildGuide.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3 .. Copyright 2019 Samsung Electronics Co., Ltd.
   4
   5 OOM ONAP Offline Installer Package Build Guide
   6 =============================================================
   7
   8 This document is describing procedure for building offline installer packages. It is supposed to be triggered on server with internet connectivity and will download all artifacts required for ONAP deployment based on our static lists. The server used for the procedure in this guide is preferred to be separate build server.
   9
  10 Procedure was completely tested on RHEL 7.6 as it’s tested target platform, however with small adaptations it should be applicable also for other platforms.
  11 Some discrepancies when Centos 7.6 is used are described below as well.
  12
  13 Part 1. Preparations
  14 --------------------
  15
  16 We assume that procedure is executed on RHEL 7.6 server with \~300G disc space, 16G+ RAM and internet connectivity
  17
  18 Some additional sw packages are required by ONAP Offline platform building tooling. in order to install them
  19 following repos has to be configured for RHEL 7.6 platform.
  20
  21
  22
  23 ::
  24
  25     ############
  26     # RHEL 7.6 #
  27     ############
  28
  29     # Register server
  30     subscription-manager register --username <rhel licence name> --password <password> --auto-attach
  31
  32     # required by special centos docker recommended by ONAP
  33     yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  34
  35     # required by docker dependencies i.e. docker-selinux
  36     subscription-manager repos --enable=rhel-7-server-extras-rpms
  37
  38     # epel is required by npm within blob build
  39     rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
  40
  41 Alternatively
  42
  43 ::
  44
  45    ToDo: newer download scripts needs to be verified on Centos with ONAP Dublin
  46
  47    ##############
  48    # Centos 7.6 #
  49    ##############
  50
  51    # required by special centos docker recommended by ONAP
  52    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  53
  54    # enable epel repo for npm and jq
  55    yum install -y epel-release
  56
  57 Subsequent steps are the same on both platforms:
  58
  59 ::
  60
  61     # install following packages
  62     yum install -y docker-ce-18.09.5 python-pip git createrepo expect nodejs npm jq
  63
  64     # twine package is needed by nexus blob build script
  65     pip install twine
  66
  67     # docker daemon must be running on host
  68     service docker start
  69
  70 Then it is necessary to clone all installer and build related repositories and prepare the directory structure.
  71
  72 ::
  73
  74     # prepare the onap build directory structure
  75     cd /tmp
  76     git clone https://gerrit.onap.org/r/oom/offline-installer onap-offline
  77     cd onap-offline
  78
  79     # install required pip packages for download scripts
  80     pip install -r ./build/download/requirements.txt
  81
  82 Part 2. Download artifacts for offline installer
  83 ------------------------------------------------
  84
  85 .. note:: Skip this step if you have already all necessary resources and continue with Part 3. Populate local nexus
  86
  87 It's possible to download all artifacts in single ./download.py execution. Recently we improved reliability of download scripts
  88 so one might try following command to download most of the required artifacts in single shot.
  89
  90 **Step1 - download wrapper script execution**
  91
  92 ::
  93
  94         # following arguments are provided
  95         # all data lists are taken in ./build/data_lists/ folder
  96         # all resources will be stored in expected folder structure within ../resources folder
  97         # for more details refer to Appendix 1.
  98
  99         ./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra --docker ./build/data_lists/rke_docker_images.list ../resources/offline_data/docker_images_for_nexus --docker ./build/data_lists/onap_docker_images.list ../resources/offline_data/docker_images_for_nexus --git ./build/data_lists/onap_git_repos.list ../resources/git-repo --npm ./build/data_lists/onap_npm.list ../resources/offline_data/npm_tar --rpm ./build/data_lists/onap_rpm.list ../resources/pkg/rhel
 100
 101
 102 Alternatively, step-by-step procedure is described in Appendix 1.
 103
 104 Following steps are still required and are not supported by current version of download.py script.
 105
 106 **Step 2 - Building own dns image**
 107
 108 ::
 109
 110         # We are building our own dns image within our offline infrastructure
 111         ./build/creating_data/create_nginx_image/01create-image.sh /tmp/resources/offline_data/docker_images_infra
 112
 113
 114 **Step 3 - Http files**
 115
 116 ToDo: complete and verified list of http files will come just during/after vFWCL testcase
 117
 118
 119 **Step 4 - Binaries**
 120
 121 ::
 122
 123        # Following step will download and prepare rke, kubectl and helm binaries
 124        # there is some post-processing needed therefore its not very convenient to add support for this step into main download.py script
 125        ./build/download/download-bin-tools.sh ../resources/downloads
 126
 127 **Step 5 - Create repo**
 128
 129 ::
 130
 131       createrepo ../resources/pkg/rhel
 132
 133 **Step 6 - pip packages**
 134
 135 Todo: will be incorporated into download.py in near future
 136
 137 ::
 138
 139       # Following step will download all pip packages
 140       ./build/download/download-pip.sh ./build/data_lists/onap_pip_packages.list ../resources/offline_data/pypi
 141
 142
 143 This concludes SW download part required for ONAP offline platform creating.
 144
 145 Part 3. Populate local nexus
 146 ----------------------------
 147
 148 Prerequisites:
 149
 150 - All data lists and resources which are pushed to local nexus repository are available
 151 - Following ports are not occupied buy another service: 80, 8081, 8082, 10001
 152 - There's no docker container called "nexus"
 153
 154 .. note:: In case you skipped the Part 2 for the artifacts download, please ensure that the copy of resources data are untarred in *./onap-offline/../resources/*
 155
 156 Whole nexus blob data will be created by running script build_nexus_blob.sh.
 157 It will load the listed docker images, run the Nexus, configure it as npm, pypi
 158 and docker repositories. Then it will push all listed npm and pypi packages and
 159 docker images to the repositories. After all is done the repository container
 160 is stopped.
 161
 162 .. note:: build_nexus_blob.sh script is using docker, npm and pip data lists for building nexus blob. Unfortunatelly we now have 2 different docker data lists (RKE & ONAP). So we need to merge them as visible from following snippet. This problem will be fixed in OOM-1890
 163
 164 You can run the script as following example:
 165
 166 ::
 167
 168         # merge RKE and ONAP app data lists
 169         cat ./build/data_lists/rke_docker_images.list >> ./build/data_lists/onap_docker_images.list
 170
 171         ./build/build_nexus_blob.sh
 172
 173 .. note:: in current release scope we aim to maintain just single example data lists set, tags used in previous releases are not needed. Datalists are also covering latest versions verified by us despite user is allowed to build data lists on his own.
 174
 175 Once the Nexus data blob is created, the docker images and npm and pypi
 176 packages can be deleted to reduce the package size as they won't be needed in
 177 the installation time:
 178
 179 E.g.
 180
 181 ::
 182
 183     rm -f /tmp/resources/offline_data/docker_images_for_nexus/*
 184     rm -rf /tmp/resources/offline_data/npm_tar
 185     rm -rf /tmp/resources/offline_data/pypi
 186
 187 Part 4. Application helm charts preparation and patching
 188 --------------------------------------------------------
 189
 190 This is about to clone oom repository and patch it to be able to use it
 191 offline. Use the following command:
 192
 193 ::
 194
 195   ./build/fetch_and_patch_charts.sh <helm charts repo> <commit/tag/branch> <patchfile> <target\_dir>
 196
 197 For example:
 198
 199 ::
 200
 201   ./build/fetch_and_patch_charts.sh https://gerrit.onap.org/r/oom 0b904977dde761d189874d6dc6c527cd45928 /tmp/onap-offline/patches/onap.patch /tmp/oom-clone
 202
 203 Part 5. Creating offline installation package
 204 ---------------------------------------------
 205
 206 For the packagin itself it's necessary to prepare configuration. You can
 207 use ./build/package.conf as template or
 208 directly modify it.
 209
 210 There are some parameters needs to be set in configuration file.
 211 Example values below are setup according to steps done in this guide to package ONAP.
 212
 213 +---------------------------------------+------------------------------------------------------------------------------+
 214 | Parameter                             | Description                                                                  |
 215 +=======================================+==============================================================================+
 216 | HELM_CHARTS_DIR                       | directory with Helm charts for the application                               |
 217 |                                       |                                                                              |
 218 |                                       | Example: /tmp/oom-clone/kubernetes                                           |
 219 +---------------------------------------+------------------------------------------------------------------------------+
 220 | APP_CONFIGURATION                     | application install configuration (application_configuration.yml) for        |
 221 |                                       | ansible installer and custom ansible role code directories if any.           |
 222 |                                       |                                                                              |
 223 |                                       | Example::                                                                    |
 224 |                                       |                                                                              |
 225 |                                       |  APP_CONFIGURATION=(                                                         |
 226 |                                       |     /tmp/onap-offline/config/application_configuration.yml                   |
 227 |                                       |     /tmp/onap-offline/patches/onap-patch-role                                |
 228 |                                       |  )                                                                           |
 229 |                                       |                                                                              |
 230 +---------------------------------------+------------------------------------------------------------------------------+
 231 | APP_BINARY_RESOURCES_DIR              | directory with all (binary) resources for offline infra and application      |
 232 |                                       |                                                                              |
 233 |                                       | Example: /tmp/resources                                                      |
 234 +---------------------------------------+------------------------------------------------------------------------------+
 235 | APP_AUX_BINARIES                      | additional binaries such as docker images loaded during runtime   [optional] |
 236 +---------------------------------------+------------------------------------------------------------------------------+
 237
 238 Offline installer packages are created with prepopulated data via
 239 following command run from onap-offline directory
 240
 241 ::
 242
 243   ./build/package.sh <project> <version> <packaging target directory>
 244
 245 E.g.
 246
 247 ::
 248
 249   ./build/package.sh onap 4.0.0 /tmp/package
 250
 251
 252 So in the target directory you should find tar files with
 253
 254 ::
 255
 256   offline-<PROJECT_NAME>-<PROJECT_VERSION>-sw.tar
 257   offline-<PROJECT_NAME>-<PROJECT_VERSION>-resources.tar
 258   offline-<PROJECT_NAME>-<PROJECT_VERSION>-aux-resources.tar
 259
 260
 261 Appendix 1. Step-by-step download procedure
 262 -------------------------------------------
 263
 264 **Step 1 - docker images**
 265
 266 ::
 267
 268         # This step will parse all 3 docker datalists (offline infrastructure images, rke k8s images & onap images)
 269         # and start building onap offline platform in /tmp/resources folder
 270
 271         ./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra --docker ./build/data_lists/rke_docker_images.list ../resources/offline_data/docker_images_for_nexus --docker ./build/data_lists/onap_docker_images.list ../resources/offline_data/docker_images_for_nexus
 272
 273
 274 **Step 2 - building own dns image**
 275
 276 ::
 277
 278         # We are building our own dns image within our offline infrastructure
 279         ./build/creating_data/create_nginx_image/01create-image.sh /tmp/resources/offline_data/docker_images_infra
 280
 281 **Step 3 - git repos**
 282
 283 ::
 284
 285         # Following step will download all git repos
 286         ./build/download/download.py --git ./build/data_lists/onap_git_repos.list ../resources/git-repo
 287
 288 **Step 4 - http files**
 289
 290 ToDo: complete and verified list of http files will come just during/after vFWCL testcase
 291
 292 **Step 5 - npm packages**
 293
 294 ::
 295
 296         # Following step will download all npm packages
 297         ./build/download/download.py --npm ./build/data_lists/onap_npm.list ../resources/offline_data/npm_tar
 298
 299 **Step 6 - binaries**
 300
 301 ::
 302
 303        # Following step will download and prepare rke, kubectl and helm binaries
 304        ./build/download/download-bin-tools.sh ../resources/downloads
 305
 306 **Step 7 - rpms**
 307
 308 ::
 309
 310       # Following step will download all rpms and create repo
 311       ./build/download/download.py --rpm ./build/data_lists/onap_rpm.list ../resources/pkg/rhel
 312
 313       createrepo ../resources/pkg/rhel
 314
 315 **Step 8 - pip packages**
 316
 317 Todo: new python script might be created for that part as well
 318
 319 ::
 320
 321       # Following step will download all pip packages
 322       ./build/download/download-pip.sh ./build/data_lists/onap_pip_packages.list ../resources/offline_data/pypi
 323