--- /dev/null
+include environment.config
+
+generate:
+ ./generate.sh $(REPOSITORY_URL_HTTPS) $(REPOSITORY_BRANCH) $(SCHEMAS_LOCATION) $(VENDOR_NAME) $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME) $(SNIPPET_FILENAME)
+
+install:
+ ./install.sh $(SPEC_CONFIGMAP_FILENAME) $(K8S_CONFIGMAP_NAME)
+
--- /dev/null
+StndDefined schemas ConfigMap generator
+=======================================
+
+## Description
+StndDefined schemas Kubernetes ConfigMap generator is a Makefile with two targets running shell scripts: 'generate' and
+'install'. This Makefile may be used by VES Collector users to generate and install ConfigMap containing schemas
+and mapping file for stndDefined validation in VES pod. Additionally script creates file with snippets containing
+auto-generated configuration of volumes finally mounted in VES deployment. Process of generation of ConfigMap spec file
+is configurable via environment.config file.
+
+Generator files are available in oom/utils ONAP Gerrit repository under external-schema-repo-generator folder.
+
+## Requirements and limitations
+
+### Environment
+Target *generate* from Makefile requires stable internet connection to properly clone Git repositories.
+Target *install* should be ran on the RKE node, so it can create ConfigMap in the same Kubernetes environment as VES Pod
+is installed. It is possible to generate a spec in one environment and move it together with all ConfigMap generator
+files to the RKE environment.
+
+### Repository limitations
+When running the script branches from selected repository are being downloaded. Time of script execution depends mostly
+on repository size and number of schemas. All .yaml files from selected directory in Git repository will be considered
+as schemas and attached to ConfigMap spec.
+
+### Generator tool files integration
+It is recommended to consider files of this tool as a unity and not split them when moving generator through
+environments. Generator tool files that are required are:
+- Makefile
+- install.sh
+- generate.sh
+- environment.config
+
+## Instruction
+
+### Parameters description
+Before running any target from Makefile, configuration in *environment.config* must be properly prepared. Description of
+the configurable properties is below.
+
+- **SPEC_CONFIGMAP_FILENAME** - Filename name of ConfigMap spec that will be generated.
+- **K8S_CONFIGMAP_NAME** - Kubernetes name of ConfigMap that will be generated and installed.
+- **SNIPPET_FILENAME** - Filename of snippet with autogenerated content that should be added to VES deployment.
+
+- **REPOSITORY_URL_HTTPS** - URL to remote Git repository which lets cloning repository via HTTPS.
+- **REPOSITORY_BRANCH** - Valid branch from selected Git repository which contains schemas desired to mount. Script
+accepts an array of branches from which schemas will be collected. To pass an array split branch names with space and
+cover list in quotation marks.
+- **SCHEMAS_LOCATION** - Path to schemas directory on selected Git repo. All YAML files from this repository will be
+considered as schema and added to ConfigMap.
+
+- **VENDOR_NAME** - Name of organisation delivering schemas, used only for naming of schemas destination directory in
+VES.
+
+### Running commands
+
+To run ConfigMap spec generation as well as snippet file used for mounting ConfigMap to specific Deployment use:
+
+```
+make generate
+```
+
+To run ConfigMap installation in Kubernetes use:
+
+**NOTE**: Remember about running this command on RKE node.
+
+```
+make install
+```
+
+**NOTE**: It is possible that ConfigMap with selected K8S_CONFIGMAP_NAME already exists in Kubernetes. In such situation
+either regenerate spec with new K8S_CONFIGMAP_NAME or remove existing ConfigMap from Kubernetes and install spec again.
+To remove ConfigMap from Kubernetes use:
+```
+kubectl -n onap delete configmap <CONFIGMAP_NAME>
+```
+
+## ConfigMap validation
+After running the script ConfigMap spec file is generated in current working directory.
+Spec file can be manually validated via any text editor. The last file included in spec is schema-map.json file with
+mappings of external URLs to prepared local URLs.
+
+To check whether it has been created use command:
+
+```
+kubectl -n onap get configmap | grep <CONFIGMAP_NAME>
+```
+
+A ConfigMap with configured name should be visible on the list.
+
+## Mounting ConfigMap into VES Collector
+
+To mount created ConfigMap in VES, its deployment must be edited. It can be done with:
+```
+kubectl -n onap edit deployment dep-dcae-ves-collector
+```
+
+Snippets with content that should be added to VES deployment are generated in file with name setup in configuration file
+under SNIPPET_FILENAME property. No extra configuration in VES is needed when using them.
+
+**NOTE**: When using Vi text editor for deployment edition, correct input mode must be set to keep proper indentation
+when pasting snippets. Use ``:set paste `` to turn paste mode on. To close paste mode use ``:set nopaste``
+
+1. Add volumeMounts element
+
+ In spec.template.spec.containers[0].volumeMounts add new list element:
+
+ **NOTE**: spec.template.spec.containers[0] should be a container with the image:
+ *nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector::x.x.x*.
+ It should be the first container, but make sure that the correct container is being edited.
+
+ ```
+ volumeMounts:
+ - ...
+ - mountPath: /opt/app/VESCollector/etc/externalRepoCustom
+ name: custom-3gpp-schemas
+ ```
+
+ - mountPath - Directory context for schemas. **NOTE**: must be the same as configuration of VES Collector property
+ *collector.externalSchema.schemasLocation* in *collector.properties*. This property might be modified via Consul UI,
+ later after changes in deployment.
+
+ - name - Name of ConfigMap volume. Must be the same as set in the 2. step in *name* field.
+
+2. Add volumes element
+
+ In spec.template.spec.volumes add a new list element with all desired to mount schemas from ConfigMap in
+ *items* list. *key* are file names from generated previously spec and *path* is relative path from directory set up
+ in step 1 as *mountPath*.
+
+ Sample spec.template.spec.volumes content:
+
+ ```
+ volumes:
+ - configMap:
+ defaultMode: 420
+ items:
+ - key: schema-map.json
+ path: schema-map.json
+ - key: SA88-Rel16-faultMnS.yaml
+ path: 3gpp/rep/sa5/data-models/SA88-Rel16/OpenAPI/faultMnS.yaml
+ - ...
+ name: stnd-defined-configmap
+ name: custom-3gpp-schemas
+ - ...
+ ```
+ Fields description:
+ - name - Name of ConfigMap volume. Must be the same as set in the 1. step in *name* field.
+ - configMap.name - name of installed Kubernetes ConfigMap described in K8S_CONFIGMAP_NAME configuration setting
+ - configMap.items[].key - name of mounted file from installed previously ConfigMap.
+ **NOTE**: Not every schema from ConfigMap must be listed in *items* (warning message will be logged on VES startup if
+ so), but mapping file named as described in MAPPING_FILE_NAME setting is required for correct stndDefined validation
+ in VES.
+ - configMap.items[].path: Relative path from *mountPath* from step 1 which describes location of schema location in
+ VES container.
+ **NOTE 1**: For correct schemas detection in VES Collector *path* of each schema must be the same as its localURL in
+ mapping file. Mapping file is included as the last file in generated ConfigMap spec.
+ **NOTE 2**: For correct mapping file detection in VES Collector its *path* must be the same as in property
+ *collector.externalSchema.mappingFileLocation* in *collector.properties*. This property might be modified via Consul
+ UI, later after changes in deployment.
+
+3. Save and close an editor, K8S will automatically detect changes, terminate old VES Pod and deploy new one with
+mounted ConfigMap. Correctness of new VES Pod initialization and mounting ConfigMap can be tracked using
+`kubectl -n onap describe pod <VES_POD_NAME>`.
+
+To check if mounted schemas and VES properties configuration are correctly aligned see logs of VES. Each schema which is
+present in mapping file, but is not detected by VES will be logged on application startup e.g.:
+```
+2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Local schema resource missing. Schema file with path /opt/app/VESCollector/etc/externalRepoCustom/3gpp/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml has not been found.
+2020-10-01 11:46:55.872 WARN 24 [ Thread-5] o.o.d.s.s.s.e.s.m.s.UrlMapperFactory : Mapping for publicURL ("https://forge.3gpp.org/rep/sa5/MnS/tree/master/OpenAPI/5gcNrm.yaml") will not be added to validator.
+```
+
+To see logs of VES use:
+```
+kubectl -n onap logs <VES_POD_NAME>
+```
--- /dev/null
+# Configuration file for externalSchemaRepoGenerator Makefile
+# Remember about escaping special characters, e.g. in REPOSITORY_URL
+
+# ConfigMap configuration
+SPEC_CONFIGMAP_FILENAME=stndDefined-schemas-configmap-spec.yaml
+K8S_CONFIGMAP_NAME=stnd-defined-configmap
+SNIPPET_FILENAME=ves-snippet.md
+
+# Source repository configuration
+REPOSITORY_URL_HTTPS=https\://forge.3gpp.org/rep/sa5/MnS.git
+REPOSITORY_BRANCH="SA88-Rel16 SA89-Rel17"
+SCHEMAS_LOCATION=OpenAPI
+
+# Vendor description
+VENDOR_NAME=3gpp
\ No newline at end of file
--- /dev/null
+#!/bin/sh
+
+# ============LICENSE_START=======================================================
+# OOM
+# ================================================================================
+# Copyright (C) 2020 Nokia. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+
+
+# Arguments renaming
+arguments_number=$#
+repo_url=$1
+branches=$2
+schemas_location=$3
+vendor=$4
+configmap_filename=$5
+configmap_name=$6
+snippet_filename=$7
+
+# Constants
+SCHEMA_MAP_FILENAME="schema-map.json"
+SUCCESS_CODE=0
+TREE=tree
+EXPECTED_1_ARG=1
+EXPECTED_7_ARGS=7
+INDENTATION_LEVEL_1=1
+INDENTATION_LEVEL_2=2
+INDENTATION_LEVEL_3=3
+INDENTATION_LEVEL_4=4
+INDENTATION_LEVEL_5=5
+
+# Variables
+tmp_location=$(mktemp -d)
+valid_branches=""
+
+# Indents each line of string by adding indent_size*indent_string spaces on the beginning
+# Optional argument is indent_string level, default: 1
+# correct usage example:
+# echo "Sample Text" | indent_string 2
+indent_string() {
+ indent_size=2
+ indent_string=1
+ if [ -n "$1" ]; then indent_string=$1; fi
+ pr -to $(expr "$indent_string" \* "$indent_size")
+}
+
+# Checks whether number of arguments is valid
+# $1 is actual number of arguments
+# $2 is expected number of arguments
+check_arguments() {
+ if [ "$1" -ne "$2" ]; then
+ echo "Incorrect number of arguments"
+ exit 1
+ fi
+}
+
+# Clones all branches selected in $BRANCH from $repo_url
+clone_repo() {
+ for actual_branch in $branches; do
+ clone_branch "$actual_branch"
+ done
+}
+
+# Clones single branch $1 from $repo_url.
+# $1 - branch name
+clone_branch() {
+ check_arguments $# $EXPECTED_1_ARG
+ if [ -d $tmp_location/"$1" ]; then
+ echo "Skipping cloning repository."
+ echo "Branch $1 has already been cloned in the directory ./$tmp_location/$1"
+ echo "To redownload branch remove ./$tmp_location/$1."
+ else
+ echo "Cloning repository from branch $1"
+ git clone --quiet --single-branch --branch "$1" "$repo_url" "$tmp_location/$1" 2>/dev/null
+ result=$?
+ if [ $result -ne $SUCCESS_CODE ] ; then
+ echo "Problem with cloning branch $1."
+ echo "Branch $1 will not be added to spec."
+ else
+ valid_branches="${valid_branches} $1"
+ fi
+ fi
+}
+
+# Creates file with name $configmap_filename
+# Inserts ConfigMap metadata and sets name as $configmap_name
+add_config_map_metadata() {
+ echo "Creating ConfigMap spec file: $configmap_filename"
+ cat << EOF > "$configmap_filename"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: $configmap_name
+ labels:
+ name: $configmap_name
+ namespace: onap
+data:
+EOF
+}
+
+# For each selected branch:
+# clones the branch from repository,
+# adds schemas from branch to ConfigMap spec
+add_schemas() {
+ for actual_branch in $valid_branches; do
+ echo "Adding schemas from branch $actual_branch to spec"
+ add_schemas_from_branch "$actual_branch"
+ done
+}
+
+# Adds schemas from single branch to spec
+# $1 - branch name
+add_schemas_from_branch() {
+ check_arguments $# $EXPECTED_1_ARG
+ schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}')
+ for schema in $schemas; do
+ echo "$1-$(basename $schema): |-" | indent_string $INDENTATION_LEVEL_1
+ cat "$schema" | indent_string $INDENTATION_LEVEL_2
+ done
+} >> "$configmap_filename"
+
+# Generates mapping file for collected schemas directly in spec
+generate_mapping_file() {
+ echo "Generating mapping file in spec"
+ echo "$SCHEMA_MAP_FILENAME"": |-" | indent_string $INDENTATION_LEVEL_1 >> "$configmap_filename"
+ echo "[" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename"
+
+ for actual_branch in $valid_branches; do
+ echo "Adding mappings from branch: $actual_branch"
+ add_mappings_from_branch "$actual_branch"
+ done
+
+ truncate -s-2 "$configmap_filename"
+ echo "" >> "$configmap_filename"
+ echo "]" | indent_string $INDENTATION_LEVEL_2 >> "$configmap_filename"
+}
+
+# Adds mappings from single branch directly to spec
+# $1 - branch name
+add_mappings_from_branch() {
+ check_arguments $# $EXPECTED_1_ARG
+ schemas=$(ls -g $tmp_location/$1/$schemas_location/*.yaml | awk '{print $NF}' )
+
+ for schema in $schemas; do
+ repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev)
+ schema_repo_path=$(echo "$schema" | cut -d/ -f4-)
+ public_url_schemas_location=${repo_url%.*}
+ public_url=$public_url_schemas_location/$TREE/$schema_repo_path
+ local_url=$vendor/$repo_endpoint/$TREE/$schema_repo_path
+
+ echo "{" | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename"
+ echo "\"publicURL\": \"$public_url\"," | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename"
+ echo "\"localURL\": \"$local_url\"" | indent_string $INDENTATION_LEVEL_4 >> "$configmap_filename"
+ echo "}," | indent_string $INDENTATION_LEVEL_3 >> "$configmap_filename"
+ done
+}
+
+create_snippet() {
+ echo "Generating snippets in file: $snippet_filename"
+ generate_entries
+
+ cat << EOF > "$snippet_filename"
+Snippets for mounting ConfigMap in DCAE VESCollector Deployment
+=========================================================================
+
+## Description
+These snippets will override existing in VESCollector schemas and mapping file.
+
+No extra configuration in VESCollector is needed with these snippets.
+
+## Snippets
+#### spec.template.spec.containers[0].volumeMounts
+\`\`\`
+ - mountPath: /opt/app/VESCollector/etc/externalRepo
+ name: custom-$vendor-schemas
+\`\`\`
+
+#### spec.template.spec.volumes
+\`\`\`
+ - configMap:
+ defaultMode: 420
+ items:
+ - key: $SCHEMA_MAP_FILENAME
+ path: schema-map.json
+$schemas_entries
+ name: $configmap_name
+ name: custom-$vendor-schemas
+\`\`\`
+EOF
+}
+
+generate_entries() {
+ for actual_branch in $valid_branches; do
+ schemas=$(ls -g $tmp_location/$actual_branch/$schemas_location/*.yaml | awk '{print $NF}')
+ for schema in $schemas; do
+ repo_endpoint=$(echo "$repo_url" | cut -d/ -f4- | rev | cut -d. -f2- | rev)
+ schema_repo_path=$(echo "$schema" | cut -d/ -f4-)
+
+ key="$actual_branch-$(basename "$schema")"
+ path=$vendor/$repo_endpoint/$TREE/$schema_repo_path
+ schemas_entries="$schemas_entries- key: $key\n path: $path\n"
+ done
+ done
+ schemas_entries=$(echo "$schemas_entries" | indent_string $INDENTATION_LEVEL_5)
+}
+
+# todo add check of global env whether script should be ran
+main() {
+ check_arguments $arguments_number $EXPECTED_7_ARGS
+ clone_repo
+ add_config_map_metadata
+ add_schemas
+ generate_mapping_file
+ create_snippet
+}
+
+main
\ No newline at end of file
--- /dev/null
+#!/bin/sh
+
+# ============LICENSE_START=======================================================
+# OOM
+# ================================================================================
+# Copyright (C) 2020 Nokia. All rights reserved.
+# ================================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============LICENSE_END=========================================================
+
+# Arguments renaming
+spec_configmap_filename=$1
+k8s_configmap_name=$2
+
+# Constants
+MAX_SPEC_SIZE=262144
+
+# Alias
+alias kubectl_onap="kubectl -n onap"
+
+# Checks whether ConfigMap $spec_configmap_filename exists
+# When file does not exist exits with return code 1
+checkIfSpecExists() {
+ if [ ! -f "$spec_configmap_filename" ]; then
+ echo "Spec file $spec_configmap_filename does not exist."
+ # todo add location of spec with filename
+ exit 1
+ fi
+}
+
+# Uploads ConfigMap spec $spec_configmap_filename to Kubernetes
+uploadConfigMap() {
+ spec_size=$(stat --printf="%s" "$spec_configmap_filename")
+ if [ "$spec_size" -ge $MAX_SPEC_SIZE ]; then
+ echo "ConfigMap spec file is too long for 'kubectl apply'. Actual spec length: $spec_size, max spec length: $MAX_SPEC_SIZE"
+ echo "Creating new ConfigMap $k8s_configmap_name"
+ kubectl_onap create -f "$spec_configmap_filename"
+ else
+ echo "Applying ConfigMap $k8s_configmap_name"
+ kubectl_onap apply -f "$spec_configmap_filename"
+ fi
+}
+
+main() {
+ checkIfSpecExists
+ uploadConfigMap
+}
+
+main
+
Code Review / oom / utils.git / commitdiff