From: FrancescoFioraEst Date: Thu, 5 Feb 2026 15:27:25 +0000 (+0000) Subject: Update doc for Smoke tests X-Git-Url: https://gerrit.onap.org/r/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F19%2F143119%2F1;p=policy%2Fparent.git Update doc for Smoke tests Issue-ID: POLICY-5540 Change-Id: Ie71a5482049ee641ed609d237c16bd6b4f347873 Signed-off-by: FrancescoFioraEst --- diff --git a/docs/development/devtools/smoke/acm-participants-smoke.rst b/docs/development/devtools/smoke/acm-participants-smoke.rst index 869205a4..988fa0b3 100644 --- a/docs/development/devtools/smoke/acm-participants-smoke.rst +++ b/docs/development/devtools/smoke/acm-participants-smoke.rst @@ -28,7 +28,7 @@ The procedure documented in this article has been verified using Ubuntu 20.04 LT 2.1 Prerequisites ================= -- Java 17 +- Java 21 - Docker - Maven 3.9 - Git @@ -39,7 +39,7 @@ The procedure documented in this article has been verified using Ubuntu 20.04 LT 2.2 Cloning CLAMP automation composition ======================================== -Run a script such as the script below to clone the required modules from the `ONAP git repository `_. This script clones CLAMP automation composition and all dependency. +Run the below command to clone the required CLAMP automation composition: .. code-block:: bash @@ -55,88 +55,98 @@ Execution of the command above results in the following directory hierarchy in y 2.3 Setting up the components ============================= -2.3.1 Running MariaDb and Kafka -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We will be using Docker to run our mariadb instance and Kafka. It will have the acm-runtime database running in it. -The easiest way to do this is to perform a SQL script. Create the *mariadb.sql* file in the directory *~/git*. +2.3.1 Setting for kubernetes. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +If the helm location is not '/usr/local/bin/helm' or the kubectl location is not '/usr/local/bin/kubectl', you have to update +the file 'participant/participant-impl/participant-impl-kubernetes/src/main/java/org/onap/policy/clamp/acm/participant/kubernetes/helm/HelmClient.java'. -.. literalinclude:: files/mariadb.sql - :language: SQL +2.3.2 Building CLAMP automation composition +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Create the '*docker-compose.yaml*' using following code: +**Step 1:** Optionally, for a completely clean build, remove the ONAP built modules from your local repository. -.. literalinclude:: files/docker-compose-local.yaml - :language: yaml + .. code-block:: bash -Run the docker composition: + rm -fr ~/.m2/repository/org/onap - .. code-block:: bash - cd ~/git/ - docker compose up +**Step 2:** You can now build the Policy framework. -2.3.2 Setting topicParameterGroup for kafka localhost -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -It needs to set 'kafka' as topicCommInfrastructure and 'localhost:29092' as server. -In the clamp repo, you should find the file 'runtime-acm/src/main/resources/application.yaml'. This file (in the 'runtime' parameters section) may need to be altered as below: +Build java artifacts and docker images: + + .. code-block:: bash + + cd ~/git/clamp + mvn clean install -P docker -DskipTests -.. literalinclude:: files/runtime-application.yaml - :language: yaml -Same changes (in the 'participant' parameters section) -may need to be apply into the file 'participant/participant-impl/participant-impl-http/src/main/resources/config/application.yaml'. +2.3.3 Running Postgres and Kafka +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +We will be using Docker to run our Postgres instance and Kafka. It will have the acm-runtime database running in it. +Create the *db-pg.conf* and *db-pg.sh* files in the directory *~/git*. + +.. literalinclude:: files/db-pg.conf + :language: yaml -.. literalinclude:: files/participant-http-application.yaml +.. literalinclude:: files/db-pg.sh :language: yaml -And into the file 'participant/participant-impl/participant-impl-kubernetes/src/main/resources/config/application.yaml'. -.. literalinclude:: files/participant-kubernetes-application.yaml +Create the '*docker-compose.yaml*' using following code: + +.. literalinclude:: files/docker-compose-local.yaml :language: yaml -If the helm location is not '/usr/local/bin/helm' or the kubectl location is not '/usr/local/bin/kubectl', you have to update -the file 'participant/participant-impl/participant-impl-kubernetes/src/main/java/org/onap/policy/clamp/acm/participant/kubernetes/helm/HelmClient.java'. +Run the docker composition: -2.3.3 Automation composition Runtime -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To start the automation composition runtime service, we need to execute the following maven command from the "runtime-acm" directory in the clamp repo. Automation composition runtime uses the config file "src/main/resources/application.yaml" by default. + .. code-block:: bash -.. code-block:: bash + cd ~/git/ + docker compose up + +2.3.4 Running ACM-R on the Command Line +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +To start the automation composition runtime service, we need to execute the following command line from the "runtime-acm" directory in the clamp repo. Automation composition runtime uses the config file "src/main/resources/application.yaml" by default. + + .. code-block:: bash - mvn spring-boot:run + cd ~/git/clamp/runtime-acm + java -DRUNTIME_USER=runtimeUser -DRUNTIME_PASSWORD=zb\!XztG34 \ + -DSQL_HOST=localhost -DSQL_PORT=5432 -DSQL_USER=policy_user -DSQL_PASSWORD=policy_user \ + -DKAFKA_SERVER=localhost:29092 -DTOPIC_COMM_INFRASTRUCTURE=kafka \ + -jar target/policy-clamp-runtime-acm-9.0.1-SNAPSHOT.jar -2.3.4 Helm chart repository +2.3.5 Helm chart repository ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Kubernetes participant consumes helm charts from the local chart database as well as from a helm repository. For the smoke testing, we are going to add `nginx-stable` helm repository to the helm client. The following command can be used to add nginx repository to the helm client. -.. code-block:: bash + .. code-block:: bash - helm repo add nginx-stable https://helm.nginx.com/stable + helm repo add nginx-stable https://helm.nginx.com/stable -2.3.5 Kubernetes and http participants +2.3.6 Kubernetes and http participants ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The participants can be started from the clamp repository by executing the following maven command from the appropriate directories. +The participants can be started from the clamp repository by executing the following command line from the appropriate directories. The participants will be started and get registered to the Automation composition runtime. Navigate to the directory "participant/participant-impl/participant-impl-kubernetes/" and start kubernetes participant. -.. code-block:: bash + .. code-block:: bash - mvn spring-boot:run + cd ~/git/clamp/participant/participant-impl/participant-impl-kubernetes + java -Dserver.port=8082 -DkafkaServer=localhost:29092 -DtopicCommInfrastructure=kafka \ + -jar target/policy-clamp-participant-impl-kubernetes-9.0.1-SNAPSHOT.jar -Navigate to the directory "participant/participant-impl/participant-impl-http/" and start http participant. -.. code-block:: bash - - mvn spring-boot:run +Navigate to the directory "participant/participant-impl/participant-impl-http/" and start http participant. -For building docker images of runtime-acm and participants: + .. code-block:: bash -.. code-block:: bash + cd ~/git/clamp/participant/participant-impl/participant-impl-http + java -Dserver.port=8083 -DkafkaServer=localhost:29092 -DtopicCommInfrastructure=kafka \ + -jar target/policy-clamp-participant-impl-http-9.0.1-SNAPSHOT.jar - cd ~/git/onap/policy/clamp/ - mvn clean install -P docker 3. Running Tests diff --git a/docs/development/devtools/smoke/clamp-ac-participant-protocol-smoke.rst b/docs/development/devtools/smoke/clamp-ac-participant-protocol-smoke.rst index 8bcdc3b8..de400b97 100644 --- a/docs/development/devtools/smoke/clamp-ac-participant-protocol-smoke.rst +++ b/docs/development/devtools/smoke/clamp-ac-participant-protocol-smoke.rst @@ -23,7 +23,7 @@ Linux user - although the majority of the steps show will be exactly the same in 2.1 Prerequisites ================= -- Java 17 +- Java 21 - Docker - Maven 3.9 - Git @@ -35,17 +35,17 @@ Linux user - although the majority of the steps show will be exactly the same in - Automation Composition runtime component docker image is started and running. - Participant docker images policy-clamp-ac-pf-ppnt, policy-clamp-ac-http-ppnt, policy-clamp-ac-k8s-ppnt are started and running. - Kafka/Zookeeper for communication between components. -- mariadb docker container for policy and clampacm database. +- Postgres docker container for policy and clampacm database. - policy-api for communication between policy participant and policy-framework In this setup guide, we will be setting up all the components technically required for a working convenient dev environment. We will not be setting up all the participants - we will setup only the policy participant as an example. -2.2.1 MariaDB Setup -=================== +2.2.1 Postgres Setup +==================== -We will be using Docker to run our mariadb instance. It will have a total of two databases running in it. +We will be using Docker to run our Postgres instance. It will have a total of two databases running in it. - clampacm: the policy-clamp-runtime-acm db - policyadmin: the policy-api db diff --git a/docs/development/devtools/smoke/clamp-smoke.rst b/docs/development/devtools/smoke/clamp-smoke.rst index d1ca6fa8..932074fc 100644 --- a/docs/development/devtools/smoke/clamp-smoke.rst +++ b/docs/development/devtools/smoke/clamp-smoke.rst @@ -27,7 +27,7 @@ The procedure documented in this article has been verified using Ubuntu 20.04 LT Cloning CLAMP automation composition ************************************ -Run a script such as the script below to clone the required modules from the `ONAP git repository `_. This script clones CLAMP automation composition and all dependency. +Run the below command to clone the required CLAMP automation composition: .. code-block:: bash @@ -43,50 +43,35 @@ Execution of the command above results in the following directory hierarchy in y Building CLAMP automation composition ************************************* -**Step 1:** Setting topicParameterGroup for kafka localhost. -It needs to set 'kafka' as topicCommInfrastructure and 'localhost:29092' as server. -In the clamp repo, you should find the file 'runtime-acm/src/main/resources/application.yaml'. This file (in the 'runtime' parameters section) may need to be altered as below: - -.. literalinclude:: files/runtime-application.yaml - :language: yaml - -Same changes (in the 'participant' parameters section) may need to be apply into the file 'participant/participant-impl/participant-impl-simulator/src/main/resources/config/application.yaml'. - -.. literalinclude:: files/participant-sim-application.yaml - :language: yaml - -**Step 2:** Optionally, for a completely clean build, remove the ONAP built modules from your local repository. +**Step 1:** Optionally, for a completely clean build, remove the ONAP built modules from your local repository. .. code-block:: bash rm -fr ~/.m2/repository/org/onap -**Step 3:** You can now build the Policy framework. +**Step 2:** You can now build the Policy framework. -Build java artifacts only: +Build java artifacts and docker images: .. code-block:: bash cd ~/git/clamp - mvn clean install -DskipTests - -Build with docker images: + mvn clean install -P docker -DskipTests - .. code-block:: bash - cd ~/git/clamp/packages/ - mvn clean install -P docker +Running Postgres and Kafka +************************** -Running MariaDb and Kafka -************************* +Assuming you have successfully built the codebase using the instructions above. There are two requirements for the Clamp automation composition component to run, Postgres database and Kafka/Zookeeper. The easiest way to do this is to run a docker compose locally. -Assuming you have successfully built the codebase using the instructions above. There are two requirements for the Clamp automation composition component to run, MariaDb database and Kafka/Zookeeper. The easiest way to do this is to run a docker compose locally. +Create the *db-pg.conf* and *db-pg.sh* files in the directory *~/git*. -A sql such as the one below can be used to build the SQL initialization. Create the *mariadb.sql* file in the directory *~/git*. +.. literalinclude:: files/db-pg.conf + :language: yaml -.. literalinclude:: files/mariadb.sql - :language: SQL +.. literalinclude:: files/db-pg.sh + :language: yaml Create the '*docker-compose.yaml*' using following code: @@ -104,24 +89,16 @@ Run the docker composition: Developing and Debugging CLAMP automation composition ***************************************************** -Running on the Command Line using Maven -+++++++++++++++++++++++++++++++++++++++ - -Once the mariadb and DMaap simulator are up and running, run the following commands: +Running ACM-R on the Command Line ++++++++++++++++++++++++++++++++++ .. code-block:: bash cd ~/git/clamp/runtime-acm - mvn spring-boot:run - - -Running on the Command Line -+++++++++++++++++++++++++++ - - .. code-block:: bash - - cd ~/git/clamp/runtime-acm - java -jar target/policy-clamp-runtime-acm-8.1.0-SNAPSHOT.jar + java -DRUNTIME_USER=runtimeUser -DRUNTIME_PASSWORD=zb\!XztG34 \ + -DSQL_HOST=localhost -DSQL_PORT=5432 -DSQL_USER=policy_user -DSQL_PASSWORD=policy_user \ + -DKAFKA_SERVER=localhost:29092 -DTOPIC_COMM_INFRASTRUCTURE=kafka \ + -jar target/policy-clamp-runtime-acm-9.0.1-SNAPSHOT.jar Running participant simulator @@ -132,7 +109,11 @@ Run the following commands: .. code-block:: bash cd ~/git/clamp/participant/participant-impl/participant-impl-simulator - java -jar target/policy-clamp-participant-impl-simulator-8.1.0-SNAPSHOT.jar + java -Dserver.port=8085 -DHTTP_USER=participantUser -DHTTP_PASSWORD=zb\!XztG34 \ + -DkafkaServer=localhost:29092 -DtopicCommInfrastructure=kafka \ + -DparticipantId=101c62b3-8918-41b9-a747-d21eb79c6c90 \ + -DapplicationName=sim-ppnt -DgroupId=policy-clamp-ac-sim-ppnt \ + -jar target/policy-clamp-participant-impl-simulator-9.0.1-SNAPSHOT.jar Running the CLAMP automation composition docker image @@ -140,76 +121,8 @@ Running the CLAMP automation composition docker image Create the '*docker-compose.yaml*' using following code: - .. code-block:: yaml - - services: - mariadb: - image: mariadb:10.10.2 - command: ['mysqld', '--lower_case_table_names=1'] - volumes: - - type: bind - source: ./mariadb.sql - target: /docker-entrypoint-initdb.d/data.sql - environment: - - MYSQL_ROOT_PASSWORD=my-secret-pw - ports: - - "3306:3306" - - zookeeper: - image: confluentinc/cp-zookeeper:latest - environment: - ZOOKEEPER_CLIENT_PORT: 2181 - ZOOKEEPER_TICK_TIME: 2000 - ports: - - 2181:2181 - kafka: - image: confluentinc/cp-kafka:latest - container_name: kafka - depends_on: - - zookeeper - ports: - - 29092:29092 - - 9092:9092 - environment: - KAFKA_BROKER_ID: 1 - KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT - KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 - - runtime-acm: - image: onap/policy-clamp-runtime-acm - depends_on: - - zookeeper - - kafka - - mariadb - environment: - MARIADB_HOST: mariadb - TOPICSERVER: kafka:9092 - SERVER_SSL_ENABLED: false - volumes: - - type: bind - source: ./clamp/runtime-acm/src/main/resources/application.yaml - target: /opt/app/policy/clamp/etc/AcRuntimeParameters.yaml - ports: - - "6969:6969" - - participant-simulator: - image: onap/policy-clamp-ac-sim-ppnt - depends_on: - - zookeeper - - kafka - environment: - MARIADB_HOST: mariadb - TOPICSERVER: kafka:9092 - SERVER_SSL_ENABLED: false - volumes: - - type: bind - source: ./clamp/participant/participant-impl/participant-impl-simulator/src/main/resources/config/application.yaml - target: /opt/app/policy/clamp/etc/SimulatorParticipantParameters.yaml - ports: - - "8085:8085" +.. literalinclude:: files/docker-compose-clamp.yaml + :language: yaml Run the docker composition: diff --git a/docs/development/devtools/smoke/files/db-pg.conf b/docs/development/devtools/smoke/files/db-pg.conf new file mode 100644 index 00000000..49103738 --- /dev/null +++ b/docs/development/devtools/smoke/files/db-pg.conf @@ -0,0 +1,19 @@ +# Copyright (C) 2022, 2024,2026 OpenInfra Foundation Europe. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +PGSQL_ROOT_PASSWORD=secret +PGSQL_USER=policy_user +PGSQL_PASSWORD=policy_user +PGPASSWORD=policy_user +POSTGRES_PASSWORD=policy_user diff --git a/docs/development/devtools/smoke/files/db-pg.sh b/docs/development/devtools/smoke/files/db-pg.sh new file mode 100644 index 00000000..31e213f7 --- /dev/null +++ b/docs/development/devtools/smoke/files/db-pg.sh @@ -0,0 +1,22 @@ +# Copyright (C) 2022, 2024,2026 OpenInfra Foundation Europe. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +psql -U postgres -d postgres --command "CREATE USER ${PGSQL_USER} WITH PASSWORD '${PGSQL_PASSWORD}';" + +for db in migration policyadmin operationshistory clampacm +do + psql -U postgres -d postgres --command "CREATE DATABASE ${db};" + psql -U postgres -d postgres --command "ALTER DATABASE ${db} OWNER TO ${PGSQL_USER} ;" + psql -U postgres -d postgres --command "GRANT ALL PRIVILEGES ON DATABASE ${db} TO ${PGSQL_USER} ;" +done diff --git a/docs/development/devtools/smoke/files/docker-compose-clamp.yaml b/docs/development/devtools/smoke/files/docker-compose-clamp.yaml new file mode 100644 index 00000000..d830c22e --- /dev/null +++ b/docs/development/devtools/smoke/files/docker-compose-clamp.yaml @@ -0,0 +1,71 @@ +services: + postgres: + image: nexus3.onap.org:10001/library/postgres:latest + container_name: postgres + hostname: postgres + command: [ '--idle_in_transaction_session_timeout=28800' ] + env_file: ./db-pg.conf + volumes: + - ./db-pg.sh:/docker-entrypoint-initdb.d/db-pg.sh:ro + expose: + - 5432 + ports: + - "5432:5432" + + zookeeper: + image: nexus3.onap.org:10001/confluentinc/cp-zookeeper:latest + environment: + ZOOKEEPER_CLIENT_PORT: 2181 + ZOOKEEPER_TICK_TIME: 2000 + ports: + - 2181:2181 + + kafka: + image: nexus3.onap.org:10001/confluentinc/cp-kafka:7.4.9 + container_name: kafka + depends_on: + - zookeeper + ports: + - 29092:29092 + - 9092:9092 + environment: + KAFKA_BROKER_ID: 1 + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + + runtime-acm: + image: onap/policy-clamp-runtime-acm + depends_on: + - zookeeper + - kafka + - postgres + environment: + RUNTIME_USER: runtimeUser + RUNTIME_PASSWORD: zb!XztG34 + SQL_HOST: postgres + SQL_PORT: 5432 + SQL_USER: policy_user + SQL_PASSWORD: policy_user + TOPIC_COMM_INFRASTRUCTURE: kafka + ports: + - "6969:6969" + + participant-simulator: + image: onap/policy-clamp-ac-sim-ppnt + depends_on: + - zookeeper + - kafka + environment: + HTTP_USER: participantUser + HTTP_PASSWORD: zb!XztG34 + SERVER_PORT: 8085 + topicCommInfrastructure: kafka + participantId: 101c62b3-8918-41b9-a747-d21eb79c6c90 + applicationName: sim-ppnt + groupId: policy-clamp-ac-sim-ppnt + supportedElementTypeName: org.onap.policy.clamp.acm.SimAutomationCompositionElement + ports: + - "8085:8085" diff --git a/docs/development/devtools/smoke/files/docker-compose-local.yaml b/docs/development/devtools/smoke/files/docker-compose-local.yaml index 5ee39f67..fcfe260f 100644 --- a/docs/development/devtools/smoke/files/docker-compose-local.yaml +++ b/docs/development/devtools/smoke/files/docker-compose-local.yaml @@ -1,18 +1,19 @@ services: - mariadb: - image: mariadb:10.10.2 - command: ['mysqld', '--lower_case_table_names=1'] + postgres: + image: nexus3.onap.org:10001/library/postgres:latest + container_name: postgres + hostname: postgres + command: [ '--idle_in_transaction_session_timeout=28800' ] + env_file: ./db-pg.conf volumes: - - type: bind - source: ./mariadb.sql - target: /docker-entrypoint-initdb.d/data.sql - environment: - - MYSQL_ROOT_PASSWORD=my-secret-pw + - ./db-pg.sh:/docker-entrypoint-initdb.d/db-pg.sh:ro + expose: + - 5432 ports: - - "3306:3306" + - "5432:5432" zookeeper: - image: confluentinc/cp-zookeeper:latest + image: nexus3.onap.org:10001/confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 @@ -20,7 +21,7 @@ services: - 2181:2181 kafka: - image: confluentinc/cp-kafka:latest + image: nexus3.onap.org:10001/confluentinc/cp-kafka:7.4.9 container_name: kafka depends_on: - zookeeper diff --git a/docs/development/devtools/smoke/files/docker-compose-policy.yaml b/docs/development/devtools/smoke/files/docker-compose-policy.yaml index cdb50ebb..6224ef69 100644 --- a/docs/development/devtools/smoke/files/docker-compose-policy.yaml +++ b/docs/development/devtools/smoke/files/docker-compose-policy.yaml @@ -1,43 +1,41 @@ services: - mariadb: - image: mariadb:10.10.2 - command: ['mysqld', '--lower_case_table_names=1'] + postgres: + image: nexus3.onap.org:10001/library/postgres:latest + container_name: postgres + hostname: postgres + command: [ '--idle_in_transaction_session_timeout=28800' ] + env_file: ./db-pg.conf volumes: - - type: bind - source: ./mariadb.sql - target: /docker-entrypoint-initdb.d/data.sql - environment: - - MYSQL_ROOT_PASSWORD=my-secret-pw + - ./db-pg.sh:/docker-entrypoint-initdb.d/db-pg.sh:ro + expose: + - 5432 ports: - - "3306:3306" + - "5432:5432" policy-db-migrator: - image: nexus3.onap.org:10001/onap/policy-db-migrator:3.1.3-SNAPSHOT + image: nexus3.onap.org:10001/onap/policy-db-migrator:5.0.1-SNAPSHOT container_name: policy-db-migrator hostname: policy-db-migrator depends_on: - mariadb expose: - 6824 + env_file: ./db-pg.conf environment: - SQL_DB: policyadmin - SQL_HOST: mariadb - MYSQL_ROOT_PASSWORD: my-secret-pw - MYSQL_USER: policy_user - MYSQL_PASSWORD: policy_user - MYSQL_CMD: mysql + POSTGRES_PASSWORD: policy_user + SQL_HOST: postgres volumes: - - ./init.sh:/opt/app/policy/bin/db_migrator_policy_init.sh:ro + - ./init_pg.sh:/opt/app/policy/bin/db_migrator_policy_init_pg.sh:ro - ./wait_for_port.sh:/opt/app/policy/bin/wait_for_port.sh:ro entrypoint: /opt/app/policy/bin/wait_for_port.sh command: [ '-c', '/opt/app/policy/bin/db_migrator_policy_init.sh', - 'mariadb', '3306' + 'postgres', '5432' ] zookeeper: - image: confluentinc/cp-zookeeper:latest + image: nexus3.onap.org:10001/confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 @@ -45,7 +43,7 @@ services: - 2181:2181 kafka: - image: confluentinc/cp-kafka:latest + image: nexus3.onap.org:10001/confluentinc/cp-kafka:7.4.9 container_name: kafka depends_on: - zookeeper diff --git a/docs/development/devtools/smoke/files/init_pg.sh b/docs/development/devtools/smoke/files/init_pg.sh new file mode 100644 index 00000000..ebdac150 --- /dev/null +++ b/docs/development/devtools/smoke/files/init_pg.sh @@ -0,0 +1,44 @@ +#!/bin/sh +# ============LICENSE_START==================================================== +# Copyright (C) 2022, 2024,2026 OpenInfra Foundation Europe. All rights reserved. +# ============================================================================= +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# ============LICENSE_END====================================================== + +export POLICY_HOME=/opt/app/policy +export SQL_USER=${PGSQL_USER} +export SQL_PASSWORD=${PGSQL_PASSWORD} +export SCRIPT_DIRECTORY=postgres + +for schema in policyadmin clampacm pooling operationshistory +do + echo "Initializing $schema..." + /opt/app/policy/bin/prepare_upgrade.sh ${schema} + + /opt/app/policy/bin/db-migrator-pg -s ${schema} -o report + + /opt/app/policy/bin/db-migrator-pg -s ${schema} -o upgrade + rc=$? + + /opt/app/policy/bin/db-migrator-pg -s ${schema} -o report + + if [ "$rc" != 0 ]; then + break + fi +done + +nc -l -p 6824 + +exit $rc diff --git a/docs/development/devtools/smoke/files/mariadb.sql b/docs/development/devtools/smoke/files/mariadb.sql deleted file mode 100644 index 95c4c766..00000000 --- a/docs/development/devtools/smoke/files/mariadb.sql +++ /dev/null @@ -1,3 +0,0 @@ -create database clampacm; -CREATE USER 'policy'@'%' IDENTIFIED BY 'P01icY'; -GRANT ALL PRIVILEGES ON clampacm.* TO 'policy'@'%'; diff --git a/docs/development/devtools/smoke/files/participant-http-application.yaml b/docs/development/devtools/smoke/files/participant-http-application.yaml deleted file mode 100644 index edf324b4..00000000 --- a/docs/development/devtools/smoke/files/participant-http-application.yaml +++ /dev/null @@ -1,31 +0,0 @@ - -participant: - intermediaryParameters: - topics: - operationTopic: policy-acruntime-participant - syncTopic: acm-ppnt-sync - reportingTimeIntervalMs: 120000 - description: Participant Description - participantId: 101c62b3-8918-41b9-a747-d21eb79c6c01 - clampAutomationCompositionTopics: - topicSources: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - - topic: ${participant.intermediaryParameters.topics.syncTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - topicSinks: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - participantSupportedElementTypes: - - - typeName: org.onap.policy.clamp.acm.HttpAutomationCompositionElement - typeVersion: 1.0.0 - diff --git a/docs/development/devtools/smoke/files/participant-kubernetes-application.yaml b/docs/development/devtools/smoke/files/participant-kubernetes-application.yaml deleted file mode 100644 index 59732bbf..00000000 --- a/docs/development/devtools/smoke/files/participant-kubernetes-application.yaml +++ /dev/null @@ -1,32 +0,0 @@ - -participant: - localChartDirectory: /home/policy/local-charts - infoFileName: CHART_INFO.json - intermediaryParameters: - topics: - operationTopic: policy-acruntime-participant - syncTopic: acm-ppnt-sync - reportingTimeIntervalMs: 120000 - description: Participant Description - participantId: 101c62b3-8918-41b9-a747-d21eb79c6c02 - clampAutomationCompositionTopics: - topicSources: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - - topic: ${participant.intermediaryParameters.topics.syncTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - topicSinks: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - participantSupportedElementTypes: - - - typeName: org.onap.policy.clamp.acm.K8SMicroserviceAutomationCompositionElement - typeVersion: 1.0.0 diff --git a/docs/development/devtools/smoke/files/participant-sim-application.yaml b/docs/development/devtools/smoke/files/participant-sim-application.yaml deleted file mode 100644 index 2a7efc3f..00000000 --- a/docs/development/devtools/smoke/files/participant-sim-application.yaml +++ /dev/null @@ -1,30 +0,0 @@ - -participant: - intermediaryParameters: - topics: - operationTopic: policy-acruntime-participant - syncTopic: acm-ppnt-sync - reportingTimeIntervalMs: 120000 - description: Participant Description - participantId: 101c62b3-8918-41b9-a747-d21eb79c6c90 - clampAutomationCompositionTopics: - topicSources: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - - topic: ${participant.intermediaryParameters.topics.syncTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - fetchTimeout: 15000 - topicSinks: - - topic: ${participant.intermediaryParameters.topics.operationTopic} - servers: - - localhost:29092 - topicCommInfrastructure: kafka - participantSupportedElementTypes: - - - typeName: org.onap.policy.clamp.acm.SimAutomationCompositionElement - typeVersion: 1.0.0 diff --git a/docs/development/devtools/smoke/files/wait_for_port.sh b/docs/development/devtools/smoke/files/wait_for_port.sh new file mode 100644 index 00000000..7888ba9f --- /dev/null +++ b/docs/development/devtools/smoke/files/wait_for_port.sh @@ -0,0 +1,91 @@ +#!/bin/sh +# ============LICENSE_START==================================================== +# Copyright (C) 2021 AT&T Intellectual Property. All rights reserved. +# Modifications Copyright (C) 2022-2023, 2026 OpenInfra Foundation Europe. All rights reserved. +# ============================================================================= +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# ============LICENSE_END====================================================== + +usage() { + echo args: [-t timeout] [-c command] hostname1 port1 hostname2 port2 ... >&2 + exit 1 +} + +tmout=300 +cmd= +while getopts c:t: opt +do + case "$opt" in + c) + cmd="$OPTARG" + ;; + + t) + tmout="$OPTARG" + ;; + + *) + usage + ;; + esac +done + +nargs=$((OPTIND-1)) +shift "$nargs" + +even_args=$(($#%2)) +if [ $# -lt 2 ] || [ "$even_args" -ne 0 ] +then + usage +fi + +while [ $# -ge 2 ] +do + export host="$1" + export port="$2" + shift + shift + + echo "Waiting for $host port $port..." + + while [ "$tmout" -gt 0 ] + do + if command -v docker > /dev/null 2>&1 + then + docker ps --format "table {{ .Names }}\t{{ .Status }}" + fi + + nc -vz "$host" "$port" + rc=$? + + if [ $rc -eq 0 ] + then + break + else + tmout=$((tmout-1)) + sleep 1 + fi + done + + if [ $rc -ne 0 ] + then + echo "$host port $port cannot be reached" + exit $rc + fi +done + +$cmd + +exit 0 diff --git a/docs/development/devtools/smoke/policy-participant-smoke.rst b/docs/development/devtools/smoke/policy-participant-smoke.rst index ba677e11..405f1dbd 100644 --- a/docs/development/devtools/smoke/policy-participant-smoke.rst +++ b/docs/development/devtools/smoke/policy-participant-smoke.rst @@ -11,23 +11,23 @@ CLAMP Policy Participant Smoke Tests The Smoke testing of the policy participant is executed in a local CLAMP/Policy environment. The CLAMP-ACM interfaces interact with the Policy Framework to perform actions based on the state of the policy participant. The goal of the Smoke tests is the ensure that CLAMP Policy Participant and Policy Framework work together as expected. All applications will be running by console, so they need to run with different ports. Configuration files should be changed accordingly. -+------------------------------+-------+ -| Application | port | -+==============================+=======+ -| MariDB | 3306 | -+------------------------------+-------+ -| Zookeeper | 2181 | -+------------------------------+-------+ -| Kafka | 29092 | -+------------------------------+-------+ -| policy-api | 6968 | -+------------------------------+-------+ -| policy-pap | 6970 | -+------------------------------+-------+ -| policy-clamp-runtime-acm | 6969 | -+------------------------------+-------+ -| onap/policy-clamp-ac-pf-ppnt | 8085 | -+------------------------------+-------+ ++------------------------------+------------+ +| Application | Port | ++==============================+============+ +| Postgres | 5432 | ++------------------------------+------------+ +| Zookeeper | 2181 | ++------------------------------+------------+ +| Kafka | 29092/9092 | ++------------------------------+------------+ +| policy-api | 6968 | ++------------------------------+------------+ +| policy-pap | 6970 | ++------------------------------+------------+ +| policy-clamp-runtime-acm | 6969 | ++------------------------------+------------+ +| onap/policy-clamp-ac-pf-ppnt | 8085 | ++------------------------------+------------+ 2. Setup Guide @@ -38,7 +38,7 @@ This section will show the developer how to set up their environment to start te 2.1 Prerequisites ================= -- Java 17 +- Java 21 - Maven 3.9 - Git - Refer to this guide for basic environment setup `Setting up dev environment `_ @@ -46,319 +46,95 @@ This section will show the developer how to set up their environment to start te 2.2 Cloning CLAMP automation composition and all dependency =========================================================== -Run a script such as the script below to clone the required modules from the `ONAP git repository `_. This script clones CLAMP automation composition and all dependency. +Run the below commands to clone the required CLAMP automation composition and all modules: .. code-block:: bash - :caption: Typical ONAP Policy Framework Clone Script - :linenos: - #!/usr/bin/env bash + cd ~/git + git clone https://gerrit.onap.org/r/policy/clamp clamp + git clone https://gerrit.onap.org/r/policy/api api + git clone https://gerrit.onap.org/r/policy/pap pap - ## script name for output - MOD_SCRIPT_NAME='basename $0' - ## the ONAP clone directory, defaults to "onap" - clone_dir="onap" +Execution of the commands above results in the following directory hierarchy in your *~/git* directory: - ## the ONAP repos to clone - onap_repos="\ - policy/api \ - policy/clamp \ - policy/pap " - - ## - ## Help screen and exit condition (i.e. too few arguments) - ## - Help() - { - echo "" - echo "$MOD_SCRIPT_NAME - clones all required ONAP git repositories" - echo "" - echo " Usage: $MOD_SCRIPT_NAME [-options]" - echo "" - echo " Options" - echo " -d - the ONAP clone directory, defaults to '.'" - echo " -h - this help screen" - echo "" - exit 255; - } - - ## - ## read command line - ## - while [ $# -gt 0 ] - do - case $1 in - #-d ONAP clone directory - -d) - shift - if [ -z "$1" ]; then - echo "$MOD_SCRIPT_NAME: no clone directory" - exit 1 - fi - clone_dir=$1 - shift - ;; - - #-h prints help and exists - -h) - Help;exit 0;; - - *) echo "$MOD_SCRIPT_NAME: undefined CLI option - $1"; exit 255;; - esac - done - - if [ -f "$clone_dir" ]; then - echo "$MOD_SCRIPT_NAME: requested clone directory '$clone_dir' exists as file" - exit 2 - fi - if [ -d "$clone_dir" ]; then - echo "$MOD_SCRIPT_NAME: requested clone directory '$clone_dir' exists as directory" - exit 2 - fi - - mkdir $clone_dir - if [ $? != 0 ] - then - echo cannot clone ONAP repositories, could not create directory '"'$clone_dir'"' - exit 3 - fi - - for repo in $onap_repos - do - repoDir=`dirname "$repo"` - repoName=`basename "$repo"` - - if [ ! -z $dirName ] - then - mkdir "$clone_dir/$repoDir" - if [ $? != 0 ] - then - echo cannot clone ONAP repositories, could not create directory '"'$clone_dir/repoDir'"' - exit 4 - fi - fi - - git clone https://gerrit.onap.org/r/${repo} $clone_dir/$repo - done - - echo ONAP has been cloned into '"'$clone_dir'"' - - -Execution of the script above results in the following directory hierarchy in your *~/git* directory: - - * ~/git/onap - * ~/git/onap/policy - * ~/git/onap/policy/api - * ~/git/onap/policy/clamp - * ~/git/onap/policy/pap + * ~/git + * ~/git/api + * ~/git/clamp + * ~/git/pap 2.3 Building CLAMP automation composition and all dependency ============================================================ -**Step 1:** Setting topicParameterGroup for kafka localhost in clamp and policy-participant. -It needs to set 'kafka' as topicCommInfrastructure and 'localhost:29092' as server. -In the clamp repo, you should find the file 'runtime-acm/src/main/resources/application.yaml'. This file (in the 'runtime' parameters section) may need to be altered as below: - -.. literalinclude:: files/runtime-application.yaml - :language: yaml - -Setting topicParameterGroup for kafka localhost and api/pap http client (in the 'participant' parameters section) may need to be apply into the file 'participant/participant-impl/participant-impl-policy/src/main/resources/config/application.yaml'. - -.. literalinclude:: files/participant-policy-application.yaml - :language: yaml - - -**Step 2:** Setting datasource.url, hibernate.ddl-auto and server.port in policy-api. +**Step 1:** Setting datasource.url, hibernate.ddl-auto and server.port in policy-api. In the api repo, you should find the file 'main/src/main/resources/application.yaml'. This file may need to be altered as below: .. literalinclude:: files/api-application.yaml :language: yaml -**Step 3:** Setting datasource.url, server.port, and api http client in policy-pap. +**Step 2:** Setting datasource.url, server.port, and api http client in policy-pap. In the pap repo, you should find the file 'main/src/main/resources/application.yaml'. This file may need to be altered as below: .. literalinclude:: files/pap-application.yaml :language: yaml -**Step 4:** Optionally, for a completely clean build, remove the ONAP built modules from your local repository. +**Step 3:** Optionally, for a completely clean build, remove the ONAP built modules from your local repository. .. code-block:: bash rm -fr ~/.m2/repository/org/onap -**Step 5:** A pom such as the one below can be used to build the ONAP Policy Framework modules. Create the *pom.xml* file in the directory *~/git/onap/policy*. +**Step 4:** You can now build the Policy framework. -.. code-block:: xml - :caption: Typical pom.xml to build the ONAP Policy Framework - :linenos: - - - 4.0.0 - org.onap - onap-policy - 1.0.0-SNAPSHOT - pom - ${project.artifactId} - 2024 - - ONAP - - - - api - clamp - pap - - +Build java artifacts and docker images: + .. code-block:: bash -**Step 6:** You can now build the Policy framework. + cd ~/git/clamp + mvn clean install -P docker -DskipTests + cd ~/git/api + mvn clean install -P docker -DskipTests + cd ~/git/pap + mvn clean install -P docker -DskipTests -Build java artifacts only: - .. code-block:: bash +2.4 Setting up the components +============================= - cd ~/git/onap/policy - mvn clean install -DskipTests +2.4.1 Postgres and Kafka Setup +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Build with docker images: +We will be using Docker to run our Postgres instance` and Zookeeper/Kafka. It will have a total of two databases running in Postgres. - .. code-block:: bash +- clampacm: the policy-clamp-runtime-acm db +- policyadmin: the policy-api db - cd ~/git/onap/policy/clamp/packages/ - mvn clean install -P docker +**Step 1:** Create the *db-pg.conf* and *db-pg.sh* files in the directory *~/git*. - cd ~/git/onap/policy/api/packages/ - mvn clean install -P docker +.. literalinclude:: files/db-pg.conf + :language: yaml - cd ~/git/onap/policy/pap/packages/ - mvn clean install -P docker +.. literalinclude:: files/db-pg.sh + :language: yaml -2.4 Setting up the components -============================= -2.4.1 MariaDB and Kafka Setup -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +**Step 2:** Create the *wait_for_port.sh* file in a directory *~/git* with execution permission. -We will be using Docker to run our mariadb instance`and Zookeeper/Kafka. It will have a total of two databases running in mariadb. +.. literalinclude:: files/wait_for_port.sh + :language: sh -- clampacm: the policy-clamp-runtime-acm db -- policyadmin: the policy-api db -**Step 1:** Create the *mariadb.sql* file in a directory *~/git*. - - .. code-block:: SQL - - create database clampacm; - CREATE USER 'policy'@'%' IDENTIFIED BY 'P01icY'; - GRANT ALL PRIVILEGES ON clampacm.* TO 'policy'@'%'; - CREATE DATABASE `policyadmin`; - CREATE USER 'policy_user'@'%' IDENTIFIED BY 'policy_user'; - GRANT ALL PRIVILEGES ON policyadmin.* to 'policy_user'@'%'; - CREATE DATABASE `migration`; - GRANT ALL PRIVILEGES ON migration.* to 'policy_user'@'%'; - FLUSH PRIVILEGES; - - -**Step 2:** Create the *init.sh* file in a directory *~/git* with execution permission. - - .. code-block:: sh - - #!/bin/sh - - export POLICY_HOME=/opt/app/policy - export SQL_USER=${MYSQL_USER} - export SQL_PASSWORD=${MYSQL_PASSWORD} - export SCRIPT_DIRECTORY=sql - - /opt/app/policy/bin/prepare_upgrade.sh ${SQL_DB} - /opt/app/policy/bin/db-migrator -s ${SQL_DB} -o report - /opt/app/policy/bin/db-migrator -s ${SQL_DB} -o upgrade - rc=$? - /opt/app/policy/bin/db-migrator -s ${SQL_DB} -o report - nc -l -p 6824 - exit $rc - - -**Step 3:** Create the *wait_for_port.sh* file in a directory *~/git* with execution permission. - - .. code-block:: sh - - #!/bin/sh - - usage() { - echo args: [-t timeout] [-c command] hostname1 port1 hostname2 port2 ... >&2 - exit 1 - } - tmout=300 - cmd= - while getopts c:t: opt - do - case "$opt" in - c) - cmd="$OPTARG" - ;; - t) - tmout="$OPTARG" - ;; - *) - usage - ;; - esac - done - nargs=$((OPTIND-1)) - shift "$nargs" - even_args=$(($#%2)) - if [ $# -lt 2 ] || [ "$even_args" -ne 0 ] - then - usage - fi - while [ $# -ge 2 ] - do - export host="$1" - export port="$2" - shift - shift - echo "Waiting for $host port $port..." - - while [ "$tmout" -gt 0 ] - do - if command -v docker > /dev/null 2>&1 - then - docker ps --format "table {{ .Names }}\t{{ .Status }}" - fi - nc -vz "$host" "$port" - rc=$? - if [ $rc -eq 0 ] - then - break - else - tmout=$((tmout-1)) - sleep 1 - fi - done - if [ $rc -ne 0 ] - then - echo "$host port $port cannot be reached" - exit $rc - fi - done - $cmd - exit 0 - - -**Step 4:** Create the '*docker-compose.yaml*' using following code: +**Step 3:** Create the '*docker-compose.yaml*' using following code: .. literalinclude:: files/docker-compose-policy.yaml :language: yaml -**Step 5:** Run the docker composition: +**Step 4:** Run the docker composition: .. code-block:: bash @@ -392,7 +168,11 @@ To start the clampacm runtime we need to go the "runtime-acm" directory in the c .. code-block:: bash - mvn spring-boot:run + cd ~/git/clamp/runtime-acm + java -DRUNTIME_USER=runtimeUser -DRUNTIME_PASSWORD=zb\!XztG34 \ + -DSQL_HOST=localhost -DSQL_PORT=5432 -DSQL_USER=policy_user -DSQL_PASSWORD=policy_user \ + -DKAFKA_SERVER=localhost:29092 -DTOPIC_COMM_INFRASTRUCTURE=kafka \ + -jar target/policy-clamp-runtime-acm-9.0.1-SNAPSHOT.jar 2.4.5 ACM Policy Participant ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -401,7 +181,9 @@ To start the policy participant we need to go to the "participant/participant-im .. code-block:: bash - mvn spring-boot:run + cd ~/git/clamp/participant/participant-impl/participant-impl-policy + java -Dserver.port=8084 -DkafkaServer=localhost:29092 -DtopicCommInfrastructure=kafka \ + -jar target/policy-clamp-participant-impl-policy-9.0.1-SNAPSHOT.jar 3. Testing Procedure ====================