Update docs for distribution s3ps

Issue-ID: POLICY-4844
Change-Id: Ic68da87bcddd6de4c54dc46dfe693e8188c1ce4b
Signed-off-by: saul.gill <saul.gill@est.tech>

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-jmeter-testcases.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-jmeter-testcases.png
index 86a437a..1159bca 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-jmeter-testcases.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-jmeter-testcases.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-visualvm-snapshot.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-visualvm-snapshot.png
index 03b73d3..b9b175a 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-visualvm-snapshot.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/distribution-visualvm-snapshot.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-monitor.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-monitor.png
index 71fd7fc..98760e0 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-monitor.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-monitor.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-statistics.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-statistics.png
index fecd6c0..440a177 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-statistics.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-statistics.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threads.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threads.png
index 2488abd..a04d98b 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threads.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threads.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threshold.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threshold.png
index 73b20ff..bf20cf3 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threshold.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/performance-threshold.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-monitor.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-monitor.png
index bebaaeb..23030a4 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-monitor.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-monitor.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-statistics.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-statistics.png
index 12ee2b5..f184a59 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-statistics.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-statistics.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threads.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threads.png
index 4cfd7a7..c7279d9 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threads.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threads.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threshold.png b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threshold.png
index f348761..37ad5c4 100644 (file)
Binary files a/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threshold.png and b/docs/development/devtools/testing/s3p/distribution-s3p-results/stability-threshold.png differ

diff --git a/docs/development/devtools/testing/s3p/distribution-s3p.rst b/docs/development/devtools/testing/s3p/distribution-s3p.rst
index 5596673..40ade31 100644 (file)
--- a/docs/development/devtools/testing/s3p/distribution-s3p.rst
+++ b/docs/development/devtools/testing/s3p/distribution-s3p.rst
@@ -13,71 +13,7 @@ Policy Distribution component
 Common Setup
 ------------
 
 Common Setup
 ------------
 

-Update the ubuntu software installer
-
-.. code-block:: bash
-
-    sudo apt update
-
-Install Java
-
-.. code-block:: bash
-
-    sudo apt install -y openjdk-11-jdk
-
-Ensure that the Java version that is executing is OpenJDK version 11
-
-.. code-block:: bash
-
-    $ java --version
-    openjdk 11.0.11 2021-04-20
-    OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
-    OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode)
-
-Install Docker and Docker Compose
-
-.. code-block:: bash
-
-    # Add docker repository
-    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
-
-    echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
-    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
-
-    sudo apt update
-
-    # Install docker
-    sudo apt-get install docker-ce docker-ce-cli containerd.io
-
-Change the permissions of the Docker socket file
-
-.. code-block:: bash
-
-    sudo chmod 666 /var/run/docker.sock
-
-Check the status of the Docker service and ensure it is running correctly
-
-.. code-block:: bash
-
-    systemctl status --no-pager docker
-    docker.service - Docker Application Container Engine
-       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
-       Active: active (running) since Wed 2020-10-14 13:59:40 UTC; 1 weeks 0 days ago
-       # ... (truncated for brevity)
-
-    docker ps
-    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
-
-Install and verify docker-compose
-
-.. code-block:: bash
-
-    # Install compose (check if version is still available or update as necessary)
-    sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
-    sudo chmod +x /usr/local/bin/docker-compose
-
-    # Check if install was successful
-    docker-compose --version
+The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-sc3-test.sh**.

 
 Clone the policy-distribution repo to access the test scripts
 
 
 Clone the policy-distribution repo to access the test scripts
 


@@ -85,130 +21,55 @@ Clone the policy-distribution repo to access the test scripts
 
     git clone https://gerrit.onap.org/r/policy/distribution
 
 
     git clone https://gerrit.onap.org/r/policy/distribution
 

-.. _setup-distribution-s3p-components:
-
-Start services for MariaDB, Policy API, PAP and Distribution
-------------------------------------------------------------
-
-Navigate to the main folder for scripts to setup services:
-
-.. code-block:: bash
-
-    cd ~/distribution/testsuites/stability/src/main/resources/setup
-
-Modify the versions.sh script to match all the versions being tested.
-
-.. code-block:: bash
-
-    vi ~/distribution/testsuites/stability/src/main/resources/setup/versions.sh
-
-Ensure the correct docker image versions are specified - e.g. for Kohn-M4
+**The following common steps are carried out by the scripts**

 
 

-- export POLICY_DIST_VERSION=2.8-SNAPSHOT
+* Updates the repo package lists for apt
+* Installs Java 17 open jdk
+* Installs docker
+* Installs docker-compose
+* Retrieves version information in environment variables from th release info file
+* Builds relevant images including the pdp simulator
+* Triggers docker compose to bring up containers required for the testing
+* Installs jmeter
+* Installs visualvm (and starts it in a GUI environment)
+* Configures permissions for monitoring
+* Starts jstatd
+* Waits for containers to come up
+* Runs either stability or performance tests for a specified duration depending on the arguments specified

 
 

-Run the start.sh script to start the components. After installation, script will execute
-``docker ps`` and show the running containers.
+For example, the below runs performance tests for 2 hours. Start from the root directory of policy distribution

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    ./start.sh
-
-    Creating network "setup_default" with the default driver
-    Creating policy-distribution ... done
-    Creating mariadb             ... done
-    Creating simulator           ... done
-    Creating policy-db-migrator  ... done
-    Creating policy-api          ... done
-    Creating policy-pap          ... done
-
-    fa4e9bd26e60   nexus3.onap.org:10001/onap/policy-pap:2.7-SNAPSHOT-latest                "/opt/app/policy/pap…"   1 second ago    Up Less than a second   6969/tcp             policy-pap
-    efb65dd95020   nexus3.onap.org:10001/onap/policy-api:2.7-SNAPSHOT-latest                "/opt/app/policy/api…"   1 second ago    Up Less than a second   6969/tcp             policy-api
-    cf602c2770ba   nexus3.onap.org:10001/onap/policy-db-migrator:2.5-SNAPSHOT-latest        "/opt/app/policy/bin…"   2 seconds ago   Up 1 second             6824/tcp             policy-db-migrator
-    99383d2fecf4   pdp/simulator                                                            "sh /opt/app/policy/…"   2 seconds ago   Up 1 second                                  pdp-simulator
-    3c0e205c5f47   nexus3.onap.org:10001/onap/policy-models-simulator:2.7-SNAPSHOT-latest   "simulators.sh"          3 seconds ago   Up 2 seconds            3904/tcp             simulator
-    3ad00d90d6a3   nexus3.onap.org:10001/onap/policy-distribution:2.8-SNAPSHOT-latest       "/opt/app/policy/bin…"   3 seconds ago   Up 2 seconds            6969/tcp, 9090/tcp   policy-distribution
-    bb0b915cdecc   nexus3.onap.org:10001/mariadb:10.5.8                                     "docker-entrypoint.s…"   3 seconds ago   Up 2 seconds            3306/tcp             mariadb
+    cd testsuites
+    ./run-s3p-test.sh performance 7200

 
 .. note::
     The containers on this docker-compose are running with HTTP configuration. For HTTPS, ports
     and configurations will need to be changed, as well certificates and keys must be generated
     for security.
 
 
 .. note::
     The containers on this docker-compose are running with HTTP configuration. For HTTPS, ports
     and configurations will need to be changed, as well certificates and keys must be generated
     for security.
 

-
-Install JMeter
---------------
-
-Download and install JMeter
-
-.. code-block:: bash
-
-    # Install required packages
-    sudo apt install -y wget unzip
-
-    # Install JMeter
-    mkdir -p jmeter
-    cd jmeter
-    wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.zip # check if valid version
-    unzip -q apache-jmeter-5.5.zip
-    rm apache-jmeter-5.5.zip
-
-
-Install & configure visualVM
---------------------------------------
-
-VisualVM needs to be installed in the virtual machine running Distribution. It will be used to
-monitor CPU, Memory and GC for Distribution while the stability tests are running.
-
-.. code-block:: bash
-
-    sudo apt install -y visualvm
-
-Run these commands to configure permissions (if permission errors happens, use ``sudo su``)
-
-.. code-block:: bash
-
-    # Set globally accessable permissions on policy file
-    sudo chmod 777 /usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy
-
-    # Create Java security policy file for VisualVM
-    sudo cat > /usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy << EOF
-    grant codebase "jrt:/jdk.jstatd" {
-       permission java.security.AllPermission;
-    };
-    grant codebase "jrt:/jdk.internal.jvmstat" {
-       permission java.security.AllPermission;
-    };
-    EOF
-
-Run the following command to start jstatd using port 1111
-
-.. code-block:: bash
-
-    /usr/lib/jvm/java-11-openjdk-amd64/bin/jstatd -p 1111 -J-Djava.security.policy=/usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy &
-
-Run visualVM to connect to POLICY_DISTRIBUTION_IP:9090
-
-.. code-block:: bash
-
-    # Get the Policy Distribution container IP
-    echo $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' policy-distribution)
-
-    # Start visual vm
-    visualvm &
-
-This will load up the visualVM GUI
+The script will load up the visualvm GUI on your virtual machine. You will need to manually connect
+it to the distribution JMX port.

 
 Connect to Distribution JMX Port.
 
     1. On the visualvm toolbar, click on "Add JMX Connection"
 
 Connect to Distribution JMX Port.
 
     1. On the visualvm toolbar, click on "Add JMX Connection"

-    2. Enter the Distribution container IP and Port 9090. This is the JMX port exposed by the
+    2. Enter localhost as the IP address and Port 9090. This is the JMX port exposed by the

        distribution container
        distribution container

-    3. Double click on the newly added nodes under "Remotes" to start monitoring CPU, Memory & GC.
+    3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC.

 
 Example Screenshot of visualVM
 
 .. image:: distribution-s3p-results/distribution-visualvm-snapshot.png
 
 
 Example Screenshot of visualVM
 
 .. image:: distribution-s3p-results/distribution-visualvm-snapshot.png
 

+Teardown Docker
+
+Once the testing is finished, you can tear down the docker setup from **./testsuites** with:
+
+.. code-block:: bash
+
+    docker-compose -f stability/src/main/resources/setup/docker-compose.yml down

 
 Stability Test of Policy Distribution
 +++++++++++++++++++++++++++++++++++++
 
 Stability Test of Policy Distribution
 +++++++++++++++++++++++++++++++++++++


@@ -237,7 +98,6 @@ The 72h stability test will run the following steps sequentially in a single thr
 - **Delete Old CSAR** - Checks if CSAR already exists in the watched directory, if so it deletes it
 - **Add CSAR** - Adds CSAR to the directory that distribution is watching
 - **Get Healthcheck** - Ensures Healthcheck is returning 200 OK
 - **Delete Old CSAR** - Checks if CSAR already exists in the watched directory, if so it deletes it
 - **Add CSAR** - Adds CSAR to the directory that distribution is watching
 - **Get Healthcheck** - Ensures Healthcheck is returning 200 OK

-- **Get Statistics** - Ensures Statistics is returning 200 OK

 - **Get Metrics** - Ensures Metrics is returning 200 OK
 - **Assert PDP Group Query** - Checks that PDPGroupQuery contains the deployed policy
 - **Assert PoliciesDeployed** - Checks that the policy is deployed
 - **Get Metrics** - Ensures Metrics is returning 200 OK
 - **Assert PDP Group Query** - Checks that PDPGroupQuery contains the deployed policy
 - **Assert PoliciesDeployed** - Checks that the policy is deployed


@@ -268,33 +128,21 @@ Screenshot of Distribution stability test plan
 Running the Test Plan
 ---------------------
 
 Running the Test Plan
 ---------------------
 

-Check if the /tmp/policydistribution/distributionmount exists as it was created during the start.sh
-script execution. If not, run the following commands to create folder and change folder permissions
-to allow the testplan to insert the CSAR into the /tmp/policydistribution/distributionmount folder.
-
-.. note::
-    Make sure that only csar file is being loaded in the watched folder and log generation is in a
-    logs folder, as any sort of zip file can be understood by distribution as a policy file. A
-    logback.xml configuration file is available under setup/distribution folder.
-
-.. code-block:: bash
-
-    sudo mkdir -p /tmp/policydistribution/distributionmount
-    sudo chmod -R a+trwx /tmp
-
-
-Navigate to the stability test folder.
+The main script takes care of everything. To run the 72 hour stability tests do as follows

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    cd ~/distribution/testsuites/stability/src/main/resources/testplans/
+    cd testsuites
+    ./run-s3p-test.sh stability 259200

 
 

-Execute the run_test.sh
+* visualvm produces the monitor and threads - we can screenshot those and add them to the test results
+* A jmeter .jtl file is produced by the run - it is called distribution-stability.jtl
+* The file can be imported into the jmeter GUI to view statistics
+* The application performance index table can be produced with jmeter on the cli as below:ls

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    ./run_test.sh
-
+    jmeter -n -t your_test_plan.jmx -l test_results.jtl -e -o report_directory

 
 Test Results
 ------------
 
 Test Results
 ------------


@@ -332,7 +180,8 @@ It also tests that distribution can handle multiple policy CSARs and that these
 Setup Details
 -------------
 
 Setup Details
 -------------
 

-The performance test is based on the same setup as the distribution stability tests.
+The performance test is based on the same setup as the distribution stability tests. This setup is done by the main
+**run-s3p-test.sh** script

 
 
 Test Plan Sequence
 
 
 Test Plan Sequence


@@ -350,23 +199,23 @@ Performance test plan is different from the stability test plan.
 Running the Test Plan
 ---------------------
 
 Running the Test Plan
 ---------------------
 

-Check if /tmp folder permissions to allow the Testplan to insert the CSAR into the
-/tmp/policydistribution/distributionmount folder.
-Clean up from previous run. If necessary, put containers down with script ``down.sh`` from setup
-folder mentioned on :ref:`Setup components <setup-distribution-s3p-components>`
+The main script takes care of everything. To run the 4 hour performance tests do as follows

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    sudo mkdir -p /tmp/policydistribution/distributionmount
-    sudo chmod -R a+trwx /tmp
+    cd testsuites
+    ./run-s3p-test.sh performance 14400

 
 

-Navigate to the testplan folder and execute the test script:
+* visualvm produces the monitor and threads - we can screenshot those and add them to the test results
+* A jmeter .jtl file is produced by the run - it is called distribution-performance.jtl
+* The file can be imported into the jmeter GUI to view statistics
+* The application performance index table can be produced with jmeter on the cli as below:

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    cd ~/distribution/testsuites/performance/src/main/resources/testplans/
-    ./run_test.sh
-    
+    jmeter -n -t your_test_plan.jmx -l test_results.jtl -e -o report_directory
+
+This produced html pages where statistics tables can be seen and added to the results.

 
 Test Results
 ------------
 
 Test Results
 ------------