72h Stability and 4h Performance Tests of Distribution
++++++++++++++++++++++++++++++++++++++++++++++++++++++
-VM Details
-----------
-
-The stability and performance tests are performed on VM's running in the OpenStack cloud
-environment in the ONAP integration lab. There are two separate VMs, one for running backend policy
-services which policy distribution needs, and the other for the policy distribution service itself
-and Jmeter.
-
-**OpenStack environment details**
-
-- Version: Windriver Titanium
-
-**Policy Backend VM details (VM1)**
-
-- OS: Ubuntu 18.04.5 LTS
-- CPU: 8 core, Intel Xeon E3-12xx v2 (Ivy Bridge), 2693.668 MHz, 16384 kB cache
-- RAM: 32 GB
-- HardDisk: 200 GB
-- Docker version 19.03.8, build afacb8b7f0
-- Java: openjdk 11.0.8 2020-07-14
-
-
Common Setup
------------
.. code-block:: bash
$ java --version
- openjdk 11.0.8 2020-07-14
- OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
- OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)
+ openjdk 11.0.11 2021-04-20
+ OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04)
+ OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode)
Install Docker and Docker Compose
.. code-block:: bash
- # Install compose
+ # Install compose (check if version is still available or update as necessary)
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
vi ~/distribution/testsuites/stability/src/main/resources/setup/versions.sh
-Ensure the correct docker image versions are specified - e.g. for Istanbul-M4
+Ensure the correct docker image versions are specified - e.g. for Kohn-M4
-- export POLICY_DIST_VERSION=2.6.1-SNAPSHOT
+- export POLICY_DIST_VERSION=2.8-SNAPSHOT
Run the start.sh script to start the components. After installation, script will execute
``docker ps`` and show the running containers.
Creating policy-api ... done
Creating policy-pap ... done
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- f91be98ad1f4 nexus3.onap.org:10001/onap/policy-pap:2.5.1-SNAPSHOT "/opt/app/policy/pap…" 1 second ago Up Less than a second 6969/tcp policy-pap
- d92cdbe971d4 nexus3.onap.org:10001/onap/policy-api:2.5.1-SNAPSHOT "/opt/app/policy/api…" 1 second ago Up Less than a second 6969/tcp policy-api
- 9a019f5d641e nexus3.onap.org:10001/onap/policy-db-migrator:2.3.1-SNAPSHOT "/opt/app/policy/bin…" 2 seconds ago Up 1 second 6824/tcp policy-db-migrator
- 108ba238edeb nexus3.onap.org:10001/mariadb:10.5.8 "docker-entrypoint.s…" 3 seconds ago Up 1 second 3306/tcp mariadb
- bec9b223e79f nexus3.onap.org:10001/onap/policy-models-simulator:2.5.1-SNAPSHOT "simulators.sh" 3 seconds ago Up 1 second 3905/tcp simulator
- 74aa5abeeb08 nexus3.onap.org:10001/onap/policy-distribution:2.6.1-SNAPSHOT "/opt/app/policy/bin…" 3 seconds ago Up 1 second 6969/tcp, 9090/tcp policy-distribution
-
+ fa4e9bd26e60 nexus3.onap.org:10001/onap/policy-pap:2.7-SNAPSHOT-latest "/opt/app/policy/pap…" 1 second ago Up Less than a second 6969/tcp policy-pap
+ efb65dd95020 nexus3.onap.org:10001/onap/policy-api:2.7-SNAPSHOT-latest "/opt/app/policy/api…" 1 second ago Up Less than a second 6969/tcp policy-api
+ cf602c2770ba nexus3.onap.org:10001/onap/policy-db-migrator:2.5-SNAPSHOT-latest "/opt/app/policy/bin…" 2 seconds ago Up 1 second 6824/tcp policy-db-migrator
+ 99383d2fecf4 pdp/simulator "sh /opt/app/policy/…" 2 seconds ago Up 1 second pdp-simulator
+ 3c0e205c5f47 nexus3.onap.org:10001/onap/policy-models-simulator:2.7-SNAPSHOT-latest "simulators.sh" 3 seconds ago Up 2 seconds 3904/tcp simulator
+ 3ad00d90d6a3 nexus3.onap.org:10001/onap/policy-distribution:2.8-SNAPSHOT-latest "/opt/app/policy/bin…" 3 seconds ago Up 2 seconds 6969/tcp, 9090/tcp policy-distribution
+ bb0b915cdecc nexus3.onap.org:10001/mariadb:10.5.8 "docker-entrypoint.s…" 3 seconds ago Up 2 seconds 3306/tcp mariadb
.. note::
The containers on this docker-compose are running with HTTP configuration. For HTTPS, ports
# Install JMeter
mkdir -p jmeter
cd jmeter
- wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.4.1.zip
- unzip -q apache-jmeter-5.4.1.zip
- rm apache-jmeter-5.4.1.zip
+ wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.zip # check if valid version
+ unzip -q apache-jmeter-5.5.zip
+ rm apache-jmeter-5.5.zip
Install & configure visualVM
sudo apt install -y visualvm
-Run these commands to configure permissions
+Run these commands to configure permissions (if permission errors happens, use ``sudo su``)
.. code-block:: bash
Connect to Distribution JMX Port.
- 1. Right click on "Local" in the left panel of the screen and select "Add JMX Connection"
+ 1. On the visualvm toolbar, click on "Add JMX Connection"
2. Enter the Distribution container IP and Port 9090. This is the JMX port exposed by the
distribution container
- 3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC.
+ 3. Double click on the newly added nodes under "Remotes" to start monitoring CPU, Memory & GC.
Example Screenshot of visualVM
-.. image:: distribution-s3p-results/distribution-s3p-vvm-sample.png
+.. image:: distribution-s3p-results/distribution-visualvm-snapshot.png
Stability Test of Policy Distribution
- **Add CSAR** - Adds CSAR to the directory that distribution is watching
- **Get Healthcheck** - Ensures Healthcheck is returning 200 OK
- **Get Statistics** - Ensures Statistics is returning 200 OK
-- **CheckPDPGroupQuery** - Checks that PDPGroupQuery contains the deployed policy
-- **CheckPolicyDeployed** - Checks that the policy is deployed
-- **Undeploy Policy** - Undeploys the policy
-- **Delete Policy** - Deletes the Policy for the next loop
-- **Check PDP Group for Deletion** - Ensures the policy has been removed and does not exist
+- **Get Metrics** - Ensures Metrics is returning 200 OK
+- **Assert PDP Group Query** - Checks that PDPGroupQuery contains the deployed policy
+- **Assert PoliciesDeployed** - Checks that the policy is deployed
+- **Undeploy/Delete Policy** - Undeploys and deletes the Policy for the next loop
+- **Assert PDP Group Query for Deleted Policy** - Ensures the policy has been removed and does not exist
The following steps can be used to configure the parameters of the test plan.
Screenshot of Distribution stability test plan
-.. image:: distribution-s3p-results/distribution-s3p-testplan.png
+.. image:: distribution-s3p-results/distribution-jmeter-testcases.png
Running the Test Plan
script execution. If not, run the following commands to create folder and change folder permissions
to allow the testplan to insert the CSAR into the /tmp/policydistribution/distributionmount folder.
+.. note::
+ Make sure that only csar file is being loaded in the watched folder and log generation is in a
+ logs folder, as any sort of zip file can be understood by distribution as a policy file. A
+ logback.xml configuration file is available under setup/distribution folder.
+
.. code-block:: bash
sudo mkdir -p /tmp/policydistribution/distributionmount
**Test Statistics**
-.. image:: distribution-s3p-results/dist_stability_statistics.PNG
-.. image:: distribution-s3p-results/dist_stability_threshold.PNG
+.. image:: distribution-s3p-results/stability-statistics.png
+.. image:: distribution-s3p-results/stability-threshold.png
**VisualVM Screenshots**
-.. image:: distribution-s3p-results/dist_stability_monitor.PNG
-.. image:: distribution-s3p-results/dist_stability_threads.PNG
+.. image:: distribution-s3p-results/stability-monitor.png
+.. image:: distribution-s3p-results/stability-threads.png
Performance Test of Policy Distribution
saturate the resource and find the bottleneck.
It also tests that distribution can handle multiple policy CSARs and that these are deployed within
-30 seconds consistently.
+60 seconds consistently.
Setup Details
- Instead of handling one policy csar at a time, multiple csar's are deployed within the watched
folder at the exact same time.
-- We expect all policies from these csar's to be deployed within 30 seconds.
+- We expect all policies from these csar's to be deployed within 60 seconds.
- There are also multithreaded tests running towards the healthcheck and statistics endpoints of
the distribution service.
Check if /tmp folder permissions to allow the Testplan to insert the CSAR into the
/tmp/policydistribution/distributionmount folder.
-Clean up from previous run. If necessary, put containers down with script `down.sh` from setup
+Clean up from previous run. If necessary, put containers down with script ``down.sh`` from setup
folder mentioned on :ref:`Setup components <setup-distribution-s3p-components>`
.. code-block:: bash
.. image:: distribution-s3p-results/performance-monitor.png
.. image:: distribution-s3p-results/performance-threads.png
+
+End of document
Code Review / policy / parent.git / blobdiff