Update S3P testing xacml PDP documentation

Issue-ID: POLICY-4873
Change-Id: I9137c4b75c2e43dc68e225ac86bd18a8a7175af2
Signed-off-by: FrancescoFioraEst <francesco.fiora@est.tech>

diff --git a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png
old mode 100644 (file)
new mode 100755 (executable)
index 2c27967..b5ef579
Binary files a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png and b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-perf-xacml.png differ

diff --git a/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png
new file mode 100755 (executable)
index 0000000..d45aeee
Binary files /dev/null and b/docs/development/devtools/testing/s3p/xacml-s3p-results/s3p-stability-xacml.png differ

diff --git a/docs/development/devtools/testing/s3p/xacml-s3p.rst b/docs/development/devtools/testing/s3p/xacml-s3p.rst
index 5ea2e28..2ca0a53 100644 (file)
--- a/docs/development/devtools/testing/s3p/xacml-s3p.rst
+++ b/docs/development/devtools/testing/s3p/xacml-s3p.rst
@@ -9,39 +9,48 @@
 
 ##########################
 
 
 ##########################
 

-Performance Test of Policy XACML PDP (Jakarta)
-**********************************************
+Policy XACML PDP S3P Tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~

 
 

-The Performance test was executed by performing requests
-against the Policy RESTful APIs.
+Both the Performance and the Stability tests were executed by performing requests
+against Policy components installed in Kubernetes environment. These tests were all
+performed on a Ubuntu VM with 32GB of memory, 16 CPU and 100GB of disk space.

 
 

-A default ONAP installation in the Policy tenant in UNH was used to run the tests.
+Policy XACML PDP Deployment
++++++++++++++++++++++++++++

 
 

-The Agent VMs in this lab have the following configuration:
+In an effort to allow the execution of the s3p tests to be as close to automatic as possible,
+a script will be executed that will perform the following:

 
 

-- 16GB RAM
-- 8 VCPU
+- Install of a microk8s kubernetes environment
+- Bring up the policy components
+- Checks that the components are successfully up and running before proceeding
+- Install Java 17
+- Install Jmeter locally and configure it
+- Specify whether you want to run stability or performance tests

 
 

-Summary
-=======

 
 

-The Performance test was executed, and the result analysed, via:
+The remainder of this document outlines how to run the tests and the test results

 
 

-.. code-block:: bash
+Common Setup
+++++++++++++
+The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-s3p-test.sh**.

 
 

-    jmeter -Jduration=1200 -Jusers=10 \
-        -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
-        -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 \
-        -n -t perf.jmx -l testresults.jtl
+Clone the policy-xacml-pdp repo to access the test scripts

 
 

-Note: the ports listed above correspond to port 6969 of the respective components.
+.. code-block:: bash

 
 

-The performance tests runs the following, all in parallel:
+    git clone https://gerrit.onap.org/r/policy/xacml-pdp xacml-pdp

 
 

-- Healthcheck, 10 simultaneous threads
-- Statistics, 10 simultaneous threads
-- Decisions, 10 simultaneous threads, each running the following in sequence:
+Stability Test of Policy XACML PDP
+++++++++++++++++++++++++++++++++++

 
 

+Test Plan
+---------
+The 24 hours stability test ran the following steps.
+
+- Healthcheck, 2 simultaneous threads
+- Decisions, 2 simultaneous threads, each running the following tasks in sequence:

    - Monitoring Decision
    - Monitoring Decision, abbreviated
    - Naming Decision
    - Monitoring Decision
    - Monitoring Decision, abbreviated
    - Naming Decision


@@ -50,62 +59,97 @@ The performance tests runs the following, all in parallel:
    - Frequency Limiter Guard Decision
    - Min/Max Guard Decision
 
    - Frequency Limiter Guard Decision
    - Min/Max Guard Decision
 

-When the script starts up, it uses policy-api to create, and policy-pap to deploy,
-the policies that are needed by the test.  It assumes that the "naming" policy has
-already been created and deployed.  Once the test completes, it undeploys and deletes
-the policies that it previously created.
+This runs for 24 hours. Test results are present in the **testsuites/automated-performance/s3pTestResults.jtl**
+file and in **/tmp/** directory. Logs are present for jmeter in **testsuites/automated-performance/jmeter.log** and
+**testsuites/automated-performance/nohup.out**

 
 

-Results
-=======
+Run Test
+--------

 
 

-The test was run for 20 minutes with 10 users (i.e., threads), with the following results:
+The code in the setup section also serves to run the tests. Just one execution needed to do it all.

 
 

-.. csv-table::
-   :header: "Number of Users", "Throughput (requests/second)", "Average Latency (ms)"
+.. code-block:: bash

 
 

-   10, 4603, 2
+    bash run-s3p-test.sh run stability

 
 

-.. image:: xacml-s3p-results/s3p-perf-xacml.png
+Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file.

 
 

+This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI.

 
 

-Stability Test of Policy XACML PDP
-**********************************
+Test Results
+------------

 
 

-This test was run using jmeter on a default
-ONAP installation in the Policy tenant in UNH.
+**Summary**

 
 

-The Agent VMs in this lab have the following configuration:
+Stability test plan was triggered for 24 hours.

 
 

-- 16GB RAM
-- 8 VCPU
+**Test Statistics**

 
 

-Summary
-=======
+=======================  =================  ========================  =========================
+**Total # of requests**  **Error %**        **Average Latency (ms)**  **Measured requests/sec**
+=======================  =================  ========================  =========================
+  63618103                 0.00 %             5 ms                      736.064 ms
+=======================  =================  ========================  =========================

 
 

-The stability test was performed on a default ONAP OOM installation in the Policy tenant of the UNH lab.
-JMeter injected the traffic defined in the
-`XACML PDP stability script
-<https://git.onap.org/policy/xacml-pdp/tree/testsuites/stability/src/main/resources/testplans/stability.jmx>`_
-with the following command:
+**JMeter Results**

 
 

-.. code-block:: bash
+.. image:: xacml-s3p-results/s3p-stability-xacml.png

 
 

-    jmeter.sh -Jduration=259200 -Jusers=2 -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
-        -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 --nongui --testfile stability.jmx
+**Policy component Setup**

 
 

-Note: the ports listed above correspond to port 6969 of the respective components.
+==============================================  ====================================================================  ===========================================
+**NAME**                                        **IMAGE**                                                             **PORT**
+==============================================  ====================================================================  ===========================================
+mariadb-galera-0                                docker.io/bitnami/mariadb-galera:10.5.8                               3306/TCP
+policy-models-simulator-bcd494d87-cldfr         nexus3.onap.org:10001/onap/policy-models-simulator:3.0.2-SNAPSHOT     30904/TCP
+policy-apex-pdp-0                               nexus3.onap.org:10001/onap/policy-apex-pdp:3.0.2-SNAPSHOT             30001/TCP
+prometheus-f66f97b6-kpg5c                       nexus3.onap.org:10001/prom/prometheus:latest                          30909/TCP
+policy-distribution-86777bdd78-95zjx            nexus3.onap.org:10001/onap/policy-distribution:3.0.2-SNAPSHOT         30006/TCP
+policy-clamp-ac-http-ppnt-7b99cbfbf8-d5zjk      nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:7.0.3-SNAPSHOT
+policy-clamp-ac-pf-ppnt-55c4cb99f4-pp699        nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:7.0.3-SNAPSHOT     30008/TCP
+policy-clamp-ac-k8s-ppnt-6d854cc8b6-nzlgh       nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:7.0.3-SNAPSHOT
+policy-clamp-runtime-acm-5c6d8fbfb-zww5j        nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:7.0.3-SNAPSHOT    30007/TCP
+policy-pap-847d89997d-gljzh                     nexus3.onap.org:10001/onap/policy-pap:3.0.2-SNAPSHOT                  30003/TCP
+policy-xacml-pdp-656d57d578-9h6fx               nexus3.onap.org:10001/onap/policy-xacml-pdp:3.0.2-SNAPSHOT            30004/TCP
+policy-api-58cb45fc9b-9qcx6                     nexus3.onap.org:10001/onap/policy-api:3.0.2-SNAPSHOT                  30002/tcp
+==============================================  ====================================================================  ===========================================

 
 

-The default log level of the root and org.eclipse.jetty.server.RequestLog loggers in the logback.xml
-of the XACML PDP
-(om/kubernetes/policy/components/policy-xacml-pdp/resources/config/logback.xml)
-was set to WARN since the OOM installation did have log rotation enabled of the
-container logs in the kubernetes worker nodes.
+.. Note::

 
 

-The stability test, stability.jmx, runs the following, all in parallel:
+              .. container:: paragraph
+
+                  There were no failures during the 24 hours test.
+
+The XACML PDP offered very good performance with JMeter for the traffic mix described above.
+The average transaction time is insignificant.
+
+
+Performance Test of Policy XACML PDP
+++++++++++++++++++++++++++++++++++++
+
+Introduction
+------------
+
+Performance test of acm components has the goal of testing the min/avg/max processing time and rest call throughput for all the requests with multiple requests at the same time.
+
+Setup Details
+-------------
+
+We can setup the environment and execute the tests like this from the **xacml-pdp/testsuites** directory
+
+Test Plan
+---------
+
+Performance test plan is the same as the stability test plan above except for the few differences listed below.
+
+- Increase the number of threads up to 10 (simulating 10 users' behaviours at the same time).
+- Reduce the test time to 20 minutes.
+
+The performance tests runs the following, all in parallel:
+
+- Healthcheck, 10 simultaneous threads
+- Decisions, 10 simultaneous threads, each running the following in sequence:

 
 

-- Healthcheck, 2 simultaneous threads
-- Statistics, 2 simultaneous threads
-- Decisions, 2 simultaneous threads, each running the following tasks in sequence:

    - Monitoring Decision
    - Monitoring Decision, abbreviated
    - Naming Decision
    - Monitoring Decision
    - Monitoring Decision, abbreviated
    - Naming Decision


@@ -119,16 +163,32 @@ the policies that are needed by the test.  It assumes that the "naming" policy h
 already been created and deployed.  Once the test completes, it undeploys and deletes
 the policies that it previously created.
 
 already been created and deployed.  Once the test completes, it undeploys and deletes
 the policies that it previously created.
 

-Results
-=======
+Run Test
+--------

 
 

-The stability summary results were reported by JMeter with the following summary line:
+The code in the setup section also serves to run the tests. Just one execution needed to do it all.

 
 .. code-block:: bash
 
 
 .. code-block:: bash
 

-    summary = 941639699 in 71:59:36 = 3633.2/s Avg:     1 Min:     0 Max:   842 Err:     0 (0.00%)
+    bash run-s3p-test.sh run performance

 
 

-The XACML PDP offered very good performance with JMeter for the traffic mix described above.
-The average transaction time is insignificant.  The maximum transaction time of 842 ms.
-There was a Drools stability test running in parallel, hence the actual load was higher.
+Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file and in **/tmp/** directory.
+
+This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI.
+
+Test Results
+------------
+
+**Summary**

 
 

+The test was run for 20 minutes with 10 users (i.e., threads), with the following results:
+
+**Test Statistics**
+
+=======================  =================  ========================  =========================
+**Total # of requests**  **Error %**        **Average Latency (ms)**  **Measured requests/sec**
+=======================  =================  ========================  =========================
+  912141                   0.00 %             25.5666 ms                741.917 ms
+=======================  =================  ========================  =========================
+
+.. image:: xacml-s3p-results/s3p-perf-xacml.png