##########################
-Performance Test of Policy XACML PDP (Jakarta)
-**********************************************
+Policy XACML PDP S3P Tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~
-The Performance test was executed by performing requests
-against the Policy RESTful APIs.
+Both the Performance and the Stability tests were executed by performing requests
+against Policy components installed in Kubernetes environment. These tests were all
+performed on a Ubuntu VM with 32GB of memory, 16 CPU and 100GB of disk space.
-A default ONAP installation in the Policy tenant in UNH was used to run the tests.
+Policy XACML PDP Deployment
++++++++++++++++++++++++++++
-The Agent VMs in this lab have the following configuration:
+In an effort to allow the execution of the s3p tests to be as close to automatic as possible,
+a script will be executed that will perform the following:
-- 16GB RAM
-- 8 VCPU
+- Install of a microk8s kubernetes environment
+- Bring up the policy components
+- Checks that the components are successfully up and running before proceeding
+- Install Java 17
+- Install Jmeter locally and configure it
+- Specify whether you want to run stability or performance tests
-Summary
-=======
-The Performance test was executed, and the result analysed, via:
+The remainder of this document outlines how to run the tests and the test results
-.. code-block:: bash
+Common Setup
+++++++++++++
+The common setup for performance and stability tests is now automated - being carried out by a script in- **testsuites/run-s3p-test.sh**.
- jmeter -Jduration=1200 -Jusers=10 \
- -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
- -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 \
- -n -t perf.jmx -l testresults.jtl
+Clone the policy-xacml-pdp repo to access the test scripts
-Note: the ports listed above correspond to port 6969 of the respective components.
+.. code-block:: bash
-The performance tests runs the following, all in parallel:
+ git clone https://gerrit.onap.org/r/policy/xacml-pdp xacml-pdp
-- Healthcheck, 10 simultaneous threads
-- Statistics, 10 simultaneous threads
-- Decisions, 10 simultaneous threads, each running the following in sequence:
+Stability Test of Policy XACML PDP
+++++++++++++++++++++++++++++++++++
+Test Plan
+---------
+The 24 hours stability test ran the following steps.
+
+- Healthcheck, 2 simultaneous threads
+- Decisions, 2 simultaneous threads, each running the following tasks in sequence:
- Monitoring Decision
- Monitoring Decision, abbreviated
- Naming Decision
- Frequency Limiter Guard Decision
- Min/Max Guard Decision
-When the script starts up, it uses policy-api to create, and policy-pap to deploy,
-the policies that are needed by the test. It assumes that the "naming" policy has
-already been created and deployed. Once the test completes, it undeploys and deletes
-the policies that it previously created.
+This runs for 24 hours. Test results are present in the **testsuites/automated-performance/s3pTestResults.jtl**
+file and in **/tmp/** directory. Logs are present for jmeter in **testsuites/automated-performance/jmeter.log** and
+**testsuites/automated-performance/nohup.out**
-Results
-=======
+Run Test
+--------
-The test was run for 20 minutes with 10 users (i.e., threads), with the following results:
+The code in the setup section also serves to run the tests. Just one execution needed to do it all.
-.. csv-table::
- :header: "Number of Users", "Throughput (requests/second)", "Average Latency (ms)"
+.. code-block:: bash
- 10, 4603, 2
+ bash run-s3p-test.sh run stability
-.. image:: xacml-s3p-results/s3p-perf-xacml.png
+Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file.
+This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI.
-Stability Test of Policy XACML PDP
-**********************************
+Test Results
+------------
-This test was run using jmeter on a default
-ONAP installation in the Policy tenant in UNH.
+**Summary**
-The Agent VMs in this lab have the following configuration:
+Stability test plan was triggered for 24 hours.
-- 16GB RAM
-- 8 VCPU
+**Test Statistics**
-Summary
-=======
+======================= ================= ======================== =========================
+**Total # of requests** **Error %** **Average Latency (ms)** **Measured requests/sec**
+======================= ================= ======================== =========================
+ 63618103 0.00 % 5 ms 736.064 ms
+======================= ================= ======================== =========================
-The stability test was performed on a default ONAP OOM installation in the Policy tenant of the UNH lab.
-JMeter injected the traffic defined in the
-`XACML PDP stability script
-<https://git.onap.org/policy/xacml-pdp/tree/testsuites/stability/src/main/resources/testplans/stability.jmx>`_
-with the following command:
+**JMeter Results**
-.. code-block:: bash
+.. image:: xacml-s3p-results/s3p-stability-xacml.png
- jmeter.sh -Jduration=259200 -Jusers=2 -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
- -Jxacml_port=30111 -Jpap_port=30197 -Japi_port=30664 --nongui --testfile stability.jmx
+**Policy component Setup**
-Note: the ports listed above correspond to port 6969 of the respective components.
+============================================== ==================================================================== ===========================================
+**NAME** **IMAGE** **PORT**
+============================================== ==================================================================== ===========================================
+mariadb-galera-0 docker.io/bitnami/mariadb-galera:10.5.8 3306/TCP
+policy-models-simulator-bcd494d87-cldfr nexus3.onap.org:10001/onap/policy-models-simulator:3.0.2-SNAPSHOT 30904/TCP
+policy-apex-pdp-0 nexus3.onap.org:10001/onap/policy-apex-pdp:3.0.2-SNAPSHOT 30001/TCP
+prometheus-f66f97b6-kpg5c nexus3.onap.org:10001/prom/prometheus:latest 30909/TCP
+policy-distribution-86777bdd78-95zjx nexus3.onap.org:10001/onap/policy-distribution:3.0.2-SNAPSHOT 30006/TCP
+policy-clamp-ac-http-ppnt-7b99cbfbf8-d5zjk nexus3.onap.org:10001/onap/policy-clamp-ac-http-ppnt:7.0.3-SNAPSHOT
+policy-clamp-ac-pf-ppnt-55c4cb99f4-pp699 nexus3.onap.org:10001/onap/policy-clamp-ac-pf-ppnt:7.0.3-SNAPSHOT 30008/TCP
+policy-clamp-ac-k8s-ppnt-6d854cc8b6-nzlgh nexus3.onap.org:10001/onap/policy-clamp-ac-k8s-ppnt:7.0.3-SNAPSHOT
+policy-clamp-runtime-acm-5c6d8fbfb-zww5j nexus3.onap.org:10001/onap/policy-clamp-runtime-acm:7.0.3-SNAPSHOT 30007/TCP
+policy-pap-847d89997d-gljzh nexus3.onap.org:10001/onap/policy-pap:3.0.2-SNAPSHOT 30003/TCP
+policy-xacml-pdp-656d57d578-9h6fx nexus3.onap.org:10001/onap/policy-xacml-pdp:3.0.2-SNAPSHOT 30004/TCP
+policy-api-58cb45fc9b-9qcx6 nexus3.onap.org:10001/onap/policy-api:3.0.2-SNAPSHOT 30002/tcp
+============================================== ==================================================================== ===========================================
-The default log level of the root and org.eclipse.jetty.server.RequestLog loggers in the logback.xml
-of the XACML PDP
-(om/kubernetes/policy/components/policy-xacml-pdp/resources/config/logback.xml)
-was set to WARN since the OOM installation did have log rotation enabled of the
-container logs in the kubernetes worker nodes.
+.. Note::
-The stability test, stability.jmx, runs the following, all in parallel:
+ .. container:: paragraph
+
+ There were no failures during the 24 hours test.
+
+The XACML PDP offered very good performance with JMeter for the traffic mix described above.
+The average transaction time is insignificant.
+
+
+Performance Test of Policy XACML PDP
+++++++++++++++++++++++++++++++++++++
+
+Introduction
+------------
+
+Performance test of acm components has the goal of testing the min/avg/max processing time and rest call throughput for all the requests with multiple requests at the same time.
+
+Setup Details
+-------------
+
+We can setup the environment and execute the tests like this from the **xacml-pdp/testsuites** directory
+
+Test Plan
+---------
+
+Performance test plan is the same as the stability test plan above except for the few differences listed below.
+
+- Increase the number of threads up to 10 (simulating 10 users' behaviours at the same time).
+- Reduce the test time to 20 minutes.
+
+The performance tests runs the following, all in parallel:
+
+- Healthcheck, 10 simultaneous threads
+- Decisions, 10 simultaneous threads, each running the following in sequence:
-- Healthcheck, 2 simultaneous threads
-- Statistics, 2 simultaneous threads
-- Decisions, 2 simultaneous threads, each running the following tasks in sequence:
- Monitoring Decision
- Monitoring Decision, abbreviated
- Naming Decision
already been created and deployed. Once the test completes, it undeploys and deletes
the policies that it previously created.
-Results
-=======
+Run Test
+--------
-The stability summary results were reported by JMeter with the following summary line:
+The code in the setup section also serves to run the tests. Just one execution needed to do it all.
.. code-block:: bash
- summary = 941639699 in 71:59:36 = 3633.2/s Avg: 1 Min: 0 Max: 842 Err: 0 (0.00%)
+ bash run-s3p-test.sh run performance
-The XACML PDP offered very good performance with JMeter for the traffic mix described above.
-The average transaction time is insignificant. The maximum transaction time of 842 ms.
-There was a Drools stability test running in parallel, hence the actual load was higher.
+Once the test execution is completed, the results are present in the **automate-performance/s3pTestResults.jtl** file and in **/tmp/** directory.
+
+This file can be imported into the Jmeter GUI for visualization. The below results are tabulated from the GUI.
+
+Test Results
+------------
+
+**Summary**
+The test was run for 20 minutes with 10 users (i.e., threads), with the following results:
+
+**Test Statistics**
+
+======================= ================= ======================== =========================
+**Total # of requests** **Error %** **Average Latency (ms)** **Measured requests/sec**
+======================= ================= ======================== =========================
+ 912141 0.00 % 25.5666 ms 741.917 ms
+======================= ================= ======================== =========================
+
+.. image:: xacml-s3p-results/s3p-perf-xacml.png
Code Review / policy / parent.git / commitdiff
