Policy PAP component
~~~~~~~~~~~~~~~~~~~~
-72 Hours Stability Test of PAP
-++++++++++++++++++++++++++++++
+Both the Performance and the Stability tests were executed by performing requests
+against Policy components installed as part of a full ONAP OOM deployment in Nordix lab.
-Introduction
-------------
-
-The 72 hour Stability Test for PAP has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter for the duration of 72 hours.
-
-Setup details
--------------
-
-The stability test is performed on VM's running in OpenStack cloud environment.
-
-There are 2 seperate VM's, one for running PAP & other one for running JMeter to simulate steady flow of transactions.
-
-All the dependencies like mariadb, dmaap simulator, pdp simulator & policy/api component are installed in the VM having JMeter.
-
-For simplicity lets assume
-
-VM1 will be running JMeter, MariaDB, DMaaP simulator, PDP simulator & API component.
-
-VM2 will be running only PAP component.
-
-**OpenStack environment details**
-
-Version: Mitaka
-
-**PAP VM details (VM2)**
-
-OS:Ubuntu 16.04 LTS
-
-CPU: 4 core
-
-RAM: 4 GB
-
-HardDisk: 40 GB
-
-Docker version 19.03.8
-
-Java: openjdk version "11.0.7" 2020-04-14
-
-**JMeter VM details (VM1)**
-
-OS: Ubuntu 16.04 LTS
-
-CPU: 4 core
-
-RAM: 4 GB
-
-HardDisk: 40 GB
-
-Docker Version: 18.09.6
-
-Java: openjdk version "11.0.7" 2020-04-14
-
-JMeter: 5.2.1
-
-Install Docker in VM1 & VM2
----------------------------
-
-Make sure to execute below commands in VM1 & VM2 both.
-
-Make the etc/hosts entries
-
-.. code-block:: bash
-
- $ echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts
-
-Make the DNS entries
-
-.. code-block:: bash
-
- $ echo "nameserver <PrimaryDNSIPIP>" >> /etc/resolvconf/resolv.conf.d/head
- $ echo "nameserver <SecondaryDNSIP>" >> /etc/resolvconf/resolv.conf.d/head
- $ resolvconf -u
-
-Update the ubuntu software installer
-
-.. code-block:: bash
-
- $ apt-get update
-
-Check and Install Java
-
-.. code-block:: bash
-
- $ apt-get install -y openjdk-11-jdk
- $ java -version
-
-Ensure that the Java version that is executing is OpenJDK version 8
-
-
-Check and install docker
-
-.. code-block:: bash
-
- $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- $ add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- $ apt-get update
- $ apt-cache policy docker-ce
- $ apt-get install -y docker-ce
- $ systemctl status docker
- $ docker ps
-
-Change the permissions of the Docker socket file
-
-.. code-block:: bash
-
- $ chmod 777 /var/run/docker.sock
-
-Check the status of the Docker service and ensure it is running correctly
-
-.. code-block:: bash
-
- $ service docker status
- $ docker ps
-
-Install JMeter in VM1
----------------------
-
-Download & install JMeter
-
-.. code-block:: bash
-
- $ mkdir jMeter
- $ cd jMeter
- $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.2.1.zip
- $ unzip apache-jmeter-5.2.1.zip
-
-Run JMeter
-
-.. code-block:: bash
-
- $ /home/ubuntu/jMeter/apache-jmeter-5.2.1/bin/jmeter
-
-The above command will load the JMeter UI. Then navigate to File ā Open ā Browse and select the test plan jmx file to open.
-The jmx file is present in the policy/pap git repository.
-
-Install simulators in VM1
--------------------------
-
-For installing simulator, there is a script placed at `install simulator script <https://gerrit.onap.org/r/gitweb?p=policy/pap.git;a=blob;f=testsuites/stability/src/main/resources/simulatorsetup/setup_components.sh;h=86de3c1efcb468431a2395eef610db209a613fc3;hb=refs/heads/master>`_
-
-Copy the script & all related files in virtual machine and run it.
-
-After installation make sure that following 4 docker containers are up and running.
-
-.. code-block:: bash
-
- root@policytest-policytest-3-p5djn6as2477:/home/ubuntu/simulator# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 887efa8dac12 nexus3.onap.org:10001/onap/policy-api "bash ./policy-api.sh" 6 days ago Up 6 days 0.0.0.0:6969->6969/tcp policy-api
- 0a931c0a63ac pdp/simulator:latest "bash pdp-sim.sh" 6 days ago Up 6 days pdp-simulator
- a41adcb32afb dmaap/simulator:latest "bash dmaap-sim.sh" 6 days ago Up 6 days 0.0.0.0:3904->3904/tcp dmaap-simulator
- d52d6b750ba0 mariadb:10.2.14 "docker-entrypoint.sā¦" 6 days ago Up 6 days 0.0.0.0:3306->3306/tcp mariadb
-
-Install PAP in VM2
-------------------
-
-For installing PAP, there is a script placed at `install pap script <https://gerrit.onap.org/r/gitweb?p=policy/pap.git;a=blob;f=testsuites/stability/src/main/resources/papsetup/setup_pap.sh;h=dc5e69e76da9f48f6b23cc012e14148f1373d1e1;hb=refs/heads/master>`_
-
-Copy the script & all related files in virtual machine and run it.
-
-After installation make sure that following docker container is up and running.
-
-.. code-block:: bash
-
- root@policytest-policytest-0-uc3y2h5x6p4j:/home/ubuntu/pap# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 42ac0ed4b713 nexus3.onap.org:10001/onap/policy-pap:2.2.3-SNAPSHOT "bash ./policy-pap.sh" 3 days ago Up 3 days 0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp policy-pap
-
-Install & configure visualVM in VM2
------------------------------------
-
-visualVM needs to be installed in the virtual machine having PAP. It will be used to monitor CPU, Memory, GC for PAP while stability test is running.
-
-Install visualVM
-
-.. code-block:: bash
-
- $ sudo apt-get install visualvm
-
-Run few commands to configure permissions
-
-.. code-block:: bash
-
- $ cd /usr/lib/jvm/java-11-openjdk-amd64/bin/
- $ sudo touch visualvm.policy
- $ sudo chmod 777 visualvm.policy
-
- $ vi visualvm.policy
-
- Add the following in visualvm.policy
-
-
- grant codebase "file:/usr/lib/jvm/java-11-openjdk-amd64/lib/tools.jar" {
- permission java.security.AllPermission;
- };
-
-Run following commands to start jstatd using port 1111
-
-.. code-block:: bash
-
- $ cd /usr/lib/jvm/java-11-openjdk-amd64/bin/
- $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &
-
-Run visualVM locally to connect to remote VM2
-
-.. code-block:: bash
-
- # On your windows machine or your linux box locally, launch visualVM
- $ nohup visualvm
-
-Connect to jstatd & remote apex-pdp JVM
-
- 1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
- 2. Enter the IP address of VM2.
- 3. Right click on IP address, select "Add JMX Connection..."
- 4. Enter the VM2 IP Address (from step 2) <IP address>:9090 ( for example -10.12.6.201:9090) and click OK.
- 5. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.
+Setup Details
++++++++++++++
-Sample Screenshot of visualVM
+- Policy-PAP along with all policy components deployed as part of a full ONAP OOM deployment.
+- A second instance of APEX-PDP is spun up in the setup. Update the configuration file (OnapPfConfig.json) such that the PDP can register to the new group created by PAP in the tests.
+- Both tests were run via jMeter.
-.. image:: images/pap-s3p-vvm-sample.png
+Stability Test of PAP
++++++++++++++++++++++
Test Plan
---------
+The 72 hours stability test ran the following steps sequentially in a single threaded loop.
-The 72 hours stability test will run the following steps sequentially in a single threaded loop.
+Setup Phase (steps running only once)
+"""""""""""""""""""""""""""""""""""""
-- **Create Policy Type** - creates an operational policy type using policy/api component
-- **Create Policy defaultDomain** - creates an operational policy using the policy type created in the above step using policy/api component
-- **Create Policy sampleDomain** - creates an operational policy using the policy type created in the above step using policy/api component
-- **Check Health** - checks the health status of pap
-- **Check Statistics** - checks the statistics of pap
-- **Change state to ACTIVE** - changes the state of defaultGroup PdpGroup to ACTIVE
-- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the ACTIVE state.
-- **Deploy defaultDomain Policy** - deploys the policy defaultDomain in the existing PdpGroup
+- **Create Policy for defaultGroup** - creates an operational policy using policy/api component
+- **Create NodeTemplate metadata for sampleGroup policy** - creates a node template containing metadata using policy/api component
+- **Create Policy for sampleGroup** - creates an operational policy that refers to the metadata created above using policy/api component
+- **Change defaultGroup state to ACTIVE** - changes the state of defaultGroup PdpGroup to ACTIVE
- **Create/Update PDP Group** - creates a new PDPGroup named sampleGroup.
-- **OS Process Sampler** - OS Process Sampler to start a new Pdp Instance
+ A second instance of the PDP that is already spun up gets registered to this new group
+- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that both PdpGroups are in ACTIVE state.
+
+PAP Test Flow (steps running in a loop for 72 hours)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+- **Check Health** - checks the health status of pap
+- **PAP Metrics** - Fetch prometheus metrics before the deployment/undeployment cycle
+ Save different counters such as deploy/undeploy-success/failure counters at API and engine level.
+- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that both PdpGroups are in the ACTIVE state.
+- **Deploy Policy for defaultGroup** - deploys the policy defaultDomain to defaultGroup
+- **Check status of defaultGroup policy** - checks the status of defaultGroup PdpGroup with the defaultDomain policy 1.0.0.
+- **Check PdpGroup Audit defaultGroup** - checks the audit information for the defaultGroup PdpGroup.
+- **Check PdpGroup Audit Policy (defaultGroup)** - checks the audit information for the defaultGroup PdpGroup with the defaultDomain policy 1.0.0.
- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that 2 PdpGroups are in the ACTIVE state and defaultGroup has a policy deployed on it.
-- **Deployment Update sampleDomain** - deploys the policy sampleDomain in sampleGroup PdpGroup using pap api
+- **Deployment Update for sampleGroup policy** - deploys the policy sampleDomain in sampleGroup PdpGroup using pap api
+- **Check status of sampleGroup** - checks the status of the sampleGroup PdpGroup.
+- **Check status of PdpGroups** - checks the status of both PdpGroups.
- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that the defaultGroup has a policy defaultDomain deployed on it and sampleGroup has policy sampleDomain deployed on it.
+- **Check Audit** - checks the audit information for all PdpGroups.
+- **Check Consolidated Health** - checks the consolidated health status of all policy components.
- **Check Deployed Policies** - checks for all the deployed policies using pap api.
-- **OS Process Sampler** - OS Process Sampler to stop the newly created Pdp Instance
-- **Undeploy Policy sampleDomain** - undeploys the policy sampleDomain from sampleGroup PdpGroup using pap api
-- **Undeploy Default Policy** - undeploys the policy defaultDomain from PdpGroup
+- **Undeploy policy in sampleGroup** - undeploys the policy sampleDomain from sampleGroup PdpGroup using pap api
+- **Undeploy policy in defaultGroup** - undeploys the policy defaultDomain from PdpGroup
+- **Check status of policies** - checks the status of all policies and make sure both the policies are undeployed
+- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the PASSIVE state.
+- **PAP Metrics after deployments** - Fetch prometheus metrics after the deployment/undeployment cycle
+ Save the new counter values such as deploy/undeploy-success/failure counters at API and engine level, and check that the deploySuccess and undeploySuccess counters are increased by 2.
+
+.. Note::
+ To avoid putting a large Constant Timer value after every deployment/undeployment, the status API is polled until the deployment/undeployment
+ is successfully completed, or until a timeout. This is to make sure that the operation is completed successfully and the PDPs gets enough time to respond back.
+ Otherwise, before the deployment is marked successful by PAP, an undeployment could be triggered as part of other tests,
+ and the operation's corresponding prometheus counter at engine level will not get updated.
+
+Teardown Phase (steps running only once after PAP Test Flow is completed)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
- **Change state to PASSIVE(sampleGroup)** - changes the state of sampleGroup PdpGroup to PASSIVE
-- **Delete PdpGroup SampleGroup** - delete the sampleGroup PdpGroup using pap api
+- **Delete PdpGroup sampleGroup** - delete the sampleGroup PdpGroup using pap api
- **Change State to PASSIVE(defaultGroup)** - changes the state of defaultGroup PdpGroup to PASSIVE
-- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the PASSIVE state.
-- **Delete Policy defaultDomain** - deletes the operational policy defaultDomain using policy/api component
-- **Delete Policy sampleDomain** - deletes the operational policy sampleDomain using policy/api component
-- **Delete Policy Type** - deletes the operational policy type using policy/api component
+- **Delete policy created for defaultGroup** - deletes the operational policy defaultDomain using policy/api component
+- **Delete Policy created for sampleGroup** - deletes the operational policy sampleDomain using policy/api component
+- **Delete Nodetemplate metadata for sampleGroup policy** - deleted the nodetemplate containing metadata for sampleGroup policy
The following steps can be used to configure the parameters of test plan.
PAP_PORT Port number of PAP for making REST API calls
API_HOST IP Address or host name of API component
API_PORT Port number of API for making REST API calls
- DIR Path where the pdp instance startup and stop script is placed
- CONFIG_DIR Path where the pdp default Config file is placed
=========== ===================================================================
-Screenshot of PAP stability test plan
+The test was run in the background via "nohup", to prevent it from being interrupted:
-.. image:: images/pap-s3p-testplan.png
+.. code-block:: bash
+
+ nohup apache-jmeter-5.5/bin/jmeter -n -t stability.jmx -l stabilityTestResults.jtl &
Test Results
------------
**Summary**
-Stability test plan was triggered for 72 hours.
+Stability test plan was triggered for 72 hours. There were no failures during the 72 hours test.
+
**Test Statistics**
======================= ================= ================== ==================================
**Total # of requests** **Success %** **Error %** **Average time taken per request**
======================= ================= ================== ==================================
-178208 100 % 0 % 76 ms
+ 102290 100 % 0.15 % 782 ms
======================= ================= ================== ==================================
-**VisualVM Screenshot**
+.. Note::
-.. image:: images/pap-s3p-vvm-1.png
-.. image:: images/pap-s3p-vvm-2.png
+ There were 0.15% failures during the 72 hours test, due to the timing between the update of the metric "undeploySuccessCount" and the Undeploy itself.
+ We suggest for the next test to increase the timeout timing up to 130s between "Undeploy policy in defaultGroup" and "PAP Metrics after deployments"
**JMeter Screenshot**
-.. image:: images/pap-s3p-jm-1.png
-.. image:: images/pap-s3p-jm-1.png
+.. image:: pap-s3p-results/pap_stability_jmeter_results.png
-Test Results Frankfurt release
--------------------------------
+**Memory and CPU usage**
-**Summary**
+The memory and CPU usage can be monitored by running "top" command in the PAP pod.
+A snapshot is taken before and after test execution to monitor the changes in resource utilization.
+Prometheus metrics is also collected before and after the test execution.
-Stability test plan was triggered for 72 hours.
+Memory and CPU usage before test execution:
-.. Note::
+.. image:: pap-s3p-results/pap_top_before_72h.png
- .. container:: paragraph
+:download:`Prometheus metrics before 72h test <pap-s3p-results/pap_metrics_before_72h.txt>`
- Test cases for starting and stopping the PDP Instance has been included in the
- test plan. These test cases have resulted in a spike in the Average time taken per request.
+Memory and CPU usage after test execution:
-**Test Statistics**
+.. image:: pap-s3p-results/pap_top_after_72h.png
-======================= ================= ================== ==================================
-**Total # of requests** **Success %** **Error %** **Average time taken per request**
-======================= ================= ================== ==================================
- 29423 100 % 0 % 948 ms
-======================= ================= ================== ==================================
-
-**VisualVM Screenshot**
-
-.. image:: images/pap-s3p-vvm-1_F.png
-.. image:: images/pap-s3p-vvm-2_F.png
-
-**JMeter Screenshot**
-
-.. image:: images/pap-s3p-jm-1_F.png
-.. image:: images/pap-s3p-jm-1_F.png
+:download:`Prometheus metrics after 72h test <pap-s3p-results/pap_metrics_after_72h.txt>`
Performance Test of PAP
++++++++++++++++++++++++
The performance test is performed on a similar setup as Stability test. The JMeter VM will be sending a large number of REST requests to the PAP component and collecting the statistics.
+
Test Plan
---------
Performance test plan is the same as the stability test plan above except for the few differences listed below.
-- Increase the number of threads up to 5 (simulating 5 users' behaviours at the same time).
+- Increase the number of threads up to 10 (simulating 10 users' behaviours at the same time).
- Reduce the test time to 2 hours.
-- Usage of counters to create different groups by the 'Create/Update PDP Group' test case.
-- Usage of If-Controller for 'Deploy defaultDomain Policy' and 'Undeploy defaultDomain Policy' test cases to install and uninstall the Default policy only in one thread.
-- OS Process Sampler for starting and stopping the PDP Instance has been disabled in the performance test plan for a better performance check.
+- Usage of counters (simulating each user) to create different pdpGroups, update their state and later delete them.
+- Removed the tests to deploy policies to newly created groups as this will need a larger setup with multiple pdps registered to each group, which will also slow down the performance test with the time needed for registration process etc.
+- Usage of counters (simulating each user) to create different drools policies and deploy them to defaultGroup.
+ In the test, a thread count of 10 is used resulting in 10 different drools policies getting deployed and undeployed continuously for 2 hours.
+ Other standard operations like checking the deployment status of policies, checking the metrics, health etc remains.
Run Test
--------
Running/Triggering the performance test will be the same as the stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* , *API_PORT* , *PAP_HOST* , *PAP_PORT* are already set up in *.jmx*.
-Once the test execution is completed, execute the below script to get the statistics:
-
.. code-block:: bash
- $ cd /home/ubuntu/pap/testsuites/performance/src/main/resources/testplans
- $ ./results.sh /home/ubuntu/pap_perf/resultTree.log
+ nohup apache-jmeter-5.5/bin/jmeter -n -t performance.jmx -l performanceTestResults.jtl &
Test Results
------------
-Test results are shown as below. Overall, the test was running smoothly and successfully. We do see some minor failed transactions, especially in the 'Deploy' and 'Undeploy' Pap API in a multi-threaded fashion .
+Test results are shown as below.
**Test Statistics**
-======================= ================= ================== ================================== =======================
-**Total # of requests** **Success %** **Error %** **Average time taken per request** **Requests/sec**
-======================= ================= ================== ================================== =======================
- 25743 99.5 % 0.50 % 397 ms 5148
-======================= ================= ================== ================================== =======================
+======================= ================= ================== ==================================
+**Total # of requests** **Success %** **Error %** **Average time taken per request**
+======================= ================= ================== ==================================
+19886 100 % 0.00 % 3107 ms
+======================= ================= ================== ==================================
**JMeter Screenshot**
-.. image:: images/pap-perf-jm-1_F.png
-.. image:: images/pap-perf-jm-2_F.png
\ No newline at end of file
+.. image:: pap-s3p-results/pap_performance_jmeter_results.png
Code Review / policy / parent.git / blobdiff