X-Git-Url: https://gerrit.onap.org/r/gitweb?a=blobdiff_plain;f=docs%2Fdevelopment%2Fdevtools%2Fpap-s3p.rst;h=b42d7eb02bf0a6cae2e856017155ec79123d5fd1;hb=68128f0c849e2b103876e42108734fb660b13644;hp=2002327f5541076bec16825013b007e66492ec8d;hpb=4750e51e7329f0ca2798316903c6a4c04fb2f017;p=policy%2Fparent.git diff --git a/docs/development/devtools/pap-s3p.rst b/docs/development/devtools/pap-s3p.rst index 2002327f..b42d7eb0 100644 --- a/docs/development/devtools/pap-s3p.rst +++ b/docs/development/devtools/pap-s3p.rst @@ -10,260 +10,75 @@ Policy PAP component ~~~~~~~~~~~~~~~~~~~~ -72 Hours Stability Test of PAP -++++++++++++++++++++++++++++++ +Both the Performance and the Stability tests were executed by performing requests +against Policy components installed as part of a full ONAP OOM deployment in Nordix lab. -Introduction ------------- - -The 72 hour Stability Test for PAP has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter for the duration of 72 hours. - -Setup details -------------- - -The stability test is performed on VM's running in OpenStack cloud environment. - -There are 2 seperate VM's, one for running PAP & other one for running JMeter to simulate steady flow of transactions. - -All the dependencies like mariadb, dmaap simulator, pdp simulator & policy/api component are installed in the VM having JMeter. - -For simplicity lets assume - -VM1 will be running JMeter, MariaDB, DMaaP simulator, PDP simulator & API component. - -VM2 will be running only PAP component. - -**OpenStack environment details** - -Version: Mitaka - -**PAP VM details (VM2)** - -OS:Ubuntu 16.04 LTS - -CPU: 4 core - -RAM: 4 GB - -HardDisk: 40 GB - -Docker version 19.03.8 - -Java: openjdk version "11.0.7" 2020-04-14 - -**JMeter VM details (VM1)** - -OS: Ubuntu 16.04 LTS - -CPU: 4 core - -RAM: 4 GB - -HardDisk: 40 GB - -Docker Version: 18.09.6 - -Java: openjdk version "11.0.7" 2020-04-14 - -JMeter: 5.2.1 - -Install Docker in VM1 & VM2 ---------------------------- - -Make sure to execute below commands in VM1 & VM2 both. - -Make the etc/hosts entries - -.. code-block:: bash - - $ echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts - -Make the DNS entries - -.. code-block:: bash - - $ echo "nameserver " >> /etc/resolvconf/resolv.conf.d/head - $ echo "nameserver " >> /etc/resolvconf/resolv.conf.d/head - $ resolvconf -u - -Update the ubuntu software installer - -.. code-block:: bash - - $ apt-get update - -Check and Install Java - -.. code-block:: bash - - $ apt-get install -y openjdk-11-jdk - $ java -version - -Ensure that the Java version that is executing is OpenJDK version 8 - - -Check and install docker - -.. code-block:: bash - - $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - $ add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" - $ apt-get update - $ apt-cache policy docker-ce - $ apt-get install -y docker-ce - $ systemctl status docker - $ docker ps - -Change the permissions of the Docker socket file - -.. code-block:: bash - - $ chmod 777 /var/run/docker.sock - -Check the status of the Docker service and ensure it is running correctly - -.. code-block:: bash - - $ service docker status - $ docker ps - -Install JMeter in VM1 ---------------------- - -Download & install JMeter - -.. code-block:: bash - - $ mkdir jMeter - $ cd jMeter - $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.2.1.zip - $ unzip apache-jmeter-5.2.1.zip - -Run JMeter - -.. code-block:: bash - - $ /home/ubuntu/jMeter/apache-jmeter-5.2.1/bin/jmeter - -The above command will load the JMeter UI. Then navigate to File → Open → Browse and select the test plan jmx file to open. -The jmx file is present in the policy/pap git repository. - -Install simulators in VM1 -------------------------- - -For installing simulator, there is a script placed at `install simulator script `_ - -Copy the script & all related files in virtual machine and run it. - -After installation make sure that following 4 docker containers are up and running. - -.. code-block:: bash - - root@policytest-policytest-3-p5djn6as2477:/home/ubuntu/simulator# docker ps - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - 887efa8dac12 nexus3.onap.org:10001/onap/policy-api "bash ./policy-api.sh" 6 days ago Up 6 days 0.0.0.0:6969->6969/tcp policy-api - 0a931c0a63ac pdp/simulator:latest "bash pdp-sim.sh" 6 days ago Up 6 days pdp-simulator - a41adcb32afb dmaap/simulator:latest "bash dmaap-sim.sh" 6 days ago Up 6 days 0.0.0.0:3904->3904/tcp dmaap-simulator - d52d6b750ba0 mariadb:10.2.14 "docker-entrypoint.s…" 6 days ago Up 6 days 0.0.0.0:3306->3306/tcp mariadb - -Install PAP in VM2 ------------------- - -For installing PAP, there is a script placed at `install pap script `_ - -Copy the script & all related files in virtual machine and run it. - -After installation make sure that following docker container is up and running. - -.. code-block:: bash - - root@policytest-policytest-0-uc3y2h5x6p4j:/home/ubuntu/pap# docker ps - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - 42ac0ed4b713 nexus3.onap.org:10001/onap/policy-pap:2.2.3-SNAPSHOT "bash ./policy-pap.sh" 3 days ago Up 3 days 0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp policy-pap - -Install & configure visualVM in VM2 ------------------------------------ - -visualVM needs to be installed in the virtual machine having PAP. It will be used to monitor CPU, Memory, GC for PAP while stability test is running. - -Install visualVM - -.. code-block:: bash - - $ sudo apt-get install visualvm - -Run few commands to configure permissions - -.. code-block:: bash - - $ cd /usr/lib/jvm/java-11-openjdk-amd64/bin/ - $ sudo touch visualvm.policy - $ sudo chmod 777 visualvm.policy - - $ vi visualvm.policy - - Add the following in visualvm.policy - - - grant codebase "file:/usr/lib/jvm/java-11-openjdk-amd64/lib/tools.jar" { - permission java.security.AllPermission; - }; - -Run following commands to start jstatd using port 1111 - -.. code-block:: bash - - $ cd /usr/lib/jvm/java-11-openjdk-amd64/bin/ - $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy & - -Run visualVM locally to connect to remote VM2 - -.. code-block:: bash - - # On your windows machine or your linux box locally, launch visualVM - $ nohup visualvm - -Connect to jstatd & remote apex-pdp JVM - - 1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..." - 2. Enter the IP address of VM2. - 3. Right click on IP address, select "Add JMX Connection..." - 4. Enter the VM2 IP Address (from step 2) :9090 ( for example -10.12.6.201:9090) and click OK. - 5. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC. +Setup Details ++++++++++++++ -Sample Screenshot of visualVM +- Policy-PAP along with all policy components deployed as part of a full ONAP OOM deployment. +- A second instance of APEX-PDP is spun up in the setup. Update the configuration file (OnapPfConfig.json) such that the PDP can register to the new group created by PAP in the tests. +- Both tests were run via jMeter. -.. image:: images/pap-s3p-vvm-sample.png +Stability Test of PAP ++++++++++++++++++++++ Test Plan --------- +The 72 hours stability test ran the following steps sequentially in a single threaded loop. -The 72 hours stability test will run the following steps sequentially in a single threaded loop. +Setup Phase (steps running only once) +""""""""""""""""""""""""""""""""""""" -- **Create Policy Type** - creates an operational policy type using policy/api component -- **Create Policy defaultDomain** - creates an operational policy using the policy type created in the above step using policy/api component -- **Create Policy sampleDomain** - creates an operational policy using the policy type created in the above step using policy/api component -- **Check Health** - checks the health status of pap -- **Check Statistics** - checks the statistics of pap -- **Change state to ACTIVE** - changes the state of defaultGroup PdpGroup to ACTIVE -- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the ACTIVE state. -- **Deploy defaultDomain Policy** - deploys the policy defaultDomain in the existing PdpGroup +- **Create Policy for defaultGroup** - creates an operational policy using policy/api component +- **Create NodeTemplate metadata for sampleGroup policy** - creates a node template containing metadata using policy/api component +- **Create Policy for sampleGroup** - creates an operational policy that refers to the metadata created above using policy/api component +- **Change defaultGroup state to ACTIVE** - changes the state of defaultGroup PdpGroup to ACTIVE - **Create/Update PDP Group** - creates a new PDPGroup named sampleGroup. -- **OS Process Sampler** - OS Process Sampler to start a new Pdp Instance + A second instance of the PDP that is already spun up gets registered to this new group +- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that both PdpGroups are in ACTIVE state. + +PAP Test Flow (steps running in a loop for 72 hours) +"""""""""""""""""""""""""""""""""""""""""""""""""""" + +- **Check Health** - checks the health status of pap +- **PAP Metrics** - Fetch prometheus metrics before the deployment/undeployment cycle + Save different counters such as deploy/undeploy-success/failure counters at API and engine level. +- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that both PdpGroups are in the ACTIVE state. +- **Deploy Policy for defaultGroup** - deploys the policy defaultDomain to defaultGroup +- **Check status of defaultGroup policy** - checks the status of defaultGroup PdpGroup with the defaultDomain policy 1.0.0. +- **Check PdpGroup Audit defaultGroup** - checks the audit information for the defaultGroup PdpGroup. +- **Check PdpGroup Audit Policy (defaultGroup)** - checks the audit information for the defaultGroup PdpGroup with the defaultDomain policy 1.0.0. - **Check PdpGroup Query** - makes a PdpGroup query request and verifies that 2 PdpGroups are in the ACTIVE state and defaultGroup has a policy deployed on it. -- **Deployment Update sampleDomain** - deploys the policy sampleDomain in sampleGroup PdpGroup using pap api +- **Deployment Update for sampleGroup policy** - deploys the policy sampleDomain in sampleGroup PdpGroup using pap api +- **Check status of sampleGroup** - checks the status of the sampleGroup PdpGroup. +- **Check status of PdpGroups** - checks the status of both PdpGroups. - **Check PdpGroup Query** - makes a PdpGroup query request and verifies that the defaultGroup has a policy defaultDomain deployed on it and sampleGroup has policy sampleDomain deployed on it. +- **Check Audit** - checks the audit information for all PdpGroups. +- **Check Consolidated Health** - checks the consolidated health status of all policy components. - **Check Deployed Policies** - checks for all the deployed policies using pap api. -- **OS Process Sampler** - OS Process Sampler to stop the newly created Pdp Instance -- **Undeploy Policy sampleDomain** - undeploys the policy sampleDomain from sampleGroup PdpGroup using pap api -- **Undeploy Default Policy** - undeploys the policy defaultDomain from PdpGroup +- **Undeploy policy in sampleGroup** - undeploys the policy sampleDomain from sampleGroup PdpGroup using pap api +- **Undeploy policy in defaultGroup** - undeploys the policy defaultDomain from PdpGroup +- **Check status of policies** - checks the status of all policies and make sure both the policies are undeployed +- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the PASSIVE state. +- **PAP Metrics after deployments** - Fetch prometheus metrics after the deployment/undeployment cycle + Save the new counter values such as deploy/undeploy-success/failure counters at API and engine level, and check that the deploySuccess and undeploySuccess counters are increased by 2. + +.. Note:: + To avoid putting a large Constant Timer value after every deployment/undeployment, the status API is polled until the deployment/undeployment + is successfully completed, or until a timeout. This is to make sure that the operation is completed successfully and the PDPs gets enough time to respond back. + Otherwise, before the deployment is marked successful by PAP, an undeployment could be triggered as part of other tests, + and the operation's corresponding prometheus counter at engine level will not get updated. + +Teardown Phase (steps running only once after PAP Test Flow is completed) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + - **Change state to PASSIVE(sampleGroup)** - changes the state of sampleGroup PdpGroup to PASSIVE -- **Delete PdpGroup SampleGroup** - delete the sampleGroup PdpGroup using pap api +- **Delete PdpGroup sampleGroup** - delete the sampleGroup PdpGroup using pap api - **Change State to PASSIVE(defaultGroup)** - changes the state of defaultGroup PdpGroup to PASSIVE -- **Check PdpGroup Query** - makes a PdpGroup query request and verifies that PdpGroup is in the PASSIVE state. -- **Delete Policy defaultDomain** - deletes the operational policy defaultDomain using policy/api component -- **Delete Policy sampleDomain** - deletes the operational policy sampleDomain using policy/api component -- **Delete Policy Type** - deletes the operational policy type using policy/api component +- **Delete policy created for defaultGroup** - deletes the operational policy defaultDomain using policy/api component +- **Delete Policy created for sampleGroup** - deletes the operational policy sampleDomain using policy/api component +- **Delete Nodetemplate metadata for sampleGroup policy** - deleted the nodetemplate containing metadata for sampleGroup policy The following steps can be used to configure the parameters of test plan. @@ -278,70 +93,56 @@ The following steps can be used to configure the parameters of test plan. PAP_PORT Port number of PAP for making REST API calls API_HOST IP Address or host name of API component API_PORT Port number of API for making REST API calls - DIR Path where the pdp instance startup and stop script is placed - CONFIG_DIR Path where the pdp default Config file is placed =========== =================================================================== -Screenshot of PAP stability test plan +The test was run in the background via "nohup", to prevent it from being interrupted: -.. image:: images/pap-s3p-testplan.png +.. code-block:: bash + + nohup apache-jmeter-5.5/bin/jmeter -n -t stability.jmx -l stabilityTestResults.jtl & Test Results ------------ **Summary** -Stability test plan was triggered for 72 hours. +Stability test plan was triggered for 72 hours. There were no failures during the 72 hours test. + **Test Statistics** ======================= ================= ================== ================================== **Total # of requests** **Success %** **Error %** **Average time taken per request** ======================= ================= ================== ================================== -178208 100 % 0 % 76 ms + 102290 100 % 0.15 % 782 ms ======================= ================= ================== ================================== -**VisualVM Screenshot** +.. Note:: -.. image:: images/pap-s3p-vvm-1.png -.. image:: images/pap-s3p-vvm-2.png + There were 0.15% failures during the 72 hours test, due to the timing between the update of the metric "undeploySuccessCount" and the Undeploy itself. + We suggest for the next test to increase the timeout timing up to 130s between "Undeploy policy in defaultGroup" and "PAP Metrics after deployments" **JMeter Screenshot** -.. image:: images/pap-s3p-jm-1.png -.. image:: images/pap-s3p-jm-1.png +.. image:: pap-s3p-results/pap_stability_jmeter_results.png -Test Results Frankfurt release -------------------------------- +**Memory and CPU usage** -**Summary** +The memory and CPU usage can be monitored by running "top" command in the PAP pod. +A snapshot is taken before and after test execution to monitor the changes in resource utilization. +Prometheus metrics is also collected before and after the test execution. -Stability test plan was triggered for 72 hours. +Memory and CPU usage before test execution: -.. Note:: +.. image:: pap-s3p-results/pap_top_before_72h.png - .. container:: paragraph +:download:`Prometheus metrics before 72h test ` - Test cases for starting and stopping the PDP Instance has been included in the - test plan. These test cases have resulted in a spike in the Average time taken per request. +Memory and CPU usage after test execution: -**Test Statistics** +.. image:: pap-s3p-results/pap_top_after_72h.png -======================= ================= ================== ================================== -**Total # of requests** **Success %** **Error %** **Average time taken per request** -======================= ================= ================== ================================== - 29423 100 % 0 % 948 ms -======================= ================= ================== ================================== - -**VisualVM Screenshot** - -.. image:: images/pap-s3p-vvm-1_F.png -.. image:: images/pap-s3p-vvm-2_F.png - -**JMeter Screenshot** - -.. image:: images/pap-s3p-jm-1_F.png -.. image:: images/pap-s3p-jm-1_F.png +:download:`Prometheus metrics after 72h test ` Performance Test of PAP ++++++++++++++++++++++++ @@ -356,43 +157,42 @@ Setup Details The performance test is performed on a similar setup as Stability test. The JMeter VM will be sending a large number of REST requests to the PAP component and collecting the statistics. + Test Plan --------- Performance test plan is the same as the stability test plan above except for the few differences listed below. -- Increase the number of threads up to 5 (simulating 5 users' behaviours at the same time). +- Increase the number of threads up to 10 (simulating 10 users' behaviours at the same time). - Reduce the test time to 2 hours. -- Usage of counters to create different groups by the 'Create/Update PDP Group' test case. -- Usage of If-Controller for 'Deploy defaultDomain Policy' and 'Undeploy defaultDomain Policy' test cases to install and uninstall the Default policy only in one thread. -- OS Process Sampler for starting and stopping the PDP Instance has been disabled in the performance test plan for a better performance check. +- Usage of counters (simulating each user) to create different pdpGroups, update their state and later delete them. +- Removed the tests to deploy policies to newly created groups as this will need a larger setup with multiple pdps registered to each group, which will also slow down the performance test with the time needed for registration process etc. +- Usage of counters (simulating each user) to create different drools policies and deploy them to defaultGroup. + In the test, a thread count of 10 is used resulting in 10 different drools policies getting deployed and undeployed continuously for 2 hours. + Other standard operations like checking the deployment status of policies, checking the metrics, health etc remains. Run Test -------- Running/Triggering the performance test will be the same as the stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* , *API_PORT* , *PAP_HOST* , *PAP_PORT* are already set up in *.jmx*. -Once the test execution is completed, execute the below script to get the statistics: - .. code-block:: bash - $ cd /home/ubuntu/pap/testsuites/performance/src/main/resources/testplans - $ ./results.sh /home/ubuntu/pap_perf/resultTree.log + nohup apache-jmeter-5.5/bin/jmeter -n -t performance.jmx -l performanceTestResults.jtl & Test Results ------------ -Test results are shown as below. Overall, the test was running smoothly and successfully. We do see some minor failed transactions, especially in the 'Deploy' and 'Undeploy' Pap API in a multi-threaded fashion . +Test results are shown as below. **Test Statistics** -======================= ================= ================== ================================== ======================= -**Total # of requests** **Success %** **Error %** **Average time taken per request** **Requests/sec** -======================= ================= ================== ================================== ======================= - 25743 99.5 % 0.50 % 397 ms 5148 -======================= ================= ================== ================================== ======================= +======================= ================= ================== ================================== +**Total # of requests** **Success %** **Error %** **Average time taken per request** +======================= ================= ================== ================================== +19886 100 % 0.00 % 3107 ms +======================= ================= ================== ================================== **JMeter Screenshot** -.. image:: images/pap-perf-jm-1_F.png -.. image:: images/pap-perf-jm-2_F.png \ No newline at end of file +.. image:: pap-s3p-results/pap_performance_jmeter_results.png