Both the Stability and the Performance tests were executed in a full ONAP OOM deployment in Nordix lab.
-
Setup Details
+++++++++++++
-- APEX-PDP along with all policy components deployed as part of a full ONAP OOM deployment.
-- Policy-models-simulator is deployed to use CDS and DMaaP simulators during policy execution.
+Deploying ONAP using OOM
+------------------------
+
+APEX-PDP along with all policy components are deployed as part of a full ONAP OOM deployment.
+At a minimum, the following ONAP components are needed: policy, mariadb-galera, aai, cassandra, aaf, and dmaap.
+
+Before deploying, the values.yaml files are changed to use NodePort instead of ClusterIP for policy-api,
+policy-pap, and policy-apex-pdp, so that they are accessible from jmeter::
+
+ policy-apex-pdp NodePort 10.43.131.43 <none> 6969:31739/TCP
+ policy-api NodePort 10.43.67.153 <none> 6969:30430/TCP
+ policy-pap NodePort 10.43.200.57 <none> 6969:30585/TCP
+
+The node ports (31739, 30430 and 30585 above) are used in JMeter.
+The HOSTNAMEs for JMeter are set to the IPs returned by running "kubectl get node -o wide"
+and to find the applications for each node by running "kubectl describe node <node-name>".
+
+Set up policy-models-simulator
+------------------------------
+
+Policy-models-simulator is deployed to use CDS and DMaaP simulators during policy execution.
Simulator configurations used are available in apex-pdp repository:
testsuites/apex-pdp-stability/src/main/resources/simulatorConfig/
-- Two APEX policies are executed in the APEX-PDP engine, and are triggered by multiple threads during the tests.
-- Both tests were run via jMeter.
+
+It is run as a docker image from a node accessible to the kubernetes cluster::
+
+ docker run -d --rm --publish 6680:6680 --publish 31054:3905 \
+ --volume "apex-pdp/testsuites/apex-pdp-stability/src/main/resources/simulatorConfig:/opt/app/policy/simulators/etc/mounted" \
+ nexus3.onap.org:10001/onap/policy-models-simulator:2.7-SNAPSHOT-latest
+
+The published ports 6680 and 31054 are used in JMeter for CDS and DMaaP simulators.
+
+Creation of VNF & PNF in AAI
+----------------------------
+
+In order for APEX-PDP engine to fetch the resource details from AAI during runtime execution, we need to create dummy
+VNF & PNF entities in AAI. In a real control loop flow, the entities in AAI will be either created during orchestration
+phase or provisioned in AAI separately.
+
+Download & execute the steps in postman collection for creating the entities along with it’s dependencies.
+The steps needs to be performed sequentially one after another. And no input is required from user.
+
+:download:`Create VNF & PNF in AAI for Apex S3P <postman/create-vnf-pnf-aai-for-apex-s3p.postman_collection.json>`
+
+Make sure to skip the delete VNF & PNF steps.
+
+JMeter Tests
+------------
+
+Two APEX policies are executed in the APEX-PDP engine, and are triggered by multiple threads during the tests.
+Both tests were run via jMeter.
Stability test script is available in apex-pdp repository:
testsuites/apex-pdp-stability/src/main/resources/apexPdpStabilityTestPlan.jmx
testsuites/performance/performance-benchmark-test/src/main/resources/apexPdpPerformanceTestPlan.jmx
.. Note::
- Policy executions are validated in a more strict fashion during the tests.
- There are test cases where upto 80 events are expected on the DMaaP topic.
+ Policy executions are validated in a stricter fashion during the tests.
+ There are test cases where up to 80 events are expected on the DMaaP topic.
DMaaP simulator is used to keep it simple and avoid any message pickup timing related issues.
Stability Test of APEX-PDP
- **Delete Policy onap.policies.apex.Simplecontrolloop** - delete the first APEX policy using policy/api component.
- **Delete Policy onap.policies.apex.Example** - delete the second APEX policy also using policy/api component.
+Test Configuration
+------------------
The following steps can be used to configure the parameters of test plan.
- **HTTP Authorization Manager** - used to store user/password authentication details.
- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests.
-- **User Defined Variables** - used to store following user defined parameters.
+- **User Defined Variables** - used to store following user defined parameters.
=================== ===============================================================================
**Name** **Description**
=================== ===============================================================================
HOSTNAME IP Address or host name to access the components
PAP_PORT Port number of PAP for making REST API calls such as deploy/undeploy of policy
- API_PORT Port number of API for making REST API calls such as create/ delete of policy
+ API_PORT Port number of API for making REST API calls such as create/delete of policy
APEX_PORT Port number of APEX for making REST API calls such as healthcheck/metrics
+ SIM_HOST IP Address or hostname running policy-models-simulator
+ DMAAP_PORT Port number of DMaaP simulator for making REST API calls such as reading notification events
+ CDS_PORT Port number of CDS simulator
wait Wait time if required after a request (in milliseconds)
threads Number of threads to run test cases in parallel
threadsTimeOutInMs Synchronization timer for threads running in parallel (in milliseconds)
.. code-block:: bash
- nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t apexPdpStabilityTestPlan.jmx -l stabilityTestResults.jtl
+ nohup ./apache-jmeter-5.4.3/bin/jmeter.sh -n -t apexPdpStabilityTestPlan.jmx -l stabilityTestResults.jtl
Test Results
------------
======================= ================= ================== ==================================
**Total # of requests** **Success %** **Error %** **Average time taken per request**
======================= ================= ================== ==================================
-428661 100 % 0.00 % 162 ms
+430397 100 % 0.00 % 151.694 ms
======================= ================= ================== ==================================
.. Note::
**JMeter Screenshot**
-.. image:: apex-s3p-results/apex_stability_jmeter_results.JPG
+.. image:: apex-s3p-results/apex_stability_jmeter_results.png
**Memory and CPU usage**
Memory and CPU usage before test execution:
-.. image:: apex-s3p-results/apex_top_before_72h.JPG
+.. image:: apex-s3p-results/apex_top_before_72h.png
:download:`Prometheus metrics before 72h test <apex-s3p-results/apex_metrics_before_72h.txt>`
Memory and CPU usage after test execution:
-.. image:: apex-s3p-results/apex_top_after_72h.JPG
+.. image:: apex-s3p-results/apex_top_after_72h.png
:download:`Prometheus metrics after 72h test <apex-s3p-results/apex_metrics_after_72h.txt>`
.. code-block:: bash
- nohup ./apache-jmeter-5.4.1/bin/jmeter.sh -n -t apexPdpPerformanceTestPlan.jmx -l perftestresults.jtl
+ nohup ./apache-jmeter-5.4.3/bin/jmeter.sh -n -t apexPdpPerformanceTestPlan.jmx -l perftestresults.jtl
Test Results
======================= ================= ================== ==================================
**Total # of requests** **Success %** **Error %** **Average time taken per request**
======================= ================= ================== ==================================
-46946 100 % 0.00 % 198 ms
+47567 100 % 0.00 % 163.841 ms
======================= ================= ================== ==================================
**JMeter Screenshot**
-.. image:: apex-s3p-results/apex_perf_jmeter_results.JPG
+.. image:: apex-s3p-results/apex_perf_jmeter_results.png
Summary
+++++++
-Multiple policies were executed in a multi threaded fashion for both stability and performance tests.
+Multiple policies were executed in a multi-threaded fashion for both stability and performance tests.
Both tests ran smoothly without any issues.
Code Review / policy / parent.git / blobdiff