X-Git-Url: https://gerrit.onap.org/r/gitweb?a=blobdiff_plain;f=docs%2Fdevelopment%2Fdevtools%2Fapex-s3p.rst;h=6a3be847da5e0c675ab4636b909d913842516a89;hb=ccc2522f2915de34c62223ab5579dabbf3a49dda;hp=3ddcfe591478b1c86004b9cab14abda5d1a4ea53;hpb=2e86eee494da758bea0e084d7bd5e8eedba7d03e;p=policy%2Fparent.git diff --git a/docs/development/devtools/apex-s3p.rst b/docs/development/devtools/apex-s3p.rst index 3ddcfe59..6a3be847 100644 --- a/docs/development/devtools/apex-s3p.rst +++ b/docs/development/devtools/apex-s3p.rst @@ -10,241 +10,249 @@ Policy APEX PDP component ~~~~~~~~~~~~~~~~~~~~~~~~~ -Setting up Stability Tests in APEX -++++++++++++++++++++++++++++++++++ +Both the Stability and the Performance tests were executed in a full ONAP OOM deployment in Nordix lab. -Introduction ------------- +Setup Details ++++++++++++++ -The 72 hour Stability Test for apex-pdp has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The pdp is configured to start a rest server inside it and take input from rest clients (JMeter) and send back output to the rest clients (JMeter). +Deploying ONAP using OOM +------------------------ -The input events will be submitted through rest interface of apex-pdp and the results are verified using the rest responses coming out from apex-pdp. +APEX-PDP along with all policy components are deployed as part of a full ONAP OOM deployment. +At a minimum, the following ONAP components are needed: policy, mariadb-galera, aai, cassandra, aaf, and dmaap. -The test will be performed in a multi-threaded environment where 20 threads running in JMeter will keep sending events to apex-pdp in every 500 milliseconds for the duration of 72 hours. +Before deploying, the values.yaml files are changed to use NodePort instead of ClusterIP for policy-api, +policy-pap, and policy-apex-pdp, so that they are accessible from jmeter:: -Setup details -------------- + policy-apex-pdp NodePort 10.43.131.43 6969:31739/TCP + policy-api NodePort 10.43.67.153 6969:30430/TCP + policy-pap NodePort 10.43.200.57 6969:30585/TCP -The stability test is performed on VM's running in OpenStack cloud environment. There are 2 seperate VM's, one for running apex pdp & other one for running JMeter to simulate steady flow of transactions. +The node ports (31739, 30430 and 30585 above) are used in JMeter. +The HOSTNAMEs for JMeter are set to the IPs returned by running "kubectl get node -o wide" +and to find the applications for each node by running "kubectl describe node ". +Set up policy-models-simulator +------------------------------ -Install & Configure VisualVM ----------------------------- +Policy-models-simulator is deployed to use CDS and DMaaP simulators during policy execution. + Simulator configurations used are available in apex-pdp repository: + testsuites/apex-pdp-stability/src/main/resources/simulatorConfig/ -VisualVM needs to be installed in the virtual machine having apex-pdp. It will be used to monitor CPU, Memory, GC for apex-pdp while stability test is running. +It is run as a docker image from a node accessible to the kubernetes cluster:: -Install visualVM + docker run -d --rm --publish 6680:6680 --publish 31054:3905 \ + --volume "apex-pdp/testsuites/apex-pdp-stability/src/main/resources/simulatorConfig:/opt/app/policy/simulators/etc/mounted" \ + nexus3.onap.org:10001/onap/policy-models-simulator:2.7-SNAPSHOT-latest -.. code-block:: bash +The published ports 6680 and 31054 are used in JMeter for CDS and DMaaP simulators. - sudo apt-get install visualvm +Creation of VNF & PNF in AAI +---------------------------- -Login to VM using graphical interface in separate terminal window. +In order for APEX-PDP engine to fetch the resource details from AAI during runtime execution, we need to create dummy +VNF & PNF entities in AAI. In a real control loop flow, the entities in AAI will be either created during orchestration +phase or provisioned in AAI separately. -.. code-block:: bash +Download & execute the steps in postman collection for creating the entities along with it’s dependencies. +The steps needs to be performed sequentially one after another. And no input is required from user. - ssh -X @ +:download:`Create VNF & PNF in AAI for Apex S3P ` -Open visualVM +Make sure to skip the delete VNF & PNF steps. -.. code-block:: bash +JMeter Tests +------------ + +Two APEX policies are executed in the APEX-PDP engine, and are triggered by multiple threads during the tests. +Both tests were run via jMeter. - visualvm & + Stability test script is available in apex-pdp repository: + testsuites/apex-pdp-stability/src/main/resources/apexPdpStabilityTestPlan.jmx -Connect to apex-pdp JVM's JMX agent -1. Right click on "Local" in the left panel of the screen and select "Add Local JMX Connection..." -2. Enter localhost:9911 for "Connection", and click OK -3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC. + Performance test script is available in apex-pdp repository: + testsuites/performance/performance-benchmark-test/src/main/resources/apexPdpPerformanceTestPlan.jmx -Sample Screenshot of visualVM +.. Note:: + Policy executions are validated in a stricter fashion during the tests. + There are test cases where up to 80 events are expected on the DMaaP topic. + DMaaP simulator is used to keep it simple and avoid any message pickup timing related issues. -.. image:: images/stability-visualvm1.PNG -.. image:: images/stability-visualvm2.PNG +Stability Test of APEX-PDP +++++++++++++++++++++++++++ Test Plan --------- -The 72 hours stability test will run the following steps in 5 threaded loop. - -- **Send Input Event** - sends an input message to rest interface of apex-pdp. -- **Assert Response Code** - assert the response code coming from apex-pdp. -- **Assert Response Message** - assert the response message coming from apex-pdp. - -The following steps can be used to configure the parameters of test plan. - -- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests. -- **HTTP Request Defaults** - used to store HTTP request details like Server Name or IP, Port, Protocol etc. -- **User Defined Variables** - used to store following user defined parameters. +The 72 hours stability test ran the following steps. -================== ============================================================================ ============================ -**Name** **Description** **Default Value** -================== ============================================================================ ============================ -wait Wait time after each request (in milliseconds) 500 -threads Number of threads to run test cases in parallel. 5 -threadsTimeOutInMs Synchronization timer for threads running in parallel (in milliseconds). 5000 -================== ============================================================================ ============================ +Setup Phase +""""""""""" +Policies are created and deployed to APEX-PDP during this phase. Only one thread is in action and this step is done only once. -Download and update the jmx file presented in the apex-pdp git repository - `jmx file path `_. +- **Create Policy onap.policies.apex.Simplecontrolloop** - creates the first APEX policy using policy/api component. + This is a sample policy used for PNF testing. +- **Create Policy onap.policies.apex.Example** - creates the second APEX policy using policy/api component. + This is a sample policy used for VNF testing. +- **Deploy Policies** - Deploy both the policies created to APEX-PDP using policy/pap component -- HTTPSampler.domain - The ip address of VM which the apex container is running -- HTTPSampler.port - The listening port, here is 23324 -- ThreadGroup.druation - Set the duration to 72 hours (in seconds) +Main Phase +"""""""""" -Use the CLI mode to start the test +Once the policies are created and deployed to APEX-PDP by the setup thread, five threads execute the below tests for 72 hours. -.. code-block:: bash +- **Healthcheck** - checks the health status of APEX-PDP +- **Prometheus Metrics** - checks that APEX-PDP is exposing prometheus metrics +- **Test Simplecontrolloop policy success case** - Send a trigger event to *unauthenticated.DCAE_CL_OUTPUT* DMaaP topic. + If the policy execution is successful, 3 different notification events are sent to *APEX-CL-MGT* topic by each one of the 5 threads. + So, it is checked if 15 notification messages are received in total on *APEX-CL-MGT* topic with the relevant messages. +- **Test Simplecontrolloop policy failure case** - Send a trigger event with invalid pnfName to *unauthenticated.DCAE_CL_OUTPUT* DMaaP topic. + The policy execution is expected to fail due to AAI failure response. 2 notification events are expected on *APEX-CL-MGT* topic by a thread in this case. + It is checked if 10 notification messages are received in total on *APEX-CL-MGT* topic with the relevant messages. +- **Test Example policy success case** - Send a trigger event to *unauthenticated.DCAE_POLICY_EXAMPLE_OUTPUT* DMaaP topic. + If the policy execution is successful, 4 different notification events are sent to *APEX-CL-MGT* topic by each one of the 5 threads. + So, it is checked if 20 notification messages are received in total on *APEX-CL-MGT* topic with the relevant messages. +- **Test Example policy failure case** - Send a trigger event with invalid vnfName to *unauthenticated.DCAE_POLICY_EXAMPLE_OUTPUT* DMaaP topic. + The policy execution is expected to fail due to AAI failure response. 2 notification events are expected on *APEX-CL-MGT* topic by a thread in this case. + So, it is checked if 10 notification messages are received in total on *APEX-CL-MGT* topic with the relevant messages. +- **Clean up DMaaP notification topic** - DMaaP notification topic which is *APEX-CL-MGT* is cleaned up after each test to make sure that one failure doesn't lead to cascading errors. - ./jmeter.sh -n -t ~/apexPdpStabilityTestPlan.jmx -Jusers=1 -l ~/stability.log -Stability Test Result ---------------------- +Teardown Phase +"""""""""""""" -**Summary** +Policies are undeployed from APEX-PDP and deleted during this phase. +Only one thread is in action and this step is done only once after the Main phase is complete. -Stability test plan was triggered for 72 hours injecting input events to apex-pdp from 5 client threads. - -Once the test has complete - we can generate a HTML test report via the following command - -.. code-block:: bash +- **Undeploy Policies** - Undeploy both the policies from APEX-PDP using policy/pap component +- **Delete Policy onap.policies.apex.Simplecontrolloop** - delete the first APEX policy using policy/api component. +- **Delete Policy onap.policies.apex.Example** - delete the second APEX policy also using policy/api component. - ~/jMeter/apache-jmeter-5.2.1/bin/jmeter -g stability.log -o ./result/ - -============================================== ================================ ============= ============ ============================ -**Number of Client Threads running in JMeter** **Total number of input events** **Success %** **Error %** **Average Time per Request** -============================================== ================================ ============= ============ ============================ -5 129326 100% 0% 6716.12 -============================================== ================================ ============= ============ ============================ - -.. image:: images/stability-jmeter.PNG - -download:`result.zip ` +Test Configuration +------------------ +The following steps can be used to configure the parameters of test plan. -Stability Test of Apex PDP -^^^^^^^^^^^^^^^^^^^^^^^^^^ +- **HTTP Authorization Manager** - used to store user/password authentication details. +- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests. +- **User Defined Variables** - used to store following user defined parameters. + +=================== =============================================================================== + **Name** **Description** +=================== =============================================================================== + HOSTNAME IP Address or host name to access the components + PAP_PORT Port number of PAP for making REST API calls such as deploy/undeploy of policy + API_PORT Port number of API for making REST API calls such as create/delete of policy + APEX_PORT Port number of APEX for making REST API calls such as healthcheck/metrics + SIM_HOST IP Address or hostname running policy-models-simulator + DMAAP_PORT Port number of DMaaP simulator for making REST API calls such as reading notification events + CDS_PORT Port number of CDS simulator + wait Wait time if required after a request (in milliseconds) + threads Number of threads to run test cases in parallel + threadsTimeOutInMs Synchronization timer for threads running in parallel (in milliseconds) +=================== =============================================================================== -The 72 hour Stability Test for apex-pdp has the goal of introducing a steady flow of transactions using jMeter. +Run Test +-------- -The input event will be submitted through the rest interface of DMaaP , which then triggers a grpc request to CDS. Based on the response, another DMaaP event is triggered. +The test was run in the background via "nohup", to prevent it from being interrupted: -This test will be performed in an OOM deployment setup. The test will be performed in a multi-threaded environment where 5 threads running in JMeter will keep sending events for the duration of 72 hours. +.. code-block:: bash -Test Plan ---------- + nohup ./apache-jmeter-5.4.3/bin/jmeter.sh -n -t apexPdpStabilityTestPlan.jmx -l stabilityTestResults.jtl -The 72 hours stability test will run the following steps in a 5 threaded loop. +Test Results +------------ -- **Create Policy** - creates a policy using the policy/api component -- **Deploy Policy** - deploys the policy in the existing PdpGroup -- **Check Health** - checks the health status of apex -- **Send Input Event** - trigger 'unauthenticated.DCAE_CL_OUTPUT' event of DMaaP. -- **Get Output Event Response** - check for the triggered output event. -- **Undeploy Policy** - undeploys the policy from PdpGroup -- **Delete Policy** - deletes the policy using the policy/api component +**Summary** -The following steps can be used to configure the parameters of the test plan. +Stability test plan was triggered for 72 hours. There were no failures during the 72 hours test. -- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests. -- **HTTP Request Defaults** - used to store HTTP request details like Server Name or IP, Port, Protocol etc. -- **User Defined Variables** - used to store the following user defined parameters: -================== ============================================================================ ============================ -**Name** **Description** **Default Value** -================== ============================================================================ ============================ -wait Wait time after each request (in milliseconds) 120000 -threads Number of threads to run test cases in parallel. 5 -threadsTimeOutInMs Synchronization timer for threads running in parallel (in milliseconds). 150000 -PAP_PORT Port number of PAP for making REST API calls -API_PORT Port number of API for making REST API calls -APEX_PORT Port number of APEX for making REST API calls -DMAAP_PORT Port number of DMAAP for making REST API calls -================== ============================================================================ ============================ +**Test Statistics** +======================= ================= ================== ================================== +**Total # of requests** **Success %** **Error %** **Average time taken per request** +======================= ================= ================== ================================== +430397 100 % 0.00 % 151.694 ms +======================= ================= ================== ================================== -Download and update the jmx file presented in the apex-pdp git repository - `jmx file path `_. +.. Note:: -- HTTPSampler.domain - The ip address of the VM in which the apex container is running -- HTTPSampler.port - The listening port, here is 23324 -- ThreadGroup.duration - Set the duration to 72 hours (in seconds) + There were no failures during the 72 hours test. -Use the CLI mode to start the test +**JMeter Screenshot** -.. code-block:: bash +.. image:: apex-s3p-results/apex_stability_jmeter_results.png - nohup ./jmeter.sh -n -t ~/apexPdpStabilityTestPlan.jmx -Jusers=1 -l ~/stability.log +**Memory and CPU usage** +The memory and CPU usage can be monitored by running "top" command in the APEX-PDP pod. +A snapshot is taken before and after test execution to monitor the changes in resource utilization. +Prometheus metrics is also collected before and after the test execution. -Stability Test Results ----------------------- +Memory and CPU usage before test execution: -The stability test plan was triggered for 72 hours, injecting input events to apex-pdp pod from 5 client threads running in JMeter. +.. image:: apex-s3p-results/apex_top_before_72h.png -The stability tests were executed as part of a full ONAP OOM deployment in Nordix lab. +:download:`Prometheus metrics before 72h test ` -Once the tests complete, we can generate an HTML test report via the command: +Memory and CPU usage after test execution: -.. code-block:: bash +.. image:: apex-s3p-results/apex_top_after_72h.png - ~/jMeter/apache-jmeter-5.2.1/bin/jmeter -g stability.log -o ./result/ +:download:`Prometheus metrics after 72h test ` -============================================== ================================ ============= ============ ============================ -**Number of Client Threads running in JMeter** **Total number of input events** **Success %** **Error %** **Average Time per Request** -============================================== ================================ ============= ============ ============================ -5 129326 100% 0% 6716.12 -============================================== ================================ ============= ============ ============================ +Performance Test of APEX-PDP +++++++++++++++++++++++++++++ +Introduction +------------ -**JMeter Screenshot** +Performance test of APEX-PDP is done similar to the stability test, but in a more extreme manner using higher thread count. -.. image:: images/apex_s3p_jm-1.png -.. image:: images/apex_s3p_jm-2.png +Setup Details +------------- -download:`result.zip ` +The performance test is performed on a similar setup as Stability test. -Setting up Performance Tests in APEX -++++++++++++++++++++++++++++++++++++ -The Performance test is performed on a similar setup to the Stability test. JMeter will send a large number of REST requests and will then retrieve those requests. +Test Plan +--------- -Performance test plan will be the same as the stability test plan except for some differences listed below: +Performance test plan is the same as the stability test plan above except for the few differences listed below. -- Increase the number of threads from 5 to 20. -- Reduce test time to ninety minutes. -- Calculate the amount of requests handled in the time frame. +- Increase the number of threads used in the Main Phase from 5 to 20. +- Reduce the test time to 2 hours. Run Test -------- -Running the performance test will be the same as the stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* , *API_PORT* , *PAP_HOST* , *PAP_PORT* are already set up in *.jmx*. - -.. code-block:: bash - - nohup ./jmeter.sh -n -t ~/performance.jmx -Jusers=1 -l ~/perf.log - -Once the tests have completed, run the following the gather results. - .. code-block:: bash - ~/jMeter/apache-jmeter-5.2.1/bin/jmeter -g perf.log -o ./performance_result/ + nohup ./apache-jmeter-5.4.3/bin/jmeter.sh -n -t apexPdpPerformanceTestPlan.jmx -l perftestresults.jtl -Performance Test Result ------------------------ -**Summary** +Test Results +------------ -Performance test was triggered for 90 minutes. The results are shown below. +Test results are shown as below. **Test Statistics** -============================ =========== ========= ================================== -**Total Number of Requests** **Success** **Error** **Average Time Taken per Request** -============================ =========== ========= ================================== -32304 99,99 % 0.01 % 8746.50 ms -============================ =========== ========= ================================== +======================= ================= ================== ================================== +**Total # of requests** **Success %** **Error %** **Average time taken per request** +======================= ================= ================== ================================== +47567 100 % 0.00 % 163.841 ms +======================= ================= ================== ================================== **JMeter Screenshot** -.. image:: images/apex_perf_jm_1.PNG +.. image:: apex-s3p-results/apex_perf_jmeter_results.png + +Summary ++++++++ -.. image:: images/apex_perf_jm_2.PNG +Multiple policies were executed in a multi-threaded fashion for both stability and performance tests. +Both tests ran smoothly without any issues.