Policy Drools PDP component
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Both the Performance and the Stability tests were executed against a default ONAP installation in the policy-k8s tenant in the windriver lab, from an independent VM running the jmeter tool to inject the load.
+Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant
+in the windriver lab, from an independent VM running the jmeter tool to inject the load.
General Setup
*************
-The kubernetes installation allocated all policy components in the same worker node VM and some additional ones.
-The worker VM hosting the policy components has the following spec:
+The installation runs the following components in a single VM:
+
+- AAF
+- AAI
+- DMAAP
+- POLICY
+
+The VM has the following hardware spec:
+
+- 126GB RAM
+- 12 VCPUs
+- 155GB Ephemeral Disk
+
+Jmeter is run from a different VM with the following configuration:
- 16GB RAM
-- 8 VCPU
-- 160GB Ephemeral Disk
+- 8 VCPUs
+- 155GB Ephemeral Disk
-The standalone VM designated to run jmeter has the same configuration. The jmeter JVM
-was instantiated with a max heap configuration of 12G.
+The drools-pdp container uses the JVM memory settings from a default OOM installation.
-Other ONAP components used during the stability tests are:
+Other ONAP components exercised during the stability tests were:
- Policy XACML PDP to process guard queries for each transaction.
- DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions.
- APPC responses for the vCPE and vFW use cases.
- AAI to answer queries for the use cases under test.
-In order to avoid interferences with the APPC component while running the tests,
-the APPC component was disabled.
-
SO, and AAI actors were simulated within the PDP-D JVM by enabling the
feature-controlloop-utils before running the tests.
PDP-D Setup
***********
-The kubernetes charts were modified previous to the installation with
-the changes below.
-
-The feature-controlloop-utils was started by adding the following script:
+The kubernetes charts were modified previous to the installation
+to add the following script that enables the controlloop-utils feature:
.. code-block:: bash
oom/kubernetes/policy/charts/drools/resources/configmaps/features.pre.sh:
- #!/bin/bash
- bash -c "features enable controlloop-utils"
+ #!/bin/sh
+ sh -c "features enable controlloop-utils"
Stability Test of Policy PDP-D
******************************
-The 72 hour stability test happened in parallel with the stability run of the API component.
-
-Worker Node performance
-=======================
-
-The VM named onap-k8s-09 was monitored for the duration of the 72 hours
-stability run. The table below show the usage ranges:
-
-.. code-block:: bash
-
- NAME CPU(cores) CPU%
- onap-k8s-09 <=1214m <=20%
-
PDP-D performance
=================
-The test set focused on the following use cases:
+The tests focused on the following use cases:
- vCPE
- vDNS
- vCPE success scenario
- vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP).
- vDNS success scenario.
-- vDNS failure scenario.
+- vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existant vserver-name reference).
- vFirewall success scenario.
-Five threads ran in parallel, one for each scenario. The transactions were initiated
+Five threads ran in parallel, one for each scenario, back to back with no pauses. The transactions were initiated
by each jmeter thread group. Each thread initiated a transaction, monitored the transaction, and
-as soon as the transaction ending was detected, it initiated the next one, so back to back with no
-pauses.
-
-All transactions completed successfully as it was expected in each scenario, with no failures.
+as soon as the transaction ending was detected, it initiated the next one.
-The command executed was
+JMeter was run in a docker container with the following command:
.. code-block:: bash
- ./jmeter -n -t /home/ubuntu/drools-applications/testsuites/stability/src/main/resources/frankfurt/s3p.jmx -l /home/ubuntu/jmeter_result/jmeter.jtl -e -o /home/ubuntu/jmeter_result > /dev/null 2>&1
+ docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN
-The results were computed by taking the ellapsed time from the audit.log
-(this log reports all end to end transactions, marking the start, end, and
-ellapsed times).
+The results were accessed by using the telemetry API to gather statistics:
-The count reflects the number of successful transactions as expected in the
-use case, as well as the average, standard deviation, and max/min. An histogram
-of the response times have been added as a visual indication on the most common
-transaction times.
vCPE Success scenario
=====================
.. code-block:: bash
- Max: 4323 ms, Min: 143 ms, Average: 380 ms [samples taken for average: 260628]
+ # Times are in milliseconds
-.. image:: images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png
+ Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
+ Number of Transactions Executed: 114007
+ Number of Successful Transactions: 112727
+ Number of Failure Transactions: 1280
+ Average Execution Time: 434.9942021103967 ms.
vCPE Failure scenario
.. code-block:: bash
- Max: 3723 ms, Min: 148 ms, Average: 671 ms [samples taken for average: 87888]
+ # Times are in milliseconds
+
+ Control Loop Name: ControlLoop-vCPE-Fail
+ Number of Transactions Executed: 114367
+ Number of Successful Transactions: 114367 (failure transactions are expected)
+ Number of Failure Transactions: 0 (success transactions are not expected)
+ Average Execution Time: 433.61750330077734 ms.
-.. image:: images/ControlLoop-vCPE-Fail.png
vDNS Success scenario
=====================
.. code-block:: bash
- Max: 6437 ms, Min: 19 ms, Average: 165 ms [samples taken for average: 59259]
+ # Times are in milliseconds
+
+ Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
+ Number of Transactions Executed: 237512
+ Number of Successful Transactions: 229532
+ Number of Failure Transactions: 7980
+ Average Execution Time: 268.028794334602 ms.
-.. image:: images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png
vDNS Failure scenario
=====================
.. code-block:: bash
- Max: 1176 ms, Min: 4 ms, Average: 5 ms [samples taken for average: 340810]
+ # Times are in milliseconds
+
+ Control Loop Name: ControlLoop-vDNS-Fail
+ Number of Transactions Executed: 1957987
+ Number of Successful Transactions: 1957987 (failure transactions are expected)
+ Number of Failure Transactions: 0 (success transactions are not expected)
+ Average Execution Time: 39.369322166081794
-.. image:: images/ControlLoop-vDNS-Fail.png
vFirewall Success scenario
==========================
.. code-block:: bash
- Max: 4016 ms, Min: 177 ms, Average: 644 ms [samples taken for average: 36460]
+ # Times are in milliseconds
+
+ Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
+ Number of Transactions Executed: 120308
+ Number of Successful Transactions: 118895
+ Number of Failure Transactions: 1413
+ Average Execution Time: 394.8609236293513 ms.
+
+
+Commentary
+==========
+
+There has been a degradation of performance observed in this release
+when compared with the previous one.
+Approximately 1% of transactions were not completed as expected for
+some use cases. Average Execution Times are extended as well.
+The unexpected results seem to point in the direction of the
+interactions of the distributed locking feature with the database.
+These areas as well as the conditions for the test need to be investigated
+further.
+
+.. code-block:: bash
+
+ # Common pattern in the audit.log for unexpected transaction completions
+
+ a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a|
+ policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]|
+ 2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0|
+ null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0|
+ ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0||
+ dev-policy-drools-pdp-0|microservice.stringmatcher|
+ {vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false,
+ generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}||||
+ INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases|
+
+ # The "Target Lock was lost" is a common message error in the unexpected results.
+
+
+END-OF-DOCUMENT
-.. image:: images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png
Code Review / policy / parent.git / blobdiff