X-Git-Url: https://gerrit.onap.org/r/gitweb?a=blobdiff_plain;ds=sidebyside;f=docs%2Fdevelopment%2Fdevtools%2Fdrools-s3p.rst;h=1586379e6c862f5044d55fdabbc5e2c2ccd70eb4;hb=e8a4d3adf8847a2eefdad7d167295b56d85407c4;hp=18bd489844f149335daf4ced9df17357cc5e1450;hpb=86dae8cdfa10cdeb0bf99fc5d104039b826f510d;p=policy%2Fparent.git diff --git a/docs/development/devtools/drools-s3p.rst b/docs/development/devtools/drools-s3p.rst index 18bd4898..1586379e 100644 --- a/docs/development/devtools/drools-s3p.rst +++ b/docs/development/devtools/drools-s3p.rst @@ -10,22 +10,34 @@ Policy Drools PDP component ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Both the Performance and the Stability tests were executed against a default ONAP installation in the policy-k8s tenant in the windriver lab, from an independent VM running the jmeter tool to inject the load. +Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant +in the windriver lab, from an independent VM running the jmeter tool to inject the load. General Setup ************* -The kubernetes installation allocated all policy components in the same worker node VM and some additional ones. -The worker VM hosting the policy components has the following spec: +The installation runs the following components in a single VM: + +- AAF +- AAI +- DMAAP +- POLICY + +The VM has the following hardware spec: + +- 126GB RAM +- 12 VCPUs +- 155GB Ephemeral Disk + +Jmeter is run from a different VM with the following configuration: - 16GB RAM -- 8 VCPU -- 160GB Ephemeral Disk +- 8 VCPUs +- 155GB Ephemeral Disk -The standalone VM designated to run jmeter has the same configuration. The jmeter JVM -was instantiated with a max heap configuration of 12G. +The drools-pdp container uses the JVM memory settings from a default OOM installation. -Other ONAP components used during the stability tests are: +Other ONAP components exercised during the stability tests were: - Policy XACML PDP to process guard queries for each transaction. - DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions. @@ -39,47 +51,29 @@ The following components are simulated during the tests. - APPC responses for the vCPE and vFW use cases. - AAI to answer queries for the use cases under test. -In order to avoid interferences with the APPC component while running the tests, -the APPC component was disabled. - SO, and AAI actors were simulated within the PDP-D JVM by enabling the feature-controlloop-utils before running the tests. PDP-D Setup *********** -The kubernetes charts were modified previous to the installation with -the changes below. - -The feature-controlloop-utils was started by adding the following script: +The kubernetes charts were modified previous to the installation +to add the following script that enables the controlloop-utils feature: .. code-block:: bash oom/kubernetes/policy/charts/drools/resources/configmaps/features.pre.sh: - #!/bin/bash - bash -c "features enable controlloop-utils" + #!/bin/sh + sh -c "features enable controlloop-utils" Stability Test of Policy PDP-D ****************************** -The 72 hour stability test happened in parallel with the stability run of the API component. - -Worker Node performance -======================= - -The VM named onap-k8s-09 was monitored for the duration of the 72 hours -stability run. The table below show the usage ranges: - -.. code-block:: bash - - NAME CPU(cores) CPU% - onap-k8s-09 <=1214m <=20% - PDP-D performance ================= -The test set focused on the following use cases: +The tests focused on the following use cases: - vCPE - vDNS @@ -90,30 +84,21 @@ For 72 hours the following 5 scenarios ran in parallel: - vCPE success scenario - vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP). - vDNS success scenario. -- vDNS failure scenario. +- vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existant vserver-name reference). - vFirewall success scenario. -Five threads ran in parallel, one for each scenario. The transactions were initiated +Five threads ran in parallel, one for each scenario, back to back with no pauses. The transactions were initiated by each jmeter thread group. Each thread initiated a transaction, monitored the transaction, and -as soon as the transaction ending was detected, it initiated the next one, so back to back with no -pauses. - -All transactions completed successfully as it was expected in each scenario, with no failures. +as soon as the transaction ending was detected, it initiated the next one. -The command executed was +JMeter was run in a docker container with the following command: .. code-block:: bash - ./jmeter -n -t /home/ubuntu/drools-applications/testsuites/stability/src/main/resources/frankfurt/s3p.jmx -l /home/ubuntu/jmeter_result/jmeter.jtl -e -o /home/ubuntu/jmeter_result > /dev/null 2>&1 + docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN -The results were computed by taking the ellapsed time from the audit.log -(this log reports all end to end transactions, marking the start, end, and -ellapsed times). +The results were accessed by using the telemetry API to gather statistics: -The count reflects the number of successful transactions as expected in the -use case, as well as the average, standard deviation, and max/min. An histogram -of the response times have been added as a visual indication on the most common -transaction times. vCPE Success scenario ===================== @@ -122,9 +107,13 @@ ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e: .. code-block:: bash - Max: 4323 ms, Min: 143 ms, Average: 380 ms [samples taken for average: 260628] + # Times are in milliseconds -.. image:: images/ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e.png + Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e + Number of Transactions Executed: 114007 + Number of Successful Transactions: 112727 + Number of Failure Transactions: 1280 + Average Execution Time: 434.9942021103967 ms. vCPE Failure scenario @@ -134,9 +123,14 @@ ControlLoop-vCPE-Fail: .. code-block:: bash - Max: 3723 ms, Min: 148 ms, Average: 671 ms [samples taken for average: 87888] + # Times are in milliseconds + + Control Loop Name: ControlLoop-vCPE-Fail + Number of Transactions Executed: 114367 + Number of Successful Transactions: 114367 (failure transactions are expected) + Number of Failure Transactions: 0 (success transactions are not expected) + Average Execution Time: 433.61750330077734 ms. -.. image:: images/ControlLoop-vCPE-Fail.png vDNS Success scenario ===================== @@ -145,9 +139,14 @@ ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3: .. code-block:: bash - Max: 6437 ms, Min: 19 ms, Average: 165 ms [samples taken for average: 59259] + # Times are in milliseconds + + Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3 + Number of Transactions Executed: 237512 + Number of Successful Transactions: 229532 + Number of Failure Transactions: 7980 + Average Execution Time: 268.028794334602 ms. -.. image:: images/ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3.png vDNS Failure scenario ===================== @@ -156,9 +155,14 @@ ControlLoop-vDNS-Fail: .. code-block:: bash - Max: 1176 ms, Min: 4 ms, Average: 5 ms [samples taken for average: 340810] + # Times are in milliseconds + + Control Loop Name: ControlLoop-vDNS-Fail + Number of Transactions Executed: 1957987 + Number of Successful Transactions: 1957987 (failure transactions are expected) + Number of Failure Transactions: 0 (success transactions are not expected) + Average Execution Time: 39.369322166081794 -.. image:: images/ControlLoop-vDNS-Fail.png vFirewall Success scenario ========================== @@ -167,6 +171,43 @@ ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a: .. code-block:: bash - Max: 4016 ms, Min: 177 ms, Average: 644 ms [samples taken for average: 36460] + # Times are in milliseconds + + Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a + Number of Transactions Executed: 120308 + Number of Successful Transactions: 118895 + Number of Failure Transactions: 1413 + Average Execution Time: 394.8609236293513 ms. + + +Commentary +========== + +There has been a degradation of performance observed in this release +when compared with the previous one. +Approximately 1% of transactions were not completed as expected for +some use cases. Average Execution Times are extended as well. +The unexpected results seem to point in the direction of the +interactions of the distributed locking feature with the database. +These areas as well as the conditions for the test need to be investigated +further. + +.. code-block:: bash + + # Common pattern in the audit.log for unexpected transaction completions + + a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a| + policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]| + 2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0| + null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0| + ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0|| + dev-policy-drools-pdp-0|microservice.stringmatcher| + {vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false, + generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}|||| + INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases| + + # The "Target Lock was lost" is a common message error in the unexpected results. + + +END-OF-DOCUMENT -.. image:: images/ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a.png