1 .. This work is licensed under a
2 .. Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
5 .. _distribution-s3p-label:
7 Policy Distribution component
8 #############################
10 72 Hours Stability Test of Distribution
11 +++++++++++++++++++++++++++++++++++++++
15 The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input CSAR will be added/removed by the test client(JMeter) and the result will be pulled from the backend(PAP and PolicyAPI) by the test client(JMeter).
18 The test will be performed in an environment where Jmeter will continuously add/remove a test csar into the special directory where policy distribuion is monitoring and will then get the processed results from PAP and PolicyAPI to verify the successful deployment of the policy. The policy will then be undeployed and the test will loop continuously until 72 hours have elapsed.
23 The stability test is performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are 2 separate VMs, one for running backend policy services which policy distribution needs, and the other is for policy distribution service itself and Jmeter.
25 **OpenStack environment details**
27 Version: Windriver Titanium
29 **Policy Backend VM details(VM1)**
39 Docker version 19.03.8, build afacb8b7f0
41 Java: openjdk version "11.0.7"
43 **JMeter and Distribution VM details(VM2)**
45 OS: Ubuntu 18.04.4 LTS
53 Docker version 19.03.8, build afacb8b7f0
55 Java: openjdk version "11.0.7"
59 Install Docker in VM1 & VM2
60 ---------------------------
61 Make sure to execute below commands in VM1 & VM2 both.
63 Update the ubuntu software installer
69 Install and check Java
73 $ apt-get install -y openjdk-11-jdk
76 Ensure that the Java version that is executing is OpenJDK version 11
78 Install and check Docker
82 $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
84 $ apt-cache policy docker-ce
85 $ apt-get install -y docker-ce
86 $ systemctl status docker
89 Change the permissions of the Docker socket file
93 $ sudo chmod 666 /var/run/docker.sock
95 Check the status of the Docker service and ensure it is running correctly
99 $ service docker status
102 Install Simulators, PAP, PolicyAPI and MariaDB in VM1
103 -----------------------------------------------------
105 To install all the components needed for Distribution, copy over the script and related files found within the simulatorsetup directory within $(REPOPATH)/distribution/testsuites/stability/src/main/resources
107 Run setup_components.sh script to bring up the required docker containers
109 After installation, ensure the following docker containers are up and running:
113 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
114 11195b01300a nexus3.onap.org:10001/onap/policy-pap:2.2.2-SNAPSHOT "bash ./policy-pap.sh" 13 seconds ago Up 9 seconds 0.0.0.0:7000->6969/tcp policy-pap
115 6266aa6b0137 nexus3.onap.org:10001/onap/policy-api:2.2.3-SNAPSHOT "bash ./policy-api.sh" 25 seconds ago Up 22 seconds 0.0.0.0:6969->6969/tcp policy-api
116 6a85d155aa8a pdp/simulator:latest "bash pdp-sim.sh" About a minute ago Up About a minute pdp-simulator
117 0b41992ccfd7 dmaap/simulator:latest "bash dmaap-sim.sh" About a minute ago Up About a minute 0.0.0.0:3904->3904/tcp message-router
118 595056b2a094 mariadb:10.2.14 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:3306->3306/tcp mariadb
120 Install Distribution in VM2
121 ---------------------------
123 To install the Distribution service, copy over the script and related files found within the distributionsetup directory within $(REPOPATH)/distribution/testsuites/stability/src/main/resources
125 Run setup_distribution.sh script to install the distribution service, provide the IP of VM1 as the arguments to the script.
130 $ ./setup_distribution.sh 10.2.0.24 10.2.0.24
132 Ensure the distribution container is running.
134 Install JMeter in VM2
135 ---------------------
137 Download and install jMeter
143 $ wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.1.1.zip
144 $ unzip apache-jmeter-5.1.1.zip
146 Install & configure visualVM in VM2
147 -----------------------------------
148 VisualVM needs to be installed in the virtual machine running Distrbution. It will be used to monitor CPU, Memory and GC for Distribution while the stability tests are running.
152 $ sudo apt-get install visualVM
154 Run these commands to configure permissions
158 $ cd /usr/lib/jvm/java-11-openjdk-amd64/bin/
159 $ sudo touch visualvm.policy
160 $ sudo chmod 777 visualvm.policy
164 Add the following in visualvm.policy
166 grant codebase "file:/usr/lib/jvm/java-11-openjdk-amd64/lib/tools.jar" {
167 permission java.security.AllPermission;
170 Run the following commands to start jstatd using port 1111
174 $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
175 $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &
177 Using the VM2 Desktop, run visualVM to connect to localhost:9090
184 This will load up the visualVM GUI
186 Connect to Distribution JMX Port.
188 1. Right click on "Local" in the left panel of the screen and select "Add JMX Connection"
189 2. Enter the Port 9090. this is the JMX port exposed by the dsitribution container
190 3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC.
194 Sample Screenshot of visualVM
196 .. image:: images/distribution-s3p-vvm-sample.png
201 The 72 hours stability test will run the following steps sequentially in a single threaded loop.
203 - **Delete Old CSAR** - Checks if CSAR already exists in the watched directory, if so it deletes it
204 - **Add CSAR** - Adds CSAR to the directory that distribution is watching
205 - **Get Healthcheck** - Ensures Healthcheck is returning 200 OK
206 - **Get Statistics** - Ensures Statistics is returning 200 OK
207 - **CheckPDPGroupQuery** - Checks that PDPGroupQuery contains the deployed policy
208 - **CheckPolicyDeployed** - Checks that the policy is deployed
209 - **Undeploy Policy** - Undeploys the policy
210 - **Delete Policy** - Deletes the Policy for the next loop
211 - **Check PDP Group for Deletion** - Ensures the policy has been removed and does not exist
213 The following steps can be used to configure the parameters of the test plan.
215 - **HTTP Authorization Manager** - used to store user/password authentication details.
216 - **HTTP Header Manager** - used to store headers which will be used for making HTTP requests.
217 - **User Defined Variables** - used to store following user defined parameters.
219 ========== ===============================================
220 **Name** **Description**
221 ========== ===============================================
222 PAP_HOST IP Address or host name of PAP component
223 PAP_PORT Port number of PAP for making REST API calls
224 API_HOST IP Address or host name of API component
225 API_PORT Port number of API for making REST API calls
226 DURATION Duration of Test
227 ========== ===============================================
229 Screenshot of Distribution stability test plan
231 .. image:: images/distribution-s3p-testplan.png
233 Running the Test Plan
234 ---------------------
236 Copy the Test Plans folder onto VM2
237 Edit the /tmp/ folder permissions to allow the Testplan to insert the CSAR into the /tmp/policydistribution/distributionmount/ folder
241 $ sudo chmod a+trwx /tmp
243 From the apache jMeter folder run the test, pointing it towards the stabiltiy.jmx file inside the testplans folder
247 $ ./bin/jmeter -n -t /home/rossc/testplans/stability.jmx -Jduration=259200 -l testresults.jtl
254 Stability test plan was triggered for 72 hours.
258 ======================= ================= ================== ==================================
259 **Total # of requests** **Success %** **Error %** **Average time taken per request**
260 ======================= ================= ================== ==================================
261 194313 100 % 0 % 145 ms
262 ======================= ================= ================== ==================================
264 **VisualVM Screenshot**
266 .. image:: images/distribution-vvm-monitor.png
267 .. image:: images/distribution-vvm-threads.png
269 **JMeter Screenshot**
271 .. image:: images/distribution-summary-report.png
272 .. image:: images/distribution-results-tree.png
274 Performance Test of Policy Distribution
275 +++++++++++++++++++++++++++++++++++++++
280 Performance test of distribution has the goal of testing the min/avg/max processing time and
281 rest call throughput for all the requests when the number of requests are large enough to saturate
282 the resource and find the bottleneck.
283 It also tests that distribution can handle multiple policy csar's and that these are deployed within 30 seconds consistently.
288 The performance test is based on the same setup as the distribution stability tests.
293 Performance test plan is different from the stability test plan.
294 Instead of handling one policy csar at a time, multiple csar's are deployed within the watched folder at the exact same time.
295 We then expect all policies from these csar's to be deployed within 30 seconds.
296 Alongside these, there are multithreaded tests running towards the healtchcheck and statistics endpoints of the distribution service.
301 Copy the performance test plans folder onto VM2.
302 Edit the /tmp/ folder permissions to allow the Testplan to insert the CSAR into the /tmp/policydistribution/distributionmount/ folder.
306 $ sudo chmod a+trwx /tmp
308 From the apache jMeter folder run the test, pointing it towards the stabiltiy.jmx file inside the testplans folder
312 $ ./bin/jmeter -n -t /home/rossc/testplans/performance.jmx -Jduration=259200 -l testresults.jtl
319 Performance test plan was triggered for 4 hours.
323 ======================= ================= ================== ==================================
324 **Total # of requests** **Success %** **Error %** **Average time taken per request**
325 ======================= ================= ================== ==================================
326 239819 100 % 0 % 100 ms
327 ======================= ================= ================== ==================================
329 **JMeter Screenshot**
331 .. image:: images/distribution-performance-summary-report.png
332 .. image:: images/distribution-performance-api-report.png