1 .. This work is licensed under a
2 .. Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
14 72 Hours Stability Test of Policy API
15 +++++++++++++++++++++++++++++++++++++
20 The 72 hour stability test of policy API has the goal of verifying the stability of running policy design API REST service by
21 ingesting a steady flow of transactions of policy design API calls in a multi-thread fashion to simulate multiple clients' behaviors.
22 All the transaction flows are initiated from a test client server running JMeter for the duration of 72+ hours.
27 The stability test is performed on VMs running in Intel Wind River Lab environment.
28 There are 2 seperate VMs. One for running API while the other running JMeter & other necessary components, e.g. MariaDB, to simulate steady flow of transactions.
29 For simplicity, let's assume:
31 VM1 will be running JMeter, MariaDB.
32 VM2 will be running API REST service and visualVM.
36 Intel ONAP Integration and Deployment Labs
37 `Physical Labs <https://wiki.onap.org/display/DW/Physical+Labs>`_,
38 `Wind River <https://www.windriver.com/>`_
40 **API VM Details (VM2)**
50 Docker Version: 18.09.8
52 Java: OpenJDK 1.8.0_212
54 **JMeter VM Details (VM1)**
64 Docker Version: 18.09.8
66 Java: OpenJDK 1.8.0_212
70 **Software Installation & Configuration**
74 **Install Java & Docker**
76 Make the etc/hosts entries
80 $ echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts
82 Update the Ubuntu software installer
88 Check and install Java
92 $ sudo apt-get install -y openjdk-8-jdk
95 Ensure that the Java version executing is OpenJDK version 8
97 Check and install docker
101 $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
102 $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
103 $ sudo apt-get update
104 $ sudo apt-cache policy docker-ce
105 $ sudo apt-get install -y docker-ce
106 $ systemctl status docker
109 Change the permissions of the Docker socket file
113 $ sudo chmod 777 /var/run/docker.sock
115 Check the status of the Docker service and ensure it is running correctly
119 $ service docker status
126 Download & install JMeter
132 $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.1.1.zip
133 $ unzip apache-jmeter-5.1.1.zip
135 **Install other necessary components**
137 Pull api code & run setup components script
142 $ git clone https://git.onap.org/policy/api
143 $ cd api/testsuites/stability/src/main/resources/simulatorsetup
144 $ ./setup_components.sh
146 After installation, make sure the following mariadb container is up and running
150 ubuntu@test:~/api/testsuites/stability/src/main/resources/simulatorsetup$ docker ps
151 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
152 3849ce44b86d mariadb:10.2.14 "docker-entrypoint.s…" 11 days ago Up 11 days 0.0.0.0:3306->3306/tcp mariadb
156 **Install policy-api**
158 Pull api code & run setup api script
163 $ git clone https://git.onap.org/policy/api
164 $ cd api/testsuites/stability/src/main/resources/apisetup
165 $ ./setup_api.sh <host ip running api> <host ip running mariadb>
167 After installation, make sure the following api container is up and running
171 ubuntu@tools-2:~/api/testsuites/stability/src/main/resources/apisetup$ docker ps
172 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
173 4f08f9972e55 nexus3.onap.org:10001/onap/policy-api:2.1.1-SNAPSHOT "bash ./policy-api.sh" 11 days ago Up 11 days 0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp policy-api
175 **Install & configure visualVM**
177 VisualVM needs to be installed in the virtual machine having API up and running. It will be used to monitor CPU, Memory, GC for API while stability test is running.
183 $ sudo apt-get install visualvm
185 Run few commands to configure permissions
189 $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
190 $ sudo touch visualvm.policy
191 $ sudo chmod 777 visualvm.policy
195 Add the following in visualvm.policy
198 grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
199 permission java.security.AllPermission;
202 Run following commands to start jstatd using port 1111
206 $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
207 $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &
211 **Run & configure visualVM**
213 Run visualVM by typing
219 Connect to jstatd & remote policy-api JVM
221 1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
222 2. Enter the IP address of VM2 (running policy-api)
223 3. Right click on IP address, select "Add JMX Connection..."
224 4. Enter the VM2 IP Address (from step 2) <IP address>:9090 ( for example, 10.12.6.151:9090) and click OK.
225 5. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.
227 Sample Screenshot of visualVM
229 .. image:: images/results-5.png
234 The 72+ hours stability test will be running the following steps sequentially in multi-threaded loops.
235 Thread number is set to 5 to simulate 5 API clients' behaviors (they can be calling the same policy CRUD API simultaneously).
237 **Setup Thread (will be running only once)**
239 - Get policy-api Healthcheck
240 - Get API Counter Statistics
241 - Get Preloaded Policy Types
243 **API Test Flow (5 threads running the same steps in the same loop)**
245 - Create a new TCA Policy Type with Version 1.0.0
246 - Create a new TCA Policy Type with Version 2.0.0
247 - Create a new TCA Policy Type with Version 3.0.0
248 - Create a new TCA Policy Type with Version 4.0.0
249 - Create a new TCA Policy Type with Version 5.0.0
250 - Create a new TCA Policy Type with Version 6.0.0
251 - Create a new TCA Policy Type with Version 7.0.0
252 - Create a new TCA Policy Type with Version 8.0.0
253 - Create a new TCA Policy Type with Version 9.0.0
254 - Create a new TCA Policy Type with Version 10.0.0
255 - Create a new TCA Policy Type with Version 11.0.0
257 - Get All Existing Policy Types
258 - Get All Existing Versions of the New TCA Policy Type
259 - Get Version 1.0.0 of the New TCA Policy Type
260 - Get Version 2.0.0 of the New TCA Policy Type
261 - Get Version 3.0.0 of the New TCA Policy Type
262 - Get Version 4.0.0 of the New TCA Policy Type
263 - Get Version 5.0.0 of the New TCA Policy Type
264 - Get Version 6.0.0 of the New TCA Policy Type
265 - Get Version 7.0.0 of the New TCA Policy Type
266 - Get Version 8.0.0 of the New TCA Policy Type
267 - Get Version 9.0.0 of the New TCA Policy Type
268 - Get Version 10.0.0 of the New TCA Policy Type
269 - Get Version 11.0.0 of the New TCA Policy Type
270 - Get the Latest Version of the New TCA Policy Type
272 - Create a New TCA Policy with Version 1.0.0 over the New TCA Policy Type Version 2.0.0
273 - Create a New TCA Policy with Version 2.0.0 over the New TCA Policy Type Version 2.0.0
274 - Create a New TCA Policy with Version 3.0.0 over the New TCA Policy Type Version 2.0.0
275 - Create a New TCA Policy with Version 4.0.0 over the New TCA Policy Type Version 2.0.0
276 - Create a New TCA Policy with Version 5.0.0 over the New TCA Policy Type Version 2.0.0
277 - Create a New TCA Policy with Version 6.0.0 over the New TCA Policy Type Version 2.0.0
278 - Create a New TCA Policy with Version 7.0.0 over the New TCA Policy Type Version 2.0.0
279 - Create a New TCA Policy with Version 8.0.0 over the New TCA Policy Type Version 2.0.0
280 - Create a New TCA Policy with Version 9.0.0 over the New TCA Policy Type Version 2.0.0
281 - Create a New TCA Policy with Version 10.0.0 over the New TCA Policy Type Version 2.0.0
282 - Create a New TCA Policy with Version 11.0.0 over the New TCA Policy Type Version 2.0.0
284 - Get All Existing TCA Policies
285 - Get All Existing Versions of TCA Policies
286 - Get Version 1.0.0 of the New TCA Policy
287 - Get Version 2.0.0 of the New TCA Policy
288 - Get Version 3.0.0 of the New TCA Policy
289 - Get Version 4.0.0 of the New TCA Policy
290 - Get Version 5.0.0 of the New TCA Policy
291 - Get Version 6.0.0 of the New TCA Policy
292 - Get Version 7.0.0 of the New TCA Policy
293 - Get Version 8.0.0 of the New TCA Policy
294 - Get Version 9.0.0 of the New TCA Policy
295 - Get Version 10.0.0 of the New TCA Policy
296 - Get Version 11.0.0 of the New TCA Policy
297 - Get the Latest Version of the New TCA Policy
299 - Create a New Guard Policy with Version 1
300 - Create a New Guard Policy with Version 5
301 - Create a New Guard Policy with Version 9
302 - Create a New Guard Policy with Version 12
304 - Get Version 1 of the New Guard Policy
305 - Get Version 5 of the New Guard Policy
306 - Get Version 9 of the New Guard Policy
307 - Get Version 12 of the New Guard Policy
308 - Get the Latest Version of the New Guard Policy
311 **TearDown Thread (will only be running after API Test Flow is completed)**
313 - Delete Version 2.0.0 of the New TCA Policy Type (suppose to return 409-Conflict)
314 - Delete Version 3.0.0 of the New TCA Policy Type
315 - Delete Version 4.0.0 of the New TCA Policy Type
316 - Delete Version 5.0.0 of the New TCA Policy Type
317 - Delete Version 6.0.0 of the New TCA Policy Type
318 - Delete Version 7.0.0 of the New TCA Policy Type
319 - Delete Version 8.0.0 of the New TCA Policy Type
320 - Delete Version 9.0.0 of the New TCA Policy Type
321 - Delete Version 10.0.0 of the New TCA Policy Type
322 - Delete Version 11.0.0 of the New TCA Policy Type
323 - Delete Version 1.0.0 of the New TCA Policy
324 - Delete Version 2.0.0 of the New TCA Policy
325 - Delete Version 3.0.0 of the New TCA Policy
326 - Delete Version 4.0.0 of the New TCA Policy
327 - Delete Version 5.0.0 of the New TCA Policy
328 - Delete Version 6.0.0 of the New TCA Policy
329 - Delete Version 7.0.0 of the New TCA Policy
330 - Delete Version 8.0.0 of the New TCA Policy
331 - Delete Version 9.0.0 of the New TCA Policy
332 - Delete Version 10.0.0 of the New TCA Policy
333 - Delete Version 11.0.0 of the New TCA Policy
334 - Re-Delete Version 2.0.0 of the New TCA Policy Type (will return 200 now since all TCA policies created over have been deleted)
335 - Delete Version 1 of the new Guard Policy
336 - Delete Version 5 of the new Guard Policy
337 - Delete Version 9 of the new Guard Policy
338 - Delete Version 12 of the new Guard Policy
349 $ sudo openvpn --config <path to lab ovpn key file>
351 SSH into JMeter VM (VM1)
355 $ ssh -i <path to lab ssh key file> ubuntu@<host ip of JMeter VM>
357 Run JMeter test in background for 72+ hours
362 $ nohup ./jMeter/apache-jmeter-5.1.1/bin/jmeter.sh -n -t ~/api/testsuites/stability/src/main/resources/testplans/policy_api_stability.jmx &
364 (Optional) Monitor JMeter test that is running in background (anytime after re-logging into JMeter VM - VM1)
368 $ tail -f s3p/stability.log nohup.out
376 Policy API stability test plan was triggered and running for 72+ hours without any error occurred.
380 ======================= ============= =========== =============================== =============================== ===============================
381 **Total # of requests** **Success %** **Error %** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request**
382 ======================= ============= =========== =============================== =============================== ===============================
383 49723 100% 0% 86 ms 4 ms 795 ms
384 ======================= ============= =========== =============================== =============================== ===============================
388 .. image:: images/results-5.png
389 .. image:: images/results-6.png
393 .. image:: images/results-1.png
394 .. image:: images/results-2.png
395 .. image:: images/results-3.png
396 .. image:: images/results-4.png
400 Performance Test of Policy API
401 ++++++++++++++++++++++++++++++
406 Performance test of policy-api has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck.
411 The performance test is performed on OOM-based deployment of ONAP Policy framework components in Intel Wind River Lab environment.
412 In addition, we use another VM with JMeter installed to generate the transactions.
413 The JMeter VM will be sending large number of REST requests to the policy-api component and collecting the statistics.
414 Policy-api component already knows how to communicate with MariaDB component if OOM-based deployment is working correctly.
419 Performance test plan is the same as stability test plan above.
420 Only differences are, in performance test, we increase the number of threads up to 20 (simulating 20 users' behaviors at the same time) whereas reducing the test time down to 1 hour.
425 Running/Triggering performance test will be the same as stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* and *API_PORT* are already set up in *.jmx*.
430 Test results are shown as below. Overall, the test was running smoothly and successfully. We do see some minor failed transactions, especially in POST calls which intend to write into DB simultaneously in a multi-threaded fashion . All GET calls (reading from DB) were succeeded.
432 .. image:: images/summary-1.png
433 .. image:: images/summary-2.png
434 .. image:: images/summary-3.png
435 .. image:: images/result-1.png
436 .. image:: images/result-2.png
437 .. image:: images/result-3.png
438 .. image:: images/result-4.png
439 .. image:: images/result-5.png
440 .. image:: images/result-6.png