1 .. This work is licensed under a
2 .. Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
14 72 Hours Stability Test of Policy API
15 +++++++++++++++++++++++++++++++++++++
20 The 72 hour stability test of policy API has the goal of verifying the stability of running policy design API REST service by
21 ingesting a steady flow of transactions of policy design API calls in a multi-thread fashion to simulate multiple clients' behaviors.
22 All the transaction flows are initiated from a test client server running JMeter for the duration of 72+ hours.
27 The stability test is performed on VMs running in Intel Wind River Lab environment.
28 There are 2 seperate VMs. One for running API while the other running JMeter & other necessary components, e.g. MariaDB, to simulate steady flow of transactions.
29 For simplicity, let's assume:
31 VM1 will be running JMeter, MariaDB.
32 VM2 will be running API REST service and visualVM.
36 Intel ONAP Integration and Deployment Labs
37 `Physical Labs <https://wiki.onap.org/display/DW/Physical+Labs>`_,
38 `Wind River <https://www.windriver.com/>`_
40 **API VM Details (VM2)**
50 Docker Version: 18.09.8
52 Java: OpenJDK 1.8.0_212
54 **JMeter VM Details (VM1)**
64 Docker Version: 18.09.8
66 Java: OpenJDK 1.8.0_212
70 **Software Installation & Configuration**
74 **Install Java & Docker**
76 Make the etc/hosts entries
80 $ echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts
82 Update the Ubuntu software installer
88 Check and install Java
92 $ sudo apt-get install -y openjdk-8-jdk
95 Ensure that the Java version executing is OpenJDK version 8
97 Check and install docker
101 $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
102 $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
103 $ sudo apt-get update
104 $ sudo apt-cache policy docker-ce
105 $ sudo apt-get install -y unzip docker-ce
106 $ systemctl status docker
109 Change the permissions of the Docker socket file
113 $ sudo chmod 777 /var/run/docker.sock
115 Or add the current user to the docker group
119 $ sudo usermod -aG docker $USER
121 Check the status of the Docker service and ensure it is running correctly
125 $ service docker status
132 Download & install JMeter
138 $ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.2.1.zip
139 $ unzip apache-jmeter-5.2.1.zip
141 **Install other necessary components**
143 Pull api code & run setup components script
148 $ git clone https://git.onap.org/policy/api
149 $ cd api/testsuites/stability/src/main/resources/simulatorsetup
150 $ . ./setup_components.sh
152 After installation, make sure the following mariadb container is up and running
156 ubuntu@test:~/api/testsuites/stability/src/main/resources/simulatorsetup$ docker ps
157 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
158 3849ce44b86d mariadb:10.2.14 "docker-entrypoint.s…" 11 days ago Up 11 days 0.0.0.0:3306->3306/tcp mariadb
162 **Install policy-api**
164 Pull api code & run setup api script
169 $ git clone https://git.onap.org/policy/api
170 $ cd api/testsuites/stability/src/main/resources/apisetup
171 $ . ./setup_api.sh <host ip running api> <host ip running mariadb>
173 After installation, make sure the following api container is up and running
177 ubuntu@tools-2:~/api/testsuites/stability/src/main/resources/apisetup$ docker ps
178 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
179 4f08f9972e55 nexus3.onap.org:10001/onap/policy-api:2.1.1-SNAPSHOT "bash ./policy-api.sh" 11 days ago Up 11 days 0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp policy-api
181 **Install & configure visualVM**
183 VisualVM needs to be installed in the virtual machine having API up and running. It will be used to monitor CPU, Memory, GC for API while stability test is running.
189 $ sudo apt-get install visualvm
191 Run few commands to configure permissions
195 $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
196 $ sudo touch visualvm.policy
197 $ sudo chmod 777 visualvm.policy
201 Add the following in visualvm.policy
204 grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
205 permission java.security.AllPermission;
208 Run following commands to start jstatd using port 1111
212 $ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
213 $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &
217 **Run & configure visualVM**
219 Run visualVM by typing
225 Connect to jstatd & remote policy-api JVM
227 1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
228 2. Enter the IP address of VM2 (running policy-api)
229 3. Right click on IP address, select "Add JMX Connection..."
230 4. Enter the VM2 IP Address (from step 2) <IP address>:9090 ( for example, 10.12.6.151:9090) and click OK.
231 5. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.
233 Sample Screenshot of visualVM
235 .. image:: images/results-5.png
246 $ sudo openvpn --config <path to lab ovpn key file>
248 SSH into JMeter VM (VM1)
252 $ ssh -i <path to lab ssh key file> ubuntu@<host ip of JMeter VM>
254 Run JMeter test in background for 72+ hours
259 $ nohup ./jMeter/apache-jmeter-5.2.1/bin/jmeter.sh -n -t ~/api/testsuites/stability/src/main/resources/testplans/policy_api_stability.jmx &
261 (Optional) Monitor JMeter test that is running in background (anytime after re-logging into JMeter VM - VM1)
265 $ tail -f s3p/stability.log nohup.out
270 The 72+ hours stability test will be running the following steps sequentially
271 in multi-threaded loops. Thread number is set to 5 to simulate 5 API clients'
272 behaviors (they can be calling the same policy CRUD API simultaneously).
273 Each thread creates a different version of the policy types and policies to not
274 interfere with one another while operating simultaneously. The point version
275 of each entity is set to the running thread number.
277 **Setup Thread (will be running only once)**
279 - Get policy-api Healthcheck
280 - Get API Counter Statistics
281 - Get Preloaded Policy Types
283 **API Test Flow (5 threads running the same steps in the same loop)**
285 - Create a new Monitoring Policy Type with Version 6.0.#
286 - Create a new Monitoring Policy Type with Version 7.0.#
287 - Create a new Optimization Policy Type with Version 6.0.#
288 - Create a new Guard Policy Type with Version 6.0.#
289 - Create a new Native APEX Policy Type with Version 6.0.#
290 - Create a new Native Drools Policy Type with Version 6.0.#
291 - Create a new Native XACML Policy Type with Version 6.0.#
292 - Get All Policy Types
293 - Get All Versions of the new Monitoring Policy Type
294 - Get Version 6.0.# of the new Monitoring Policy Type
295 - Get Version 6.0.# of the new Optimzation Policy Type
296 - Get Version 6.0.# of the new Guard Policy Type
297 - Get Version 6.0.# of the new Native APEX Policy Type
298 - Get Version 6.0.# of the new Native Drools Policy Type
299 - Get Version 6.0.# of the new Native XACML Policy Type
300 - Get the Latest Version of the New Monitoring Policy Type
301 - Create Monitoring Policy Ver 6.0.# w/Monitoring Policy Type Ver 6.0.#
302 - Create Monitoring Policy Ver 7.0.# w/Monitoring Policy Type Ver 7.0.#
303 - Create Optimization Policy Ver 6.0.# w/Optimization Policy Type Ver 6.0.#
304 - Create Guard Policy Ver 6.0.# w/Guard Policy Type Ver 6.0.#
305 - Create Native APEX Policy Ver 6.0.# w/Native APEX Policy Type Ver 6.0.#
306 - Create Native Drools Policy Ver 6.0.# w/Native Drools Policy Type Ver 6.0.#
307 - Create Native XACML Policy Ver 6.0.# w/Native XACML Policy Type Ver 6.0.#
308 - Get Version 6.0.# of the new Monitoring Policy
309 - Get Version 6.0.# of the new Optimzation Policy
310 - Get Version 6.0.# of the new Guard Policy
311 - Get Version 6.0.# of the new Native APEX Policy
312 - Get Version 6.0.# of the new Native Drools Policy
313 - Get Version 6.0.# of the new Native XACML Policy
314 - Get the Latest Version of the new Monitoring Policy
315 - Delete Version 6.0.# of the new Monitoring Policy
316 - Delete Version 7.0.# of the new Monitoring Policy
317 - Delete Version 6.0.# of the new Optimzation Policy
318 - Delete Version 6.0.# of the new Guard Policy
319 - Delete Version 6.0.# of the new Native APEX Policy
320 - Delete Version 6.0.# of the new Native Drools Policy
321 - Delete Version 6.0.# of the new Native XACML Policy
322 - Delete Monitoring Policy Type with Version 6.0.#
323 - Delete Monitoring Policy Type with Version 7.0.#
324 - Delete Optimization Policy Type with Version 6.0.#
325 - Delete Guard Policy Type with Version 6.0.#
326 - Delete Native APEX Policy Type with Version 6.0.#
327 - Delete Native Drools Policy Type with Version 6.0.#
328 - Delete Native XACML Policy Type with Version 6.0.#
330 **TearDown Thread (will only be running after API Test Flow is completed)**
332 - Get policy-api Healthcheck
333 - Get Preloaded Policy Types
341 Policy API stability test plan was triggered and running for 72+ hours without any error occurred.
345 ======================= ============= =========== =============================== =============================== ===============================
346 **Total # of requests** **Success %** **Error %** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request**
347 ======================= ============= =========== =============================== =============================== ===============================
348 49723 100% 0% 86 ms 4 ms 795 ms
349 ======================= ============= =========== =============================== =============================== ===============================
353 .. image:: images/results-5.png
354 .. image:: images/results-6.png
358 .. image:: images/results-1.png
359 .. image:: images/results-2.png
360 .. image:: images/results-3.png
361 .. image:: images/results-4.png
364 Test Results Frankfurt
365 ----------------------
367 PFPP ONAP Windriver lab
371 Policy API stability test plan was triggered and running for 72+ hours without
372 any real errors occurring. The single failure was on teardown and was due to
373 simultaneous test plans running concurrently on the lab system.
375 Compared to El-Alto, 10x the number of API calls were made in the 72 hour run.
376 However, the latency increased (most likely due to the synchronization added
378 `POLICY-2533 <https://jira.onap.org/browse/POLICY-2533>`_.
379 This will be addressed in the next release.
383 ======================= ============= =========== =============================== =============================== ===============================
384 **Total # of requests** **Success %** **Error %** **Avg. time taken per request** **Min. time taken per request** **Max. time taken per request**
385 ======================= ============= =========== =============================== =============================== ===============================
386 514953 100% 0% 2510 ms 336 ms 15034 ms
387 ======================= ============= =========== =============================== =============================== ===============================
391 VisualVM results were not captured as this was run in the PFPP ONAP Windriver
396 .. image:: images/api-s3p-jm-1_F.png
399 Performance Test of Policy API
400 ++++++++++++++++++++++++++++++
405 Performance test of policy-api has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck.
410 The performance test is performed on OOM-based deployment of ONAP Policy framework components in Intel Wind River Lab environment.
411 In addition, we use another VM with JMeter installed to generate the transactions.
412 The JMeter VM will be sending large number of REST requests to the policy-api component and collecting the statistics.
413 Policy-api component already knows how to communicate with MariaDB component if OOM-based deployment is working correctly.
418 Performance test plan is the same as stability test plan above.
419 Only differences are, in performance test, we increase the number of threads up to 20 (simulating 20 users' behaviors at the same time) whereas reducing the test time down to 1 hour.
424 Running/Triggering performance test will be the same as stability test. That is, launch JMeter pointing to corresponding *.jmx* test plan. The *API_HOST* and *API_PORT* are already set up in *.jmx*.
429 Test results are shown as below. Overall, the test was running smoothly and successfully. We do see some minor failed transactions, especially in POST calls which intend to write into DB simultaneously in a multi-threaded fashion . All GET calls (reading from DB) were succeeded.
431 .. image:: images/summary-1.png
432 .. image:: images/summary-2.png
433 .. image:: images/summary-3.png
434 .. image:: images/result-1.png
435 .. image:: images/result-2.png
436 .. image:: images/result-3.png
437 .. image:: images/result-4.png
438 .. image:: images/result-5.png
439 .. image:: images/result-6.png