5 ONAP Maturity Testing Notes
6 ---------------------------
9 The Release stability has been evaluated by:
11 - The Daily Guilin CI/CD chain
12 - A simple 24h healthcheck verification
13 - A 7 days stability test
16 The scope of these tests remains limited and does not provide a full set of
17 KPIs to determinate the limits and the dimensioning of the ONAP solution.
22 As usual, a daily CI chain dedicated to the release is created after RC0.
23 A Daily Guilin has been created on the 18th of November 2020.
25 Unfortunately several technical issues disturbed the chain:
27 - Due to policy changes in DockerHub (new quotas), the installation chain was
28 not stable as the quota limit was rapidly reached. As a consequence the
29 installation was incomplete and most of the tests were failing. The problem
30 was fixed by the subscription of unlimitted account on DockerHub.
31 - Due to an upgrade of the Git Jenkins plugin done by LF IT, the synchronization
32 of the miror of the xtesting repository, used daily to generate the test suite
33 dockers was corrupted. The dockers were built daily from Jenkins but with an
34 id from the 25th of September. As a consequence the tests reported lots of
35 failure because they were corresponding to Frankfurt tests without the
36 adaptations done for Guilin. The problem was fixed temporarily by moving to
37 GitLab.com Docker registry then by the downgrade of the plugin executed by LF
38 IT during Thanksgiving break.
40 The first week of the Daily Guilin results are therefore not really usable.
41 Most of the results from the `daily Guilin result portal
42 <https://logs.onap.org/onap-integration/daily/onap_daily_pod4_guilin/>`_
43 are not trustable and may be misleading.
44 The results became more stable from the the 6th of December.
46 The graphs given hereafter are based on the data collected until the 8th of
47 december. This Daily chain will be maintained during the Honolulu development
48 cycle (Daily Master) and can be audited at any time. In case of reproducible
49 errors, the integration team will open JIRA on Guilin.
51 Several public Daily Guilin chains have been put in place, one in Orange
52 (Helm v2) and one in DT (Helm v3). DT results are pushed in the test DB and can
54 `ONAP Testing DT lab result page <http://testresults.opnfv.org/onap-integration/dt/dt.html>`_.
56 Infrastructure Healthcheck Tests
57 ................................
59 These tests deal with the Kubernetes/Helm tests on ONAP cluster.
60 The global expected criteria is **50%** when installing with Helm 2.
61 The onap-k8s and onap-k8s-teardown providing a snapshop of the onap namespace in
62 kubernetes are expected to be PASS but two tests are expected to fail:
64 - onap-helm (32/33 OK) due to the size of the SO helm chart (too big for Helm2).
65 - nodeport_check_certs due to bad certificate issuers (Root CA certificate non
66 valid). In theory all the certificate shall be generated during the installation
67 and be valid for the 364 days after the installation. It is still not the case.
68 However, for the first time, no certificate was expired. Next certificates to
72 - Message-router-external (2021-03-25)
74 - AAI and AAI-SPARKY-BE (2021-03-17)
76 .. image:: files/s3p/guilin_daily_infrastructure_healthcheck.png
82 These tests are the traditionnal robot healthcheck tests and additional tests
83 dealing with a single component.
85 The expectation is **100% OK**.
87 .. image:: files/s3p/guilin_daily_healthcheck.png
93 These tests are end to end tests.
94 See the :ref:`the Integration Test page <integration-tests>` for details.
96 The expectation is **100% OK**.
98 .. figure:: files/s3p/guilin_daily_smoke.png
101 An error has been detected on the SDC when performing parallel tests.
102 See `SDC-3366 <https://jira.onap.org/browse/SDC-3366>`_ for details.
107 These tests are tests dealing with security.
108 See the :ref:`the Integration Test page <integration-tests>` for details.
110 The expectation is **66% OK**. The criteria is met.
112 It may even be above as 2 fail tests are almost correct:
114 - the unlimited pod test is still fail due to only one pod: onap-ejbca.
115 - the nonssl tests is FAIL due to so and os-vnfm adapter, which were supposed to
116 be managed with the ingress (not possible for this release) and got a waiver
119 .. figure:: files/s3p/guilin_daily_security.png
122 A simple 24h healthcheck verification
123 =====================================
125 This test consists in running the Healthcheck tests every 10 minutes during
128 The test was run from the 6th of december to the 7th of december.
130 The success rate was 100%.
132 The results are stored in the
133 `test database <http://testresults.opnfv.org/onap/api/v1/results?pod_name=onap_daily_pod4_master-ONAP-oom&case_name=full>`_
135 A 6 days stability test
136 =======================
138 This test consists on running the test basic_vm continuously during 1 week.
140 We observe the cluster metrics as well as the evolution of the test duration.
141 The test basic_vm is describe in :ref:`the Integration Test page <integration-tests>`.
143 Within a long duration test context, the test will onboard a service once then
144 instantiate this service multiple times. Before instantiating, it will
145 systematically contact the SDC and the AAI to verify that the resources already
146 exist. In this context the most impacted component is SO, which was delivered
147 relatively late compared to the other components.
152 The basic_vm test consists in the different following steps:
154 - [SDC] VendorOnboardStep: Onboard vendor in SDC.
155 - [SDC] YamlTemplateVspOnboardStep: Onboard vsp described in YAML file in SDC.
156 - [SDC] YamlTemplateVfOnboardStep: Onboard vf described in YAML file in SDC.
157 - [SDC] YamlTemplateServiceOnboardStep: Onboard service described in YAML file
159 - [AAI] RegisterCloudRegionStep: Register cloud region.
160 - [AAI] ComplexCreateStep: Create complex.
161 - [AAI] LinkCloudRegionToComplexStep: Connect cloud region with complex.
162 - [AAI] CustomerCreateStep: Create customer.
163 - [AAI] CustomerServiceSubscriptionCreateStep: Create customer's service
165 - [AAI] ConnectServiceSubToCloudRegionStep: Connect service subscription with
167 - [SO] YamlTemplateServiceAlaCarteInstantiateStep: Instantiate service described
168 in YAML using SO a'la carte method.
169 - [SO] YamlTemplateVnfAlaCarteInstantiateStep: Instantiate vnf described in YAML
170 using SO a'la carte method.
171 - [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module
172 described in YAML using SO a'la carte method.
174 The test has been initiated on a weekly lab on the 2nd of december.
175 The results provided hereafter correspond to the period from 2020-12-02 to
178 .. csv-table:: Basic_vm results
179 :file: ./files/csv/stability_basic_vm.csv
186 The corrected success rate excludes the FAIL results obtained during the SDNC
188 The cause of the errors shall be analyzed more in details. The huge majority of
189 errors (79%) occurs on SO service creation, 18% on VNF creation and 3% on
193 The test success rate is about 86%.
194 CPU consumption is low (see next section).
195 Memory consumption is high.
197 After ~ 24-48h, the test is systematically FAIL. The trace shows that the SDNC
198 is no more responding. This error required the manual restart of the SDNC.
199 It seems that the SDNC exceeds its limits set in OOM. The simple manual
200 restart (delete of the pod was enough, the test after the restart is PASS,
201 and keep most of the time PASS for the next 24-48h)
203 We can observe the consequences of the manual restart of the SDNC on its memory
204 graph as well as the memory threshold.
206 .. figure:: files/s3p/stability_sdnc_memory.png
209 The duration of the test is increasing slowly over the week and can be described
212 .. figure:: files/s3p/basic_vm_duration.png
215 If we consider the histogram, we can see the distribution of the duration.
217 .. figure:: files/s3p/basic_vm_duration_histo.png
220 As a conclusion, the solution seems stable.
222 The memory issue detected in the SDNC may be due to a bad sizing of the limits
223 and requests in OOM but a problem of light memory leak cannot be exclude.
224 The workaround consisting in restarting of the SDNC seems to fix the issue.
225 The issue is tracked in `SDNC-1430 <https://jira.onap.org/browse/SDNC-1430>`_.
226 Further study shall be done on this topic to consildate the detection of the
232 The Metrics of the ONAP cluster on this 6 days period are given by the
236 :file: ./files/csv/stability_cluster_metric_cpu.csv
237 :widths: 20,10,10,10,10,10,10,10
241 .. csv-table:: Memory
242 :file: ./files/csv/stability_cluster_metric_memory.csv
243 :widths: 20,10,10,10,10,10,10,10
247 .. csv-table:: Network
248 :file: ./files/csv/stability_cluster_metric_network.csv
249 :widths: 10,15,15,15,15,15,15
253 The Top Ten for Memory consumption is given in the table below:
255 .. csv-table:: Memory
256 :file: ./files/csv/stability_top10_memory.csv
257 :widths: 20,15,15,20,15,15
261 At least 9 components exceeds their Memory Requests. And 7 are over the Memory
262 limits set in OOM: the 2 Opendaylight controllers and the cassandra Databases.
264 As indicated CPU consumption is negligeable and not dimensioning.
265 It shall be reconsider for use cases including extensive computation (loops,
266 optimization algorithms).