cps.git
4 weeks agoMake Groovy Performance Test Stable #7 67/140967/1
ToineSiebelink [Wed, 28 May 2025 08:00:19 +0000 (09:00 +0100)]
Make Groovy Performance Test Stable #7

- Last tweaks (I hope)
- Changed assert of 1 (ungraphed) test to use resource meter so it can be graphed in future

Issue-ID: CPS-2743
Change-Id: Ie4d480331964cd69d3e119fbd4bb9fdeb1499631
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
4 weeks agoMake Groovy Performance Test Stable #6 65/140965/1
ToineSiebelink [Tue, 27 May 2025 15:38:39 +0000 (16:38 +0100)]
Make Groovy Performance Test Stable #6

- Fixed many expectations again now Groovy Perf does not run at same time as KPI
  (strangely enough most create use cases got worse)
- Default margin now 100% instead of 50%
- Increased volume of 1 very fast test to get more stable measurements

Issue-ID:CPS-2743

Change-Id: I081d48926a9e283a429603bd78dcd79c5d8a2394
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
4 weeks agoMerge "Part 1: Grouping of Data Nodes in Delta Report"
Toine Siebelink [Tue, 27 May 2025 12:23:09 +0000 (12:23 +0000)]
Merge "Part 1: Grouping of Data Nodes in Delta Report"

4 weeks agoPart 1: Grouping of Data Nodes in Delta Report 50/139850/39
Arpit Singh [Mon, 13 Jan 2025 13:23:30 +0000 (18:53 +0530)]
Part 1: Grouping of Data Nodes in Delta Report

Add-on feature in delta report to generate condensed delta report by
grouping data nodes based on parent child relationship.

The patch adds grouping feature for "create" operation in delta report.

- Added boolean flag "groupingEnabled" to enable or disable grouping of
  data nodes in delta report. Default value is false.
- Added a method getCondensedAddedDeltaReports to generate condensed
  delta reports for create operation.
- Part 2 of this patch will have code for updated and removed delta
  report entries when grouping is enabled.
- A separate patch to add integration tests will be done after feature
  is implemented. This is done to keep patch sizes small

Issue-ID: CPS-2547
Change-Id: Ibb4d35b03098be7b57cb59852a87f6b4e0c7b706
Signed-off-by: Arpit Singh <AS00745003@techmahindra.com>
4 weeks agoMerge "Make Groovy Performance Test Stable"
Sourabh Sourabh [Fri, 23 May 2025 10:35:39 +0000 (10:35 +0000)]
Merge "Make Groovy Performance Test Stable"

4 weeks agoFix maven merge master job 54/140954/1
mpriyank [Fri, 23 May 2025 09:56:21 +0000 (10:56 +0100)]
Fix maven merge master job

- distribution management section was incorrectly removed before,
  results in failure of maven merge master job

Issue-ID: CPS-2825
Change-Id: I5cf3e2b3344669b4f2dbb7c37a87c0903589a865
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
5 weeks agoMake Groovy Performance Test Stable 52/140952/1
ToineSiebelink [Thu, 22 May 2025 16:49:15 +0000 (17:49 +0100)]
Make Groovy Performance Test Stable

- use double-type on groovy method to prevent accidental conversion issues

Issue-ID:CPS-2743

Change-Id: I32e31df8d7cbc4e0268aff8a0fc052b030173aa4
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
5 weeks agoMake Groovy Performance Test Stable 31/140931/1
ToineSiebelink [Thu, 22 May 2025 11:58:13 +0000 (12:58 +0100)]
Make Groovy Performance Test Stable

- Increased threshold on some graphs to prevent failure when running parallel with endurance

Issue-ID: CPS-2743
Change-Id: Ic96dd1a29ca3046668b18195399be37786fb965a
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
5 weeks agoAdded JettyConfig with support for ambiguous path separators 26/140926/4
sourabh_sourabh [Wed, 21 May 2025 11:54:15 +0000 (12:54 +0100)]
Added JettyConfig with support for ambiguous path separators

- Added JettyConfig to customize Jetty's HttpConfiguration
- For ambiguous path separators via UriCompliance settings.

Issue-ID: CPS-2819
Change-Id: I30cbb14cf31ac808a9a9761cf8dbf9ba0df870ae
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
5 weeks agoMerge "Make Groovy Performance Test Stable"
Sourabh Sourabh [Wed, 21 May 2025 13:56:57 +0000 (13:56 +0000)]
Merge "Make Groovy Performance Test Stable"

5 weeks agoMake Groovy Performance Test Stable 23/140923/4
ToineSiebelink [Tue, 20 May 2025 15:43:49 +0000 (16:43 +0100)]
Make Groovy Performance Test Stable

- Removed average calculation where this lead to very small duration and therefore seemingly more variation
- Tweak a few expected duration to get closer to actual average
- Remove memory assertion from resource meter test since they are unreliable and cause the odd error
- Corrected wording on description related to limits

Issue-ID:CPS-2743
Change-Id: Id30bdbe7074297c32a236322fc66ee698fdd8e2b
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
5 weeks agoMerge openapi.yml and dmi-components.yml into one file 21/140921/2
egernug [Tue, 20 May 2025 12:33:51 +0000 (13:33 +0100)]
Merge openapi.yml and dmi-components.yml into one file

To aid downloading of Openapi spec for downloading in DMI-Plugin the spec and components file are bing combined.
This allows for the use of the existing openapi-generator-maven-pluginto be used to grab the file from CPS repo.

Issue-ID: CPS-2820

Change-Id: I08b4ae15fffb195e8bdd7a243de557f4b4fe9d60
Signed-off-by: egernug <gerard.nugent@est.tech>
5 weeks agoMake Groovy Performance Test Stable 16/140916/2
ToineSiebelink [Tue, 20 May 2025 08:50:41 +0000 (09:50 +0100)]
Make Groovy Performance Test Stable

- Tweak/correct more graphs
- Fix functional issue in test for alt-id lookup

Issue-ID: CPS-2743
Change-Id: I6f7a8d7cdac5b32a0a988fdac88c33b36f621499
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
5 weeks agoMerge "Fix broken docker-compose link"
Toine Siebelink [Tue, 20 May 2025 10:58:43 +0000 (10:58 +0000)]
Merge "Fix broken docker-compose link"

5 weeks agoFix broken docker-compose link 18/140918/1
halil.cakal [Tue, 20 May 2025 10:40:32 +0000 (11:40 +0100)]
Fix broken docker-compose link

- There is still a link that point the old docker-compose file that
  should be updated in deployment.rst

Issue-ID: CPS-2730

Change-Id: I3c328435531f6e14b84f2f2cf2b8cd9934ae46b4
Signed-off-by: halil.cakal <halil.cakal@est.tech>
5 weeks agoMerge "Remove -quiet option on k6 run"
Toine Siebelink [Tue, 20 May 2025 10:13:45 +0000 (10:13 +0000)]
Merge "Remove -quiet option on k6 run"

5 weeks agoRemove -quiet option on k6 run 17/140917/1
emaclee [Mon, 19 May 2025 10:30:23 +0000 (11:30 +0100)]
Remove -quiet option on k6 run

     - Removing -quiet option to unsuppressed the progress logs by k6

Issue-ID: CPS-2794
Change-Id: I5bd81b68cef41522433faba714f1427d5df05cbd
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
5 weeks agoMerge "Make Groovy Performance Test Stable"
Lee Anjella Macabuhay [Mon, 19 May 2025 13:49:22 +0000 (13:49 +0000)]
Merge "Make Groovy Performance Test Stable"

5 weeks agoMake Groovy Performance Test Stable 89/140889/2
ToineSiebelink [Thu, 15 May 2025 12:08:15 +0000 (13:08 +0100)]
Make Groovy Performance Test Stable

- Disable memory assertion
- Allow default 50% margin on expected time
- Allow 300% margin on very fast test < 1ms
- Extend resource meter method with optional exceptional margin for an unstable test
- Update all time limits on plotted test to be average (rather then limit)

Issue-ID: CPS-2743
Change-Id: Icdcbe01b048c2d04166684cda4bf341a204c9a1d
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
5 weeks agoSplit big docker-compose into smaller ones 70/140870/3
halil.cakal [Fri, 9 May 2025 10:13:07 +0000 (11:13 +0100)]
Split big docker-compose into smaller ones

- split the big docker-compose and remove
- two new docker-compose: cps-base and dmi-services
- remove monitoring profile from: kafka-ui, prometheus, and grafana
  since they are not an overhead to performance
- kept 'tracing' profile for 'jaeger-service' because it reduced the
  performance by around 4 times (see the jira for more details)
- documentation updated

Issue-ID: CPS-2730

Change-Id: I4d9cdd82c96c2864c8eb7415b6857a4d2df898fc
Signed-off-by: halil.cakal <halil.cakal@est.tech>
5 weeks agoMerge "Fix component field in Datajobs API"
Sourabh Sourabh [Fri, 16 May 2025 08:55:45 +0000 (08:55 +0000)]
Merge "Fix component field in Datajobs API"

5 weeks agoMerge "Use an environment variable for the on-demand KPI pipeline"
Sourabh Sourabh [Fri, 16 May 2025 08:53:10 +0000 (08:53 +0000)]
Merge "Use an environment variable for the on-demand KPI pipeline"

5 weeks agoFix component field in Datajobs API 94/140894/2
egernug [Fri, 16 May 2025 08:23:25 +0000 (09:23 +0100)]
Fix component field in Datajobs API

When using an external component file it can be named anything but
an internal file must use components as the top level key

Issue-ID: CPS-2811

Change-Id: Id2e5ae24c9a78afe81b92fc972fcc2ffe38fab74
Signed-off-by: egernug <gerard.nugent@est.tech>
6 weeks agoUse an environment variable for the on-demand KPI pipeline 90/140890/2
halil.cakal [Thu, 15 May 2025 15:28:42 +0000 (16:28 +0100)]
Use an environment variable for the on-demand KPI pipeline

- introduce an environment variable that can be set (export) from
  the onap-cps-kpi-on-demand job (jenkins)

Issue-ID: CPS-2694

Change-Id: I33a2e583e06c4b5cbafcdcf0d05e8cb89a573c8e
Signed-off-by: halil.cakal <halil.cakal@est.tech>
6 weeks agoMerge changes Ie80fdd4c,I1663d8aa
Sourabh Sourabh [Thu, 15 May 2025 15:14:08 +0000 (15:14 +0000)]
Merge changes Ie80fdd4c,I1663d8aa

* changes:
  DCM Write optimization using batching
  Use Alternate Id to CM-handle ID map for DCM write datajobs

6 weeks agoMerge "Update KPI Test load for 2 tests"
Lee Anjella Macabuhay [Thu, 15 May 2025 10:27:53 +0000 (10:27 +0000)]
Merge "Update KPI Test load for 2 tests"

6 weeks agoDCM Write optimization using batching 85/140885/4
ToineSiebelink [Thu, 8 May 2025 16:24:39 +0000 (17:24 +0100)]
DCM Write optimization using batching

- Add method to AlternateIdMatcher to match a batch of paths and return the relevant yang model cm handles
- Extended inventory service to load cm handles without properties (i.e. descendants)
- Switch from NcmpServiceCmHandles to YangModel Cm Handles
- Preload required cm handles (without properties) using batching
- Removed (now) unused methods
- Improved test coverage on some legacy but related functionality

Issue-ID: CPS-2743
Change-Id: Ie80fdd4c12b72fc72ab1a87aa463ec0e6b664e3a
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
6 weeks agoMerge "Enhance the Multipart Delta API to support JSON file"
Priyank Maheshwari [Thu, 15 May 2025 09:19:48 +0000 (09:19 +0000)]
Merge "Enhance the Multipart Delta API to support JSON file"

6 weeks agoMerge "Move DMI-Plugin and DataJob API into CPS repo"
Toine Siebelink [Thu, 15 May 2025 09:15:55 +0000 (09:15 +0000)]
Merge "Move DMI-Plugin and DataJob API into CPS repo"

6 weeks agoUpdate KPI Test load for 2 tests 80/140880/2
ToineSiebelink [Tue, 13 May 2025 14:27:34 +0000 (15:27 +0100)]
Update KPI Test load for 2 tests

- TODO Remove KPI test tweaks (for local testing)
- Removed storm load from CM AVC background load
- Changed Legacy Batch test to fixed rate of 200 request events per second
- Changed Legacy Batch test to fixed time of 15 min (=180,000 events)

Issue-ID: CPS-2743
Change-Id: Id1c795a56797637efc7cbd74b5c2bbb688648ac9
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
6 weeks agoUse Alternate Id to CM-handle ID map for DCM write datajobs 27/140827/2
emaclee [Mon, 28 Apr 2025 10:13:15 +0000 (11:13 +0100)]
Use Alternate Id to CM-handle ID map for DCM write datajobs

Issue-Id: CPS-2743
Change-Id: I1663d8aab4a3d4cee898a3171dad926643ccbe79
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
6 weeks agoEnhance the Multipart Delta API to support JSON file 68/140268/20
Rudrangi Anupriya [Wed, 9 Apr 2025 09:55:17 +0000 (15:25 +0530)]
Enhance the Multipart Delta API to support JSON file

Here we upload Json file as payloads instead of  JSON data as text input

Issue-ID: CPS-2657
Change-Id: Ibc9bbb99aabb07302366321c6d5c2eade0e7e5fb
Signed-off-by: Rudrangi Anupriya <ra00745022@techmahindra.com>
6 weeks agoMerge "Increase startup delay from 1 sec to 2 sec"
Lee Anjella Macabuhay [Wed, 14 May 2025 08:55:08 +0000 (08:55 +0000)]
Merge "Increase startup delay from 1 sec to 2 sec"

6 weeks agoIncrease startup delay from 1 sec to 2 sec 72/140872/2
halil.cakal [Tue, 13 May 2025 10:19:54 +0000 (11:19 +0100)]
Increase startup delay from 1 sec to 2 sec

- if cps-ncmp-1 got a head start before cps-ncmp-0, and spin-off faster
  than cps-ncmp-0 the current delay '1 sec' is not enough to prevent race
condition between liquibase instances

Issue-ID: CPS-2807

Change-Id: I9ed5780daea4b95e6a2026e46427c7f4e464818e
Signed-off-by: halil.cakal <halil.cakal@est.tech>
6 weeks agoTest Cm Event Storm 74/140874/2
ToineSiebelink [Tue, 13 May 2025 11:41:50 +0000 (12:41 +0100)]
Test Cm Event Storm

- 6 minutes CM Event Background load to 26*500 = 13000 AVC/sec
- 8 minutes CM Event Background load to 6*500 = 3000 AVC/sec

Issue-ID: CPS-2743
Change-Id: Ide4201609e5d9766b3d3ab7c855dfd11159dafd0
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
6 weeks agoMove DMI-Plugin and DataJob API into CPS repo 73/140873/1
egernug [Tue, 13 May 2025 11:10:54 +0000 (12:10 +0100)]
Move DMI-Plugin and DataJob API into CPS repo

Issue-ID: CPS-2811

Change-Id: Idc5931ab81c7d6b836e589a024cb7df153bd48c8
Signed-off-by: egernug <gerard.nugent@est.tech>
6 weeks agoMerge "Ensure teardown script removes all onap images"
Sourabh Sourabh [Tue, 13 May 2025 08:58:02 +0000 (08:58 +0000)]
Merge "Ensure teardown script removes all onap images"

6 weeks agoIncrease Kafka rate in KPI 63/140863/1
egernug [Fri, 9 May 2025 10:47:36 +0000 (11:47 +0100)]
Increase Kafka rate in KPI

Issue-ID: CPS-2805

Change-Id: I7c0feb72d844bfd251c5dea9d3d22c3c35645884
Signed-off-by: egernug <gerard.nugent@est.tech>
7 weeks agoMerge "Adjusted NCMP architecture rule"
Toine Siebelink [Thu, 8 May 2025 13:06:41 +0000 (13:06 +0000)]
Merge "Adjusted NCMP architecture rule"

7 weeks agoAdjusted NCMP architecture rule 47/140847/3
leventecsanyi [Wed, 7 May 2025 10:20:04 +0000 (12:20 +0200)]
Adjusted NCMP architecture rule

  - removed spi package from ncmp service test

Issue-ID: CPS-2767
Change-Id: I482685ef4f8760a3cff3f3a6dfca12298b467209
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
7 weeks agoEnsure teardown script removes all onap images 50/140850/1
emaclee [Thu, 8 May 2025 07:29:30 +0000 (08:29 +0100)]
Ensure teardown script removes all onap images

- ensure that script removes test profile containers, volumes
  and images
- add method to remove all onap images when
  testProfile=endurance
- add method to verify that no onap images is left dangling when
  endurance test profile is finished

Issue-ID: CPS-2802
Change-Id: Ib319f8595555d315c4ec97795827d99d22fb29af
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
7 weeks agoMerge "Revert "Add CM Write Data Job Scenario to K6 Test Suite""
Toine Siebelink [Wed, 7 May 2025 17:49:40 +0000 (17:49 +0000)]
Merge "Revert "Add CM Write Data Job Scenario to K6 Test Suite""

7 weeks agoRevert "Add CM Write Data Job Scenario to K6 Test Suite" 97/140797/1
Sourabh Sourabh [Wed, 7 May 2025 16:57:30 +0000 (16:57 +0000)]
Revert "Add CM Write Data Job Scenario to K6 Test Suite"

This reverts commit affb89aa97cc436d873065a5daec2fc3eb103f8f.

Reason for revert: Performance

Change-Id: Ic3a441f8d83ba2c347a23026348db09347956f85
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
7 weeks agoMerge "Extend CpsPathQuerySpec for dangerous SQLi characters"
Toine Siebelink [Wed, 7 May 2025 12:42:28 +0000 (12:42 +0000)]
Merge "Extend CpsPathQuerySpec for dangerous SQLi characters"

7 weeks agoMerge "Fix the tests that failing irregularly in groovy pipeline"
Toine Siebelink [Wed, 7 May 2025 12:41:19 +0000 (12:41 +0000)]
Merge "Fix the tests that failing irregularly in groovy pipeline"

7 weeks agoAdd CM Write Data Job Scenario to K6 Test Suite 67/140667/10
sourabh_sourabh [Tue, 8 Apr 2025 17:08:09 +0000 (18:08 +0100)]
Add CM Write Data Job Scenario to K6 Test Suite

- Added 'write_data_job_scenario' in the K6 test configuration for small
  and large data size.
- It helps to simulate lower / higher load and monitor system behavior under infrequent
  write operations.

Issue-ID: CPS-2716
Change-Id: Ic5a5f3642ff57b294c822541dcc4b1458301ccac
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
7 weeks agoFix the tests that failing irregularly in groovy pipeline 43/140843/2
halil.cakal [Tue, 6 May 2025 12:40:46 +0000 (13:40 +0100)]
Fix the tests that failing irregularly in groovy pipeline

- Don't check if the key is present in the map anymore instead keep
  looking for if the key deleted after TTL period
- Optimize the test by reducing the max wait time from 10 seconds to 1
  second

Issue-ID: CPS-2798

Change-Id: Ibbd40e682c28fa0c52da3f07d3687927fa42aa02
Signed-off-by: halil.cakal <halil.cakal@est.tech>
7 weeks agoInconsistency With JSON Response(List Items) Using GetANode API 45/140045/31
Rudrangi Anupriya [Mon, 5 May 2025 14:34:39 +0000 (20:04 +0530)]
Inconsistency With JSON Response(List Items) Using GetANode API

Jira - https://lf-onap.atlassian.net/browse/CPS-2566
Documentation -https://lf-onap.atlassian.net/wiki/x/OQA1AQ

- Introduced versioned V3 approach to retrieve list elements
- added controller method getNodeByDataspaceAndAnchorV3
- Updated listelementAsMap,containerElementsAsMap to group list elements

Issue-ID: CPS-2566
Change-Id: Ice9ffb99aabb03702355321c6d5c2eade0e7ef5b
Signed-off-by: Rudrangi Anupriya <ra00745022@techmahindra.com>
7 weeks agoAdd non blocking comments from previous commit 40/140840/1
emaclee [Fri, 2 May 2025 12:23:37 +0000 (13:23 +0100)]
Add non blocking comments from previous commit

- Adding non blocking comments changes from 140838: Increase CM Notification Event KPI Background Load | https://gerrit.onap.org/r/c/cps/+/140838

Issue-ID: CPS-2752
Change-Id: I14c28466f69d9ff83692075f4283b7631e947fd3
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
8 weeks agoIncrease CM Notification Event KPI Background Load 38/140838/1
ToineSiebelink [Tue, 29 Apr 2025 11:13:26 +0000 (12:13 +0100)]
Increase CM Notification Event KPI Background Load

- Update KPI script to send batches of messages
- Increased KPI load from 500 to 2,500 events/sec using 10 VUs and batchsize of 250
- Added instrumentation on cm avc forwarding method
- verified using Grafana (instrumentation) and kafka ui to confirm correct load of messages consumed and forwarded

Issue-ID: CPS-2752
Change-Id: I90951bf7844e087b908082c2609d0f3a564d50f9
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
8 weeks agoExtend CpsPathQuerySpec for dangerous SQLi characters 35/140835/2
egernug [Wed, 30 Apr 2025 09:16:40 +0000 (10:16 +0100)]
Extend CpsPathQuerySpec for dangerous SQLi characters

Issue-ID: CPS-2781

Change-Id: Ic658028d0ed48c3345db218ab913732d7665e2c6
Signed-off-by: egernug <gerard.nugent@est.tech>
8 weeks agoControl start-up delay: set host-names explicitly in docker compose 26/140826/7
ToineSiebelink [Mon, 28 Apr 2025 08:15:06 +0000 (09:15 +0100)]
Control start-up delay: set host-names explicitly in docker compose

- Use sequenced hostnames for cps-ncmp  in docker compose based on a common template
- Use explicit port definitions for each instance instead of range (instrumentation/Grafana)
- Calculated startup delay as sequence number times 1 second when correct pattern used
- Fall back on hash code modulus 10 seconds otherwise
- clear logging which algorithm is used, why and outcome
- update kpi and endurance environment files to use correct instance names

Issue-ID:CPS-2752

Change-Id: Ie788aff60d6e9c470136b2a2051d0c5ccc173e9a
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
8 weeks agoMerge "Increase the setup timeout in k6"
Sourabh Sourabh [Thu, 1 May 2025 09:55:36 +0000 (09:55 +0000)]
Merge "Increase the setup timeout in k6"

8 weeks agoMerge "Adjust KPI threshold and rate as per latest KPI agreement"
Priyank Maheshwari [Thu, 1 May 2025 09:47:46 +0000 (09:47 +0000)]
Merge "Adjust KPI threshold and rate as per latest KPI agreement"

8 weeks agoIncrease the setup timeout in k6 33/140833/2
halil.cakal [Thu, 1 May 2025 08:30:46 +0000 (09:30 +0100)]
Increase the setup timeout in k6

- Since the new server for on demand KPI jobs is slower than our regular
  server and fails because of a limit on the set up time this patch is
to increase that

Issue-ID: CPS-2694

Change-Id: I406faadec474f3147eaedee67ba1137a82207ccb
Signed-off-by: halil.cakal <halil.cakal@est.tech>
8 weeks agoAdjust KPI threshold and rate as per latest KPI agreement 34/140834/1
ToineSiebelink [Wed, 30 Apr 2025 13:25:29 +0000 (14:25 +0100)]
Adjust KPI threshold and rate as per latest KPI agreement

- CM Handle ID searches now can take 2.6 sec
- CM Handle searches now can take 24 sec
- Updated 'blue' lines to reflect KPI
- Updated 'green' lines to reflect current average (i.e. we expect measurements to be 'around' this)
- Experimental: increased cm notification forwarding by using more VUs
- Add timer metric to be able to check cm notification forwarding rate
- Temporary: removed redirecting of warning and errors to null device

Issue-ID: CPS-2793
Change-Id: I560985d4f9ce7a4c8aa27b3ae089a43a8a76b28a
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
8 weeks agoMerge "Fix legacy batch response with not found CM-handle"
Priyank Maheshwari [Wed, 30 Apr 2025 14:40:52 +0000 (14:40 +0000)]
Merge "Fix legacy batch response with not found CM-handle"

2 months agoFix legacy batch response with not found CM-handle 54/140654/3
danielhanrahan [Thu, 17 Apr 2025 17:21:14 +0000 (18:21 +0100)]
Fix legacy batch response with not found CM-handle

An unhandled exception leads to the entire batch operation failing,
even if only a single CM handle in the batch was not found. This
fixes the issue by swallowing the exception. Note there is existing
logic in DmiDataOperationsHelper for reporting not-found CM handles.

Issue-ID: CPS-2769
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: Ifff2c4bd291107480ca7462d721a02f8ea138e58

2 months agoRemove Basic Authentication References 08/140708/16
ToineSiebelink [Wed, 16 Apr 2025 12:47:09 +0000 (13:47 +0100)]
Remove Basic Authentication References

- Remove whole ‘security’ section from application.yml file(s) also in testware
- Remove whole ‘security’ section from  open api yml file(s) except Policy Executor
- Remove whole ‘component.securitySchemes’ section from  open api yml files
- Remove references to security.auth.username from xml config files
- Removed cps authentication details from Docker compose and readme files (tested)
- Removed authentication groovy test rom cps-application
- Side note: cps-application added test, coverage increased to 100%
- Updated Docker-Compose (CSIT & K6) to latest version
- Minor csit script updates to help troubleshooting
- Removed auth header from ALL csit tests files

Issue-ID: CPS-2600
Change-Id: Ie1cf02461943f4e43029a3dbfaef052e347a4d4d
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
2 months agoMerge "Automate the CPS-NCMP metrics scraping"
Toine Siebelink [Thu, 24 Apr 2025 16:42:12 +0000 (16:42 +0000)]
Merge "Automate the CPS-NCMP metrics scraping"

2 months agoAutomate the CPS-NCMP metrics scraping 87/140787/6
halil.cakal [Thu, 24 Apr 2025 08:47:57 +0000 (09:47 +0100)]
Automate the CPS-NCMP metrics scraping

- import ScrapeMetrics, test_ScrapeMetrics and run from conf .py to
  automate the scraping
- the scripts will be running with tox when building the docs
- update unit tests to use test specific names
- highlight the header row of metrics' table

Issue-ID: CPS-2778

Change-Id: I843221ec21454220ea1182c488f88813c0691c4a
Signed-off-by: halil.cakal <halil.cakal@est.tech>
2 months agoMerge "Add integration tests of legacy batch data operation"
Toine Siebelink [Thu, 24 Apr 2025 11:48:51 +0000 (11:48 +0000)]
Merge "Add integration tests of legacy batch data operation"

2 months agoMerge "Scrape all metrics and add them to cps documentation"
Toine Siebelink [Wed, 23 Apr 2025 15:54:47 +0000 (15:54 +0000)]
Merge "Scrape all metrics and add them to cps documentation"

2 months agoScrape all metrics and add them to cps documentation 00/140700/8
halil.cakal [Thu, 10 Apr 2025 13:02:54 +0000 (14:02 +0100)]
Scrape all metrics and add them to cps documentation

- add a new dummy annotation (TimedCustom) for Gauge metrics (cm handle state)
- add a description field for the counter annotation (CountCmHandleSearchExecution)
- add a python script (with tests) to scrape the metrics mentioned above
  using a single regex for all
- remove redundancies (distribution management) from pom.xml of
  checkstyle module since there is no artifact released from checkstyle
- import and display the output (metrics.cvs) in admin-guide docs

Issue-ID: CPS-2709

Change-Id: Iaee23f0a20c05e5aea033baacad1f23cb61e8b34
Signed-off-by: halil.cakal <halil.cakal@est.tech>
2 months agoAdd integration tests of legacy batch data operation 10/140710/3
danielhanrahan [Wed, 16 Apr 2025 11:20:58 +0000 (12:20 +0100)]
Add integration tests of legacy batch data operation

Add integration tests for:
1. happy path: batch request forwarded to DMI
2. negative case: not ready CM-handles report error
3. negative case: not existing CM-handles report error

Case 3 is Ignored as there is currently a bug which causes the
request to fail, instead of reporting error on kafka topic.

Issue-ID: CPS-2769
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: Ic31bf856c09dd63ba80019fc251a437458184bba

2 months agoSimplify hostname-based startup delay logic with consistent hash-based approach 11/140711/1
sourabh_sourabh [Thu, 17 Apr 2025 16:38:54 +0000 (17:38 +0100)]
Simplify hostname-based startup delay logic with consistent hash-based approach

- Removed conditional logic based on last character of the hostname

- Applied a uniform hash-based delay calculation up to 5000 ms

- Improves consistency and simplifies the code logic

- Updated JavaDoc to reflect the new approach

Issue-ID: CPS-2752
Change-Id: Ic9e5cffb7396695e90b94edba79984f7f0f859aa
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2 months agoBump CPS to 3.6.3-SNAPSHOT 07/140707/1
mpriyank [Wed, 16 Apr 2025 12:16:25 +0000 (13:16 +0100)]
Bump CPS to 3.6.3-SNAPSHOT

- bumped CPS-NCMP to 3.6.3-SNAPSHOT version

Issue-ID: CPS-2770
Change-Id: I7221363b048abacdf2ef9011efb5b98caeaa043c
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2 months agoMerge "Container yaml for release"
Lee Anjella Macabuhay [Wed, 16 Apr 2025 12:09:36 +0000 (12:09 +0000)]
Merge "Container yaml for release"

2 months agoContainer yaml for release 06/140706/1
mpriyank [Wed, 16 Apr 2025 11:35:23 +0000 (12:35 +0100)]
Container yaml for release

Issue-ID: CPS-2770
Change-Id: If0f852fcdf32b658343de1d682019d6edb27f2b7
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2 months agoMaven stage yaml 05/140705/1
mpriyank [Wed, 16 Apr 2025 11:23:56 +0000 (12:23 +0100)]
Maven stage yaml

- maven stage yaml for 3.6.2 release

Issue-ID: CPS-2770
Change-Id: Ia143acf696ddc7e8c31d2b1ea8e8772f1452a3e7
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2 months agoRelease note to uplift spring boot 04/140704/2
mpriyank [Wed, 16 Apr 2025 10:32:02 +0000 (11:32 +0100)]
Release note to uplift spring boot

Issue-ID: CPS-2770
Change-Id: Icc66cde5de51e1f5d74d52556cd3068e07600f2e
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2 months agoUpdate info and OAS for the release 03/140703/1
mpriyank [Wed, 16 Apr 2025 09:58:13 +0000 (10:58 +0100)]
Update info and OAS for the release

- info version set to 3.6.2 as it is the version being released

Issue-ID: CPS-2770
Change-Id: Idd78aded458710b39fcaaab24343fd93aee3f658
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>
2 months agoReview RTD Configuration Properties 02/140702/2
ToineSiebelink [Tue, 15 Apr 2025 15:17:14 +0000 (16:17 +0100)]
Review RTD Configuration Properties

- split into 3 tables
- introduced CSV-file-defined tables for easier maintenance
- removed unnecessary 3PP properties definitions
- checked all custom properties still used and default values
- corrected some default values (sometime just wrong casing)
- updated application yaml spring.datasource.url so it has a default host value
- removed code-formatting for property values (except for URLS to prevent failing link-checks)
- added notes about using environment variables where applicable

Issue-ID: CPS-2596
Change-Id: I6f413dd726088ca8a002da145bf4f6c52a1c7b2a
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
2 months agoMerge "Use a Hazelcast map of alternate ids to Cm handle ids"
Daniel Hanrahan [Tue, 15 Apr 2025 10:44:56 +0000 (10:44 +0000)]
Merge "Use a Hazelcast map of alternate ids to Cm handle ids"

2 months agoUse a Hazelcast map of alternate ids to Cm handle ids 64/140664/9
emaclee [Tue, 8 Apr 2025 11:48:48 +0000 (12:48 +0100)]
Use a Hazelcast map of alternate ids to Cm handle ids

Issue-ID: CPS-2753
Change-Id: I6f3657bf14d78e412f650ee536e79beefb601c09
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
2 months agoAdd HTTP 409 Conflict error response to NCMP OpenAPI Doc 91/140691/2
ToineSiebelink [Mon, 14 Apr 2025 12:50:50 +0000 (13:50 +0100)]
Add HTTP 409 Conflict error response to NCMP OpenAPI Doc

- This error can occur when Policy Execution denies write operation but was never documented

Issue-ID: CPS-2764
Change-Id: Idc47371b63b1348f86384e1ff3a8ae0d50056cdf
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
2 months agoMerge "Refactoring to combine LcmEventsCreator and LcmEventsCreatorHelper"
Priyank Maheshwari [Fri, 11 Apr 2025 13:08:58 +0000 (13:08 +0000)]
Merge "Refactoring to combine LcmEventsCreator and LcmEventsCreatorHelper"

2 months agoIntroduced a startup delay mechanism based on the container hostname 59/140659/8
sourabh_sourabh [Tue, 8 Apr 2025 10:13:08 +0000 (11:13 +0100)]
Introduced a startup delay mechanism based on the container hostname

- Extracted hostname using InetAddress and Calculated delay or fallback hash-based delay.
- This helps prevent the 'relation "databasechangelog" already exists' error seen in concurrent startups.

Issue-ID:CPS-2752
Change-Id: I051a8edd5ddab5a9fb012183b0526c113d90304e
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2 months agoRefactoring to combine LcmEventsCreator and LcmEventsCreatorHelper 58/140658/2
emaclee [Mon, 7 Apr 2025 12:44:28 +0000 (13:44 +0100)]
Refactoring to combine LcmEventsCreator and LcmEventsCreatorHelper

- 1. Move all in LcmEventsCreatorHelper to LcmEventsCreator
- 2. Rename LcmEventsCreator to LcmEventsProducerHelper
- 3. Rename and update relevant tests
- 4. Refactor words 'publisher' to 'producer' and 'publish' to 'send' where applicable

Issue-ID: CPS-2726
Change-Id: Id0d53f851d25823e0456e95917c2be8856a20c40
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
2 months agoMerge "Fix release notes: only mention end user features"
Toine Siebelink [Wed, 9 Apr 2025 16:00:49 +0000 (16:00 +0000)]
Merge "Fix release notes: only mention end user features"

2 months agoFix release notes: only mention end user features 53/140653/3
ToineSiebelink [Thu, 3 Apr 2025 17:23:10 +0000 (18:23 +0100)]
Fix release notes: only mention end user features

Issue-ID: CPS-2169

Change-Id: I5c1572d687fa7b4b0de0f57a318b63988b508121
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
2 months agoUpdate the script that is responsible for copyright checks 65/140665/1
halil.cakal [Mon, 7 Apr 2025 16:26:59 +0000 (17:26 +0100)]
Update the script that is responsible for copyright checks

- Since no more Nordix Foundation license header must be present in the codes, the script
  should look for OpenInfra Foundation Europe
- As an example LcmEventsCmHandleStateHandler.java has been modified and
  there is no issue reported

Issue-ID: CPS-2754

Change-Id: I4f251328649916fc482756ec94302b426e224e3b
Signed-off-by: halil.cakal <halil.cakal@est.tech>
2 months agoMerge "Refactor Cps Delta code from CpsDataService to CpsDeltaService"
Priyank Maheshwari [Mon, 7 Apr 2025 10:23:28 +0000 (10:23 +0000)]
Merge "Refactor Cps Delta code from CpsDataService to CpsDeltaService"

2 months agoMerge "Refactor Consumers/Producers based on agreed format #2"
Priyank Maheshwari [Mon, 7 Apr 2025 09:45:24 +0000 (09:45 +0000)]
Merge "Refactor Consumers/Producers based on agreed format #2"

2 months agoRefactor Consumers/Producers based on agreed format #2 07/140607/3
emaclee [Thu, 27 Mar 2025 12:40:26 +0000 (12:40 +0000)]
Refactor Consumers/Producers based on agreed format #2

- 'EventsPublisher' to 'EventsProducer'
- terms 'publish' is replaced with 'send' to follow kafka
  methods
- LcmEventsProducer is not fully changed as it may affect
  metrics; will handle on seperate patch

Issue-ID: CPS-2597
Change-Id: I310fc60fd0ff85eb83f2f3c6f9b54c569b3ff902
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
2 months agoUpdate Springboot to eliminate transitive vulnerabilities 33/140633/2
egernug [Tue, 1 Apr 2025 09:12:57 +0000 (10:12 +0100)]
Update Springboot to eliminate transitive vulnerabilities

Issue-ID: CPS-2725

Change-Id: Iafa07af4d95bc2f396385c607dde63f2d3eab8c8
Signed-off-by: egernug <gerard.nugent@est.tech>
2 months agoMerge "Set initial delay in ModuleSyncWatchdog and DataSyncWatchdog"
Sourabh Sourabh [Thu, 3 Apr 2025 17:01:51 +0000 (17:01 +0000)]
Merge "Set initial delay in ModuleSyncWatchdog and DataSyncWatchdog"

2 months agoMerge "Use alternate IDs for Legacy Batch Read operation"
Daniel Hanrahan [Thu, 3 Apr 2025 12:09:05 +0000 (12:09 +0000)]
Merge "Use alternate IDs for Legacy Batch Read operation"

2 months agoSet initial delay in ModuleSyncWatchdog and DataSyncWatchdog 10/140610/11
leventecsanyi [Thu, 27 Mar 2025 15:26:31 +0000 (16:26 +0100)]
Set initial delay in ModuleSyncWatchdog and DataSyncWatchdog

  - added initial delay values to yml files
  - modified the RTD

Issue-ID: CPS-2731
Change-Id: I3dc8fd9ee44428c91c7f73e9e6a351ea61c0653c
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
2 months agoUse alternate IDs for Legacy Batch Read operation 44/140644/2
halil.cakal [Wed, 2 Apr 2025 09:49:07 +0000 (10:49 +0100)]
Use alternate IDs for Legacy Batch Read operation

- generate a random set of alternate ids out of 50K
- each alternate id is unique in the batches
- this is for both kpi and endurance pipelines

Issue-ID: CPS-2417

Change-Id: Id848593d472fa1df677ded301948e526e193aa5a
Signed-off-by: halil.cakal <halil.cakal@est.tech>
2 months agoRefactor Cps Delta code from CpsDataService to CpsDeltaService 43/140343/22
Arpit Singh [Wed, 12 Mar 2025 07:03:35 +0000 (12:33 +0530)]
Refactor Cps Delta code from CpsDataService to CpsDeltaService

- added new open api yaml file defining CPS delta endpoints
- Added new REST controller for CPS Delta
- Moved CPS Delta endpoints to new java interface, out from CPS Data
- Added integration tests for CPS Delta
- CPS Delta rest controller and java interface have the two delta
  endpoints: Delta between anchors and Delta between anchor and payload.

Issue-ID: CPS-2320
Change-Id: I556e70623a8c18f8cde3cd28bac890296019c0e2
Signed-off-by: Arpit Singh <AS00745003@techmahindra.com>
2 months agoAdd 'unpublished' REST interface for DataJobs Write to allow K6 testing 04/140604/8
sourabh_sourabh [Thu, 27 Mar 2025 10:15:46 +0000 (10:15 +0000)]
Add 'unpublished' REST interface for DataJobs Write to allow K6 testing

- Created a new hidden REST endpoint to use write job for internal
  testing like KPI and endurence.

Issue-ID: CPS-2718
Change-Id: I1c4c7bc145dd522d344bd98ea04f05068b3e43eb
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2 months agoMerge "Move SDNC and PNFSim into docker-compose for CSITs"
Priyank Maheshwari [Wed, 2 Apr 2025 12:23:00 +0000 (12:23 +0000)]
Merge "Move SDNC and PNFSim into docker-compose for CSITs"

2 months agoPerformance fix for Module search outputting Alternate IDs 40/140640/3
danielhanrahan [Tue, 1 Apr 2025 15:04:39 +0000 (16:04 +0100)]
Performance fix for Module search outputting Alternate IDs

Use OMIT_DESCENDANTS instead of INCLUDE_ALL_DESCENDANTS.

Issue-ID: CPS-2738
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: If0ee84baa297137f7275e611905e4183a16c7ee1

2 months agoMerge "Added logging around write job operation to verify request and response detail...
Toine Siebelink [Wed, 2 Apr 2025 08:33:43 +0000 (08:33 +0000)]
Merge "Added logging around write job operation to verify request and response details, with a focus on the number of jobs involved"

2 months agoMerge "Fix for Trust Level search with alternate ID output"
Toine Siebelink [Wed, 2 Apr 2025 07:42:35 +0000 (07:42 +0000)]
Merge "Fix for Trust Level search with alternate ID output"

2 months agoFix for Trust Level search with alternate ID output 41/140641/1
danielhanrahan [Tue, 1 Apr 2025 14:18:48 +0000 (15:18 +0100)]
Fix for Trust Level search with alternate ID output

Currently the requests are timing out, over 60 seconds.
This commit restores the older implementation not using
attribute-axis, so that when the DataNodes are fetched,
we can get both the Cm Handle ID and Alternate ID.

Issue-ID: CPS-2738
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: I2072eb5cae2fd04d8217be48c261e219c948a339

2 months agoAdded logging around write job operation to verify request and response details,... 24/140624/8
sourabh_sourabh [Sat, 29 Mar 2025 15:26:06 +0000 (15:26 +0000)]
Added logging around write job operation to verify request and response details, with a focus on the number of jobs involved

Issue-ID: CPS-2735
Change-Id: I8dd6260556b4949bda5695d22f3f862c0786e385
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
2 months agoMerge "Update release notes for Flux improvement"
Priyank Maheshwari [Tue, 1 Apr 2025 14:15:07 +0000 (14:15 +0000)]
Merge "Update release notes for Flux improvement"

2 months agoUpdate batch id searches to return alternate ids 12/140612/3
halil.cakal [Thu, 27 Mar 2025 11:13:03 +0000 (11:13 +0000)]
Update batch id searches to return alternate ids

- keep the query parameter (?outputAlternateId=true) common for id-searches and searches
  since it does not have any negative impact for the searches when provided this way the code
  is more simple and readable
- function names kept the same to comply with the response messages
- the plot names kept the same to maintain the history in local storage of kpi plot job

Issue-ID: CPS-2715

Change-Id: I8146e7ecb04ddd4b907fae94e91fd78bd535b58c
Signed-off-by: halil.cakal <halil.cakal@est.tech>