Merge "Use native query to delete data nodes"

Use native query to delete data nodes

- Used hashcode to remove child fragment based on it's xpath
- Used native query to delete fragment by altering fragment_parent_id_fkey CONSTRAINT.

Issue-ID: CPS-1439

Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: If19c449818e18f8fd666503b7346704eeb4a95d0
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

Merge "Further TTL increase"

Increase duration for CpsPathUtilPerfTest

Issue-ID: CPS-1460
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: I47f226c44351513b77b5cbb9b7d91e13f8bf4db6

Merge "Added depth parameter in query nodes API."

Merge "[NCMP] Consume & Forward to client topic"

[NCMP] Consume & Forward to client topic

-Consumes event from dmi-cm-events
-Immediately forwards to static topic (topic selection for events comes
later from subscription information)
-Added Kafka test
-SHOULD BE MERGED BEFORE DMI PART

Issue-ID: CPS-138
Signed-off-by: JosephKeenan <joseph.keenan@est.tech>
Change-Id: I0a426381e2c3f9173b8d3916960c05722ad4f77d
Signed-off-by: seanbeirne <sean.beirne@est.tech>

Added depth parameter in query nodes API.

Issue-ID: CPS-1381
Change-ID: I73f97f986a817d423f93a8d922dcd9647b1206bb
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Further TTL increase

- increased TTL to 10mins or 600secs now.
- Added few more log informations.

Issue-ID: CPS-1446
Change-Id: Ie0508b426fb8d1232f0cc3c1d428a27178294323
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Normalize xpaths for getDataNodes

Issue-ID: CPS-1457
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: I93d19666c168aa69da73eadbfef0fc54181aec52

Optimal TTL value for Module Sync WD

- setting optimal value for TTL to 120secs from 200secs.
- Since the retry parameter is configurable , not changing the same in
  patch , but can be set as per the environment we are working on

Issue-ID: CPS-1446
Change-Id: Ifaca07ad1a049a88b2609fd7a1a8e1e2fa478dd9
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Merge "Bug Analysis for LOCKED cmHandles"

Bug Analysis for LOCKED cmHandles

- Increased TTL time from 60secs to 200secs
- Added few info level logs to analyze more

Issue-ID: CPS-1446
Change-Id: If4456040e51bd33a5bf321f48440e44af7698154
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Add delete performance tests

- move existing delete tests to new class
- add tests for deleting lists and list elements
- add test for deleting root node

Issue-ID: CPS-1437
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: I81228aa9473ed28d550db64b28c38abb1c1016f5

Merge "Fetch CM handles by collection of xpaths"

Merge "Updating CmHandleStates using batch operation"

Updating CmHandleStates using batch operation

Issue-ID: CPS-1424
Signed-off-by: JosephKeenan <joseph.keenan@est.tech>
Change-Id: Ia67db468ece4a7ab694d95cb63a954f24dd8cb55

Fetch CM handles by collection of xpaths

- Added FragmentRepositoryMultiPathQuery
- Removed Hibernate method for same
- Added perf. test
- Handle escaping of single qoutes in sql-data
- Increased timing for path paser performance test

Issue-ID: CPS-1422
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: Ibea12a44bffd29ed43cc1560b507d1fa7e968b8b

CPS PATCH operation does not merge existing data

Altered code to add attributes and not overwrite attributes + tests

Issue-ID: CPS-1442
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: I23c5f6a65b98ea1b05af62a38a874c228cc82067

DocBuilderFactory one instance

- setting the property for DocumentBuilderFactory only once.

Issue-ID: CPS-1435
Change-Id: Ib1509e86c2f5572d184945dc55f1818a7147c259
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Local DocBuilderFactory fix XEE

- local DocumentBuilderFactory fix for prevention of XML External Entity

Issue-ID: CPS-1435
Change-Id: Ib88268edc5975bf0fe4e3e56bc704f266280af4b
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Test XEE in SonarQube

- test by removing the attributes for transformerfactory to check if
  sonarqube gives the vulnerability or not

Issue-ID: CPS-1435
Change-Id: I087796b1bbc465655fd741f678a9b2b417d174dd
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Local DocumentBuilderFactory

- trying to use local DocumentBuilderFactory without any condition

Issue-ID: CPS-1435
Change-Id: I209a31633427cbf6fdac3e465154a1b89b9ca89f
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XEE prevention with all props

- include all properties to prevent XEE for DocumentBuilderFactory

Issue-ID: CPS-1435
Change-Id: I5a740f34072af348fe2df282fba7babeff4299d8
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Add delete performance tests

Issue-ID: CPS-1437
Signed-off-by: danielhanrahan <daniel.hanrahan@est.tech>
Change-Id: If940d7633ab6f85b6e1e311af7077ce2006cfa6a

Document and Transformer Factory refactor

- Extracted out document and transformer factory code

Issue-ID: CPS-1435
Change-Id: I7c296a797871b0724f6de51833396a5d822c7d76
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XEE remove few property

- remove and test if it fixes the issue

Issue-ID: CPS-1435
Change-Id: Id70509494c105c3e73e8217d0bfb6334aa0cc609
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XEE add more property

- adding few more properties to fix the problem

Issue-ID: CPS-1435
Change-Id: I2e952b38bae5dab396911ea1ce4a1125d0402369
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XXE prevention

- xml external entity prevention in the XmlFileUtils
- setting the features only once for the document builder factory

Issue-ID: CPS-1435
Change-Id: I06f9ac4bcdb0a90262f237489c6c50d8fde33c0d
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Minor documentation fix

- minor tag fix for param in documentation

Issue-ID: CPS-1435
Change-Id: I23ff1ebce54ee30cd84a58be0340ab41ad32b69a
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Merge "Fix SonarQube warnings"

Fix SonarQube warnings

- fixed most new warnings (required some rerfactoring and renaming)
- had to change method order (becasue private/public)
  which loosk liek a big change because gerrit cannot match thee old/new correctly :-(
- some reformatting to improve readability
- used autoclosable fro streams
- use of deprecated ODL methods will be addressed by separate commit (Lee-Anjella)

Issue-ID: CPS-475
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I4b2f74492ac9e2c644659edab1e7ff05dddcbd20

Merge "Update Release Notes"

Update Release Notes

- Added CPS-1171 and others
- Corrected mistakes

Issue-ID: CPS-1171
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I30f51c1b295e2e28fa07a2174fcc2c5db761ce5e

Organize performance test

- Separated CpsPathUtil performance tests in dedicated file
- Used smaller sample to speed up CpsPathUtil test
- Increased margin on CpsModuleReferenceRepository performance test to rpevent accidental failures
- Added profiles to cps path parser module
- Use Regex to include any (future) perforamcne test using same naming

Issue-ID: CPS-1421
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I9da1614b6f067e946a4f088c6417479fa6c71988

Merge "Temp Table Creation improvements"

Merge "XML content on create anchors node support"

Temp Table Creation improvements

- extracted methods for more generic temp table creation
- optimized
- added perf test on use case that depends on temp table creation
- had some doubts about stopwatch use in recent tests, but was all OK, just renamed some

Issue-ID: CPS-1422
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I22cabb9b0ba1b0aa8576a2d756d77af46eebc1b0

Merge "Add fix for posting nodes with xPath with '/'"

XML content on create anchors node support

Add XML content type support on anchor node creation.

Issue-ID: CPS-1257
Change-Id: I7e7a9a1961b6e81de93a4e32e842b47f8a163a09
Signed-off-by: Michal Jagiello <michal.jagiello@t-mobile.pl>
Signed-off-by: Lee Anjella Macabuhay <lee.anjella.macabuhay@est.tech>

Merge "Consume Subscription Creation Event"

Consume Subscription Creation Event

Issue-ID: CPS-1392
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I0a5a8c256319a1a2944ee6606db1c14b50e8f8e4

CSIT fix to use correct lib

- Explicitly setting pylibs.txt file to use 3.0.0 version of
  robotframework-selenium2library
- Also updating the script to be read from WORKSPACE.

Change-Id: Ic66c5357f53e73678aac439d095d318dbd03c333
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Add fix for posting nodes with xPath with '/'

- YangUtils method changed from using REGEX to cps path parser
- unit test added for cps path util

Issue-ID: CPS-1433
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
Change-Id: Ibb9efdd09423f9bade4a4a557d7d9ed49aa44ef4

CpsPath Query Optimization

- Optimized CpsPathqueries with descendants that only care about the xpath (no attribuets checks)
- Use native query with regular expression for target xpath and descendants
- Refactored so existing sql-geneartion code can be re-used in different repository implementations
- Adjusted related performance test expectations

Issue-ID: CPS-1421
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I3a807a14478c4b3272a5335d31c9aa3615eb2bee

Merge "API versioning supported and added different versions for POST APIs"

Check preformance impacts

-Added a basic perf. test to process 200K CPS paths using parser

Issue-ID: CPS-1409
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Ibf8aed53506d28ba08f7166a44bd6fe9aefa5952

Fix Delete uses case with '/' in path

-Extend and use cpsPath parser (util) classes instead of regex

Issue-ID: CPS-1409
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I230c9eb71cc34264983830b39149511b95c4b4a6

Merge "Use public exposed interface of CPS service instead of spi"

RTD Support for multiple data trees under 1 anchor

Signed-off-by: arpitsingh <as00745003@techmahindra.com>
Change-Id: Icab15ac79053f91b2a126f944fec86474c944f80

API versioning supported and added different versions for POST APIs

Issue-ID: CPS-1189
Change-ID: I73f97f986a817d423f92f8d922dcd9647b1214aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Use public exposed interface of CPS service instead of spi

- Used public interface of cps service to get data node or anchor names.

Issue-ID: CPS-1427
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: I2a891dd5f8955977295f32005e49543886eac88c
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

Merge "CPS-341 Support for multiple data tree instances under 1 anchor"

Chore: Update robot-framework scripts to use Python3.10

Migrate these CSIT scripts to explicitly use the latest Python 3.x
version available in pyenv, the same as the earlier jobs that setup
the robot framework environment. Also, output some useful versioning
information to aid debugging.

Issue-ID: CCSDK-3820
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I332f2c46018d4c3433f0a3df9a0f35bb90241b07

CPS-341 Support for multiple data tree instances under 1 anchor

- Updated the parseJsonData method so it can parse JSON with multiple data trees, now it returns a ContainerNode
- ContainerNode holds a collection of NormalizedNodes
- Updated DataNodeBuilder and FragmentRepository as well to support collection of NormalizedNodes
- Added new methods in CpsDataPersistenceService to store multiple Data Nodes
- Added new test cases
- Updated existing test cases and fixed code coverage
- Addressed comments from previous patch

Issue-ID: CPS-341
Change-Id: Ie893e91c0fbfb139a1a406e962721b0f52412ced
Signed-off-by: arpitsingh <as00745003@techmahindra.com>

Merge "Query Optimization"

Merge "Analyze outdated CPS Dependencies"

Analyze outdated CPS Dependencies

- Added guava & postgres dependency update

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Change-Id: I28dab7c083330fff704d5e438566878f78eb53f9
Issue-Id: CPS-1408

Merge "Fixing SonarQube violations"

Query Optimization

- Refactor performance tests as multiple separate (data driven) tests
- This to improve redability and re-use and amke sure al tets are always ran
  (makign it easier to test/change thresholds for multiple scenarios at once)
- Add new query performance tests (on many descendants)

Issue-ID: CPS-475
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: If5703fcb9e2233ded64d31d2342de2e7f5549687

Merge "Fix DataNodeBuilder for ChoiceNodes"

Fix DataNodeBuilder for ChoiceNodes

Fix implemented to handle YangChoiceNode in right format

Issue-ID: CPS-1352
Signed-off-by: Lena Peuker <PeukerL@telekom.de>
Change-Id: I48d433bac96cfc647bc31c82817870995bace860

Increase performance test margins

- Increased duration times

Issue-Id: CPS-1420
Change-Id: I510801a65194b63ce72a716ee3fc7258aef61207
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Fixing SonarQube violations

- Code smell fixes

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Issue-ID: CPS-475
Change-Id: I4b47ece82aed7e4b31aaa28e601e419fb6f06718

RTD Release Notes get all schema sets

Issue-ID: CPS-1187
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: Ic175350593b8be8671dce52fa8ccbe05820a86d5

Merge "Added API to get all schema sets for a given dataspace."

Added API to get all schema sets for a given dataspace.

Issue-ID: CPS-1187
Change-ID: I73f97f986a817d423f93a8d922dcd9647b0914aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Merge "Fix SonarQube Violations"

Merge "Chore: Re-instate robotframework-requests==0.9.3"

Chore: Re-instate robotframework-requests==0.9.3

Need to re-instate a pinned version for the Python test suite.
Also we can add back the backup installation method from the script
in ci-management.

Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: If0ec07edb1c156b06a9c2621a4e22bd35b9240a9

Merge "Update outdated dependencies"

Merge "Enhance read performance tests"

Merge "Upgrade Open daylight yang tool to version 8.0.6"

Upgrade Open daylight yang tool to version 8.0.6

- changed classes according to upgrade of yang tool
- changed methods in YangUtils class to accomodate change for
  jSonParserStream method
- updated unit tests for the upgrade

Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
Change-Id: Ib62fe424150ed0695eed168e1ae21f4d9a61cdf4

Fix SonarQube Violations

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Issue-ID: CPS-475
Change-Id: Ic5c83cf2952c649fb634c6a1660696e7575cc1c7

Update outdated dependencies

- Updated cps-dependecies spring-web & jackson-databind & postgres version numbers

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Change-Id: I3c9e72cafafb734852a594d14184edfc459db9fb
Issue-ID: CPS-1397

Enhance read performance tests

- Use cleaner base-data to prevent aready existing exception on generated IDs
- Increased sample size to 10K
- Test 3 scenarios now: get parent, get root and query as all 3 required separate perf improvements
- Adjusted Timing for new size (allowing approx 10% margin in Nordix build)

Issue-ID: CPS-1171

Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Id3867c8d33cd2c57d4d815c687561e23d1029bfd

Merge "Filter on private properties of CM Handles"

Filter on private properties of CM Handles

- Moved cm handle query validation to cps-ncmp-service (where it belongs!)
- Added new enum type for private/public field types
- Created new methods for private and public queries
- Added new REST endpoint
- Created service methods for filtering on different types of properties
- Refactored getPublicPropertyPairs and queryCmHandleAnyProperties
- Added unit test for the controller layer
- Fixed refactoring suggestions
- Imporved code coverage with unit tests
- Refactoring
- Added new functionality to NcmpRestInputMapper
- Updated version number to 3.2.1-SNAPSHOT and updated release-notes.rst

Issue-ID: CPS-1236
Change-Id: I0ddf6866473f7c3c6b8507d222d441bf97ca6bdc
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Documentation update for enabled dataspace filter

Issue-ID: CPS-1361
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I988350289a02e44515aeb2acfe6a83845d66e0d9

Fix: Update python environment to ROBOT3_VENV

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: Id5ea08481bdf14cce149f047d7cd5b8d8254d200

Fix: Call source_safely after function declaration

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: Id74b1f24655a83f68b5f28c9eab68bb069f0bc74

Merge "Performance Test settings exploration"

Merge "Added get APIs for dataspace."

Performance Test settings exploration

Issue-ID: CPS-1351
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I91624b74fc712078a22b9dc18851ca0b5de6a911

Added get APIs for dataspace.

Issue-ID: CPS-1186
Change-ID: I73f97f986a817d423f93a8d922dcd9647b0829aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Fix: Swap prepare-csit.sh to Python3 robot framework

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I246f9f76996943826afc7d05796e6ddc5c0de6dc

Merge "Revert Hack to get CSIT test working Reverted to using ROBOT3_VENV  (tetsing CI-mgmnt uodate)"

Fix: Swap run-csit.sh to Python3 robot framework

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I15ad8a147a669c503ea023f79cef101baf99c21f

Revert Hack to get CSIT test working
Reverted to using ROBOT3_VENV  (tetsing CI-mgmnt uodate)

Issue-ID: RELENG-4523
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I1e8ae6cba2b303042d9f22e1601e9846ebd2f7bf

Merge "Query data NCMP-Operational with CPSpath"

Query data NCMP-Operational with CPSpath

New GET Endpoint: /v1/ch/{cm-handle}/data/ds/{ncmp-datastore-name}/query?cps-path={CPSPath}
Implemented error for {ncmp-datastore-name} other than operational
- (Toine) Refactored and renamed (abstract) handler for better re-use
Mainly by introducing a separate handler for OperationalQuery

Reviewers Toine, Sourabh, Priyank

Issue-ID: CPS-1002
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: Iaca018869d95d4ce800072431baa190050a6dad0

Upgrade testcontainers to 1.17.3

- upgrade the testcontainer version to run the integration tests

Issue-ID: CPS-1360
Change-Id: Ia84ebd68a10c72405d8038dbc30a12c97d37f188
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Merge "Node API - Find root by anchor method performance"

Merge "HTTP error code 500 changed to 400 for invalid JSON payload"

Node API - Find root by anchor method performance

- Modified findFirstRootByAnchor
- Used fragment extract to build fragment entity to get root data node.

Reviewers : Toine, Priyank and Joe

Issue-ID: CPS-1171
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: I843f270e4781f91ae496f39b976e2a7f2a14d55e
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

Temporary registry update

- temporarily updating the registry to test the failing tests.

Issue-ID: CPS-1360
Change-Id: I7e303bb72b3f87565e27bcca7b1d9f5afc2b1417
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

HTTP error code 500 changed to 400 for invalid JSON payload

Issue-ID: CPS-1289
Change-ID: I73f97f986a817d423f93a8d922dcd9647b1012aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Merge "Hack to get CSIT test working"

Fix RTD broken links

- Broken link for logback xml file
- Broken link for oom_user_guide

Issue-ID: CPS-1378
Change-Id: I071d5ad890c1471b50a43517427eeab27631fd52
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>