Document and Transformer Factory refactor

- Extracted out document and transformer factory code

Issue-ID: CPS-1435
Change-Id: I7c296a797871b0724f6de51833396a5d822c7d76
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XEE remove few property

- remove and test if it fixes the issue

Issue-ID: CPS-1435
Change-Id: Id70509494c105c3e73e8217d0bfb6334aa0cc609
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XEE add more property

- adding few more properties to fix the problem

Issue-ID: CPS-1435
Change-Id: I2e952b38bae5dab396911ea1ce4a1125d0402369
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

XXE prevention

- xml external entity prevention in the XmlFileUtils
- setting the features only once for the document builder factory

Issue-ID: CPS-1435
Change-Id: I06f9ac4bcdb0a90262f237489c6c50d8fde33c0d
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Minor documentation fix

- minor tag fix for param in documentation

Issue-ID: CPS-1435
Change-Id: I23ff1ebce54ee30cd84a58be0340ab41ad32b69a
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Merge "Fix SonarQube warnings"

Fix SonarQube warnings

- fixed most new warnings (required some rerfactoring and renaming)
- had to change method order (becasue private/public)
  which loosk liek a big change because gerrit cannot match thee old/new correctly :-(
- some reformatting to improve readability
- used autoclosable fro streams
- use of deprecated ODL methods will be addressed by separate commit (Lee-Anjella)

Issue-ID: CPS-475
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I4b2f74492ac9e2c644659edab1e7ff05dddcbd20

Merge "Update Release Notes"

Update Release Notes

- Added CPS-1171 and others
- Corrected mistakes

Issue-ID: CPS-1171
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I30f51c1b295e2e28fa07a2174fcc2c5db761ce5e

Organize performance test

- Separated CpsPathUtil performance tests in dedicated file
- Used smaller sample to speed up CpsPathUtil test
- Increased margin on CpsModuleReferenceRepository performance test to rpevent accidental failures
- Added profiles to cps path parser module
- Use Regex to include any (future) perforamcne test using same naming

Issue-ID: CPS-1421
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I9da1614b6f067e946a4f088c6417479fa6c71988

Merge "Temp Table Creation improvements"

Merge "XML content on create anchors node support"

Temp Table Creation improvements

- extracted methods for more generic temp table creation
- optimized
- added perf test on use case that depends on temp table creation
- had some doubts about stopwatch use in recent tests, but was all OK, just renamed some

Issue-ID: CPS-1422
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I22cabb9b0ba1b0aa8576a2d756d77af46eebc1b0

Merge "Add fix for posting nodes with xPath with '/'"

XML content on create anchors node support

Add XML content type support on anchor node creation.

Issue-ID: CPS-1257
Change-Id: I7e7a9a1961b6e81de93a4e32e842b47f8a163a09
Signed-off-by: Michal Jagiello <michal.jagiello@t-mobile.pl>
Signed-off-by: Lee Anjella Macabuhay <lee.anjella.macabuhay@est.tech>

Merge "Consume Subscription Creation Event"

Consume Subscription Creation Event

Issue-ID: CPS-1392
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I0a5a8c256319a1a2944ee6606db1c14b50e8f8e4

CSIT fix to use correct lib

- Explicitly setting pylibs.txt file to use 3.0.0 version of
  robotframework-selenium2library
- Also updating the script to be read from WORKSPACE.

Change-Id: Ic66c5357f53e73678aac439d095d318dbd03c333
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Add fix for posting nodes with xPath with '/'

- YangUtils method changed from using REGEX to cps path parser
- unit test added for cps path util

Issue-ID: CPS-1433
Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
Change-Id: Ibb9efdd09423f9bade4a4a557d7d9ed49aa44ef4

CpsPath Query Optimization

- Optimized CpsPathqueries with descendants that only care about the xpath (no attribuets checks)
- Use native query with regular expression for target xpath and descendants
- Refactored so existing sql-geneartion code can be re-used in different repository implementations
- Adjusted related performance test expectations

Issue-ID: CPS-1421
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I3a807a14478c4b3272a5335d31c9aa3615eb2bee

Merge "API versioning supported and added different versions for POST APIs"

Check preformance impacts

-Added a basic perf. test to process 200K CPS paths using parser

Issue-ID: CPS-1409
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Ibf8aed53506d28ba08f7166a44bd6fe9aefa5952

Fix Delete uses case with '/' in path

-Extend and use cpsPath parser (util) classes instead of regex

Issue-ID: CPS-1409
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I230c9eb71cc34264983830b39149511b95c4b4a6

Merge "Use public exposed interface of CPS service instead of spi"

RTD Support for multiple data trees under 1 anchor

Signed-off-by: arpitsingh <as00745003@techmahindra.com>
Change-Id: Icab15ac79053f91b2a126f944fec86474c944f80

API versioning supported and added different versions for POST APIs

Issue-ID: CPS-1189
Change-ID: I73f97f986a817d423f92f8d922dcd9647b1214aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Use public exposed interface of CPS service instead of spi

- Used public interface of cps service to get data node or anchor names.

Issue-ID: CPS-1427
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: I2a891dd5f8955977295f32005e49543886eac88c
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

Merge "CPS-341 Support for multiple data tree instances under 1 anchor"

Chore: Update robot-framework scripts to use Python3.10

Migrate these CSIT scripts to explicitly use the latest Python 3.x
version available in pyenv, the same as the earlier jobs that setup
the robot framework environment. Also, output some useful versioning
information to aid debugging.

Issue-ID: CCSDK-3820
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I332f2c46018d4c3433f0a3df9a0f35bb90241b07

CPS-341 Support for multiple data tree instances under 1 anchor

- Updated the parseJsonData method so it can parse JSON with multiple data trees, now it returns a ContainerNode
- ContainerNode holds a collection of NormalizedNodes
- Updated DataNodeBuilder and FragmentRepository as well to support collection of NormalizedNodes
- Added new methods in CpsDataPersistenceService to store multiple Data Nodes
- Added new test cases
- Updated existing test cases and fixed code coverage
- Addressed comments from previous patch

Issue-ID: CPS-341
Change-Id: Ie893e91c0fbfb139a1a406e962721b0f52412ced
Signed-off-by: arpitsingh <as00745003@techmahindra.com>

Merge "Query Optimization"

Merge "Analyze outdated CPS Dependencies"

Analyze outdated CPS Dependencies

- Added guava & postgres dependency update

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Change-Id: I28dab7c083330fff704d5e438566878f78eb53f9
Issue-Id: CPS-1408

Merge "Fixing SonarQube violations"

Query Optimization

- Refactor performance tests as multiple separate (data driven) tests
- This to improve redability and re-use and amke sure al tets are always ran
  (makign it easier to test/change thresholds for multiple scenarios at once)
- Add new query performance tests (on many descendants)

Issue-ID: CPS-475
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: If5703fcb9e2233ded64d31d2342de2e7f5549687

Merge "Fix DataNodeBuilder for ChoiceNodes"

Fix DataNodeBuilder for ChoiceNodes

Fix implemented to handle YangChoiceNode in right format

Issue-ID: CPS-1352
Signed-off-by: Lena Peuker <PeukerL@telekom.de>
Change-Id: I48d433bac96cfc647bc31c82817870995bace860

Increase performance test margins

- Increased duration times

Issue-Id: CPS-1420
Change-Id: I510801a65194b63ce72a716ee3fc7258aef61207
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Fixing SonarQube violations

- Code smell fixes

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Issue-ID: CPS-475
Change-Id: I4b47ece82aed7e4b31aaa28e601e419fb6f06718

RTD Release Notes get all schema sets

Issue-ID: CPS-1187
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: Ic175350593b8be8671dce52fa8ccbe05820a86d5

Merge "Added API to get all schema sets for a given dataspace."

Added API to get all schema sets for a given dataspace.

Issue-ID: CPS-1187
Change-ID: I73f97f986a817d423f93a8d922dcd9647b0914aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Merge "Fix SonarQube Violations"

Merge "Chore: Re-instate robotframework-requests==0.9.3"

Chore: Re-instate robotframework-requests==0.9.3

Need to re-instate a pinned version for the Python test suite.
Also we can add back the backup installation method from the script
in ci-management.

Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: If0ec07edb1c156b06a9c2621a4e22bd35b9240a9

Merge "Update outdated dependencies"

Merge "Enhance read performance tests"

Merge "Upgrade Open daylight yang tool to version 8.0.6"

Upgrade Open daylight yang tool to version 8.0.6

- changed classes according to upgrade of yang tool
- changed methods in YangUtils class to accomodate change for
  jSonParserStream method
- updated unit tests for the upgrade

Signed-off-by: emaclee <lee.anjella.macabuhay@est.tech>
Change-Id: Ib62fe424150ed0695eed168e1ae21f4d9a61cdf4

Fix SonarQube Violations

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Issue-ID: CPS-475
Change-Id: Ic5c83cf2952c649fb634c6a1660696e7575cc1c7

Update outdated dependencies

- Updated cps-dependecies spring-web & jackson-databind & postgres version numbers

Signed-off-by: leventecsanyi <levente.csanyi@est.tech>
Change-Id: I3c9e72cafafb734852a594d14184edfc459db9fb
Issue-ID: CPS-1397

Enhance read performance tests

- Use cleaner base-data to prevent aready existing exception on generated IDs
- Increased sample size to 10K
- Test 3 scenarios now: get parent, get root and query as all 3 required separate perf improvements
- Adjusted Timing for new size (allowing approx 10% margin in Nordix build)

Issue-ID: CPS-1171

Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Id3867c8d33cd2c57d4d815c687561e23d1029bfd

Merge "Filter on private properties of CM Handles"

Filter on private properties of CM Handles

- Moved cm handle query validation to cps-ncmp-service (where it belongs!)
- Added new enum type for private/public field types
- Created new methods for private and public queries
- Added new REST endpoint
- Created service methods for filtering on different types of properties
- Refactored getPublicPropertyPairs and queryCmHandleAnyProperties
- Added unit test for the controller layer
- Fixed refactoring suggestions
- Imporved code coverage with unit tests
- Refactoring
- Added new functionality to NcmpRestInputMapper
- Updated version number to 3.2.1-SNAPSHOT and updated release-notes.rst

Issue-ID: CPS-1236
Change-Id: I0ddf6866473f7c3c6b8507d222d441bf97ca6bdc
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Documentation update for enabled dataspace filter

Issue-ID: CPS-1361
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I988350289a02e44515aeb2acfe6a83845d66e0d9

Fix: Update python environment to ROBOT3_VENV

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: Id5ea08481bdf14cce149f047d7cd5b8d8254d200

Fix: Call source_safely after function declaration

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: Id74b1f24655a83f68b5f28c9eab68bb069f0bc74

Merge "Performance Test settings exploration"

Merge "Added get APIs for dataspace."

Performance Test settings exploration

Issue-ID: CPS-1351
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I91624b74fc712078a22b9dc18851ca0b5de6a911

Added get APIs for dataspace.

Issue-ID: CPS-1186
Change-ID: I73f97f986a817d423f93a8d922dcd9647b0829aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Fix: Swap prepare-csit.sh to Python3 robot framework

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I246f9f76996943826afc7d05796e6ddc5c0de6dc

Merge "Revert Hack to get CSIT test working Reverted to using ROBOT3_VENV  (tetsing CI-mgmnt uodate)"

Fix: Swap run-csit.sh to Python3 robot framework

Issue-ID: RELENG-4523
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Change-Id: I15ad8a147a669c503ea023f79cef101baf99c21f

Revert Hack to get CSIT test working
Reverted to using ROBOT3_VENV  (tetsing CI-mgmnt uodate)

Issue-ID: RELENG-4523
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I1e8ae6cba2b303042d9f22e1601e9846ebd2f7bf

Merge "Query data NCMP-Operational with CPSpath"

Query data NCMP-Operational with CPSpath

New GET Endpoint: /v1/ch/{cm-handle}/data/ds/{ncmp-datastore-name}/query?cps-path={CPSPath}
Implemented error for {ncmp-datastore-name} other than operational
- (Toine) Refactored and renamed (abstract) handler for better re-use
Mainly by introducing a separate handler for OperationalQuery

Reviewers Toine, Sourabh, Priyank

Issue-ID: CPS-1002
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: Iaca018869d95d4ce800072431baa190050a6dad0

Upgrade testcontainers to 1.17.3

- upgrade the testcontainer version to run the integration tests

Issue-ID: CPS-1360
Change-Id: Ia84ebd68a10c72405d8038dbc30a12c97d37f188
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Merge "Node API - Find root by anchor method performance"

Merge "HTTP error code 500 changed to 400 for invalid JSON payload"

Node API - Find root by anchor method performance

- Modified findFirstRootByAnchor
- Used fragment extract to build fragment entity to get root data node.

Reviewers : Toine, Priyank and Joe

Issue-ID: CPS-1171
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: I843f270e4781f91ae496f39b976e2a7f2a14d55e
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

Temporary registry update

- temporarily updating the registry to test the failing tests.

Issue-ID: CPS-1360
Change-Id: I7e303bb72b3f87565e27bcca7b1d9f5afc2b1417
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

HTTP error code 500 changed to 400 for invalid JSON payload

Issue-ID: CPS-1289
Change-ID: I73f97f986a817d423f93a8d922dcd9647b1012aa
Signed-off-by: rajesh.kumar <rk00747546@techmahindra.com>

Merge "Hack to get CSIT test working"

Fix RTD broken links

- Broken link for logback xml file
- Broken link for oom_user_guide

Issue-ID: CPS-1378
Change-Id: I071d5ad890c1471b50a43517427eeab27631fd52
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Hack to get CSIT test working

Issue-ID: CPS-1353
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I47b0b4e628e6685c0b63bf64ef3a147d6d9de774

Node API - GET Method performance issue

- Modified toDataNode call based on fetch descendants option.
- Used fragment extract to build fragment entity.
- Modified data set to have correct parent id for descendants.

Reviewers : Toine, Priyank and Joe

Issue-ID: CPS-1171
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Change-Id: I27a537fe72dd396722e6cfde7d8c454ed2579ec0
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>

CPS-RI performance profiles implementaion

Reviewers: Joe, Levente, Luke

Issie-ID: CPS-1351
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: Ia7b9a4bbf103d6b54f6fde9dd2d80a3c825c3046

Ensure prefix is correct module prefix

- Moved Prefix logic from RI to Service layer
- Prefix is no PREFIX propety, not the moduel name!
== RI (DB Layer) Changes
- Removed prefix logic incl hazelcast
- Added new basic ri-test for getDataNode and assert prefix is null
- Updated exsiting ri-test to us getDataNode
- Updated existing ri-test to only use " where really needed
== CPS Service Layer Changes
- Introduced PrefixResolver with clear and limited responsibility
- Use PrefixResolver where needed
- Cache prefix map per anchor, use cached entry when available
- Disabled SONAR on new Regex
== REST Layer
- Use PrefixResolver where needed

Issue-ID: CPS-1353
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Ie16f0e1ee1c280f3eb69c9e64fab69a780fb692a

Read Performance Improvement - Using Native Query

- Native query for FragmentExtracts
- Convert FragmentExtracts to tree of FragmentEntity
- Native Query now used for all Gets with descendants (orignal hibernate option only used when descendanst ommited)
- Added error handling for not-found on native query
- Ommit descendants by default on many udpate use-cases (this might have a signifcant perf. improvemnt impact too)
- Improved legacy tests for delete use-cases
- Corrected performace test expectation
- Fix TTL test realizing TTL resolution is whole seconds!

Issue-ID: CPS-1301
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: I658ac1b7b7036f01050f30bdf9e5bd175725ef1d

Updating CSIT DMI Snapshot version

Update default and DMI_VERSION property in test.prooerties

Issue-ID: CPS-1353

Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Change-Id: Ibe56f85fb047aca1d10e0d61a521c7448bd4306b

Merge "Release Notes Heading Edit"

Release Notes Heading Edit

Issue-ID: CPS-1321
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: Id2371a744ae216a5e7b2c1a50c88ad68d5883b5b

Merge "[NCMP] Add Basic Auth to OpenAPI Definitions"

Creation of DataNodeBuilder with module name prefix is very slow

- Created a new hazelcast distributed cache for anchor data config use cases.
- Module name prefix is added for root data node only.
- Cached module name prefix by anchor name on demand from database at
  first call.
- Introduced a new cache holder to have module name prefix of diff.
  levels.

Issue-ID: CPS-1326
Change-Id: I9072f5efdeea59843cd827ac556d3c0547a3a0cf
Signed-off-by: sourabh_sourabh <sourabh.sourabh@est.tech>
Signed-off-by: ToineSiebelink <toine.siebelink@est.tech>
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

[NCMP] Add Basic Auth to OpenAPI Definitions

-Adding basic auth to OpenAPI definitions

Issue-ID: CPS-1350
Signed-off-by: JosephKeenan <joseph.keenan@est.tech>
Change-Id: Iab677f2abc07eed8c3e2910191b15ba27028ee88

CPS Validator Changes

Changed CPS Validator from static to non-static
Moved CPSValidator implementation to rest interface layer to prevent duplicated usage
Created CPSValidator interface in persistence layer
Removed unnecessary tests
Refactored misplaced tests
Copyright fixes
Copyright Check Script output formatting fix

Issue-ID: CPS-1228
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: I4c99a51870e7d3b8e416bc1f909aeed5dc3baa42

Merge "Improve code coverage"

Merge "Increase read time of get datanode test"

Improve code coverage

- Fixed and refactored unit test to get over the 97% coverage baseline.
- Removed unused exception.

Issue-ID: CPS-475
Change-Id: I6dbcba58b880a584f6d9346e2aca6c763e5d2081
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Increase read time of get datanode test

Build was failing locally for me with development applications open (intellij, chrome, etc...)
After discussion with Toine we are prioritising performance test pipeline with suggested increase from 3.5 seconds to 6 seconds.

Issue-ID: CPS-1171
Signed-off-by: lukegleeson <luke.gleeson@est.tech>
Change-Id: I335fda7ba19b1fb8f4aa543833df65de26d9ccc3

CPS upgradeability doc correction

Issue-ID: CPS-1312
Signed-off-by: seanbeirne <sean.beirne@est.tech>
Change-Id: I5f35dc4a13b1898638246ebb91726c5487cbef57

Merge "Fix SonarQube Violations"

Merge "Node API - GET Method performance issue (disable sql logging)"

Bump the minor CPS version

- Bumped the CPS version from 3.1.5-SNAPSHOT to 3.2.0-SNAPSHOT
- Step 11 of the release process.

Issue-ID: CPS-1318
Change-Id: I0dce0070afcc019d2562c2f3ab9fe02207dcd86b
Signed-off-by: mpriyank <priyank.maheshwari@est.tech>

Fix SonarQube Violations

- Fixed SonarCube issue: added private constructor to TopicValidator and a unit test.

Issue-ID: CPS-475
Change-Id: I4379522f023f32c8999a3ee595392bfbab147050
Signed-off-by: leventecsanyi <levente.csanyi@est.tech>

Node API - GET Method performance issue (disable sql logging)

Reviewers: Toine, Sourabh, & Priyank

Issue-ID: CPS-1171
Signed-off-by: JosephKeenan <joseph.keenan@est.tech>
Change-Id: Ifc5c9c57c3e8bad15780c8cac09dcd3eea05df94

Merge " Node API - GET Method performance issue (junit)"

 Node API - GET Method performance issue (junit)

Reviewers: Toine, Sourabh, Levente

Issue-ID: CPS-1171
Signed-off-by: JosephKeenan <joseph.keenan@est.tech>
Change-Id: I8e59d686ead079d751a459c5933592423aa15ace

Merge "TTL for module and data sync"