Update db-migrator docs and references to MariaDB

- as MariaDB is no longer supported by PF, removing references
from docs and guides.

Issue-ID: POLICY-5256
Change-Id: I227730c4debce015f9ef9291ed7a1066ddf5ac71
Signed-off-by: adheli.tavares <adheli.tavares@est.tech>

diff --git a/docs/development/devtools/devtools.rst b/docs/development/devtools/devtools.rst
index de0a625..b0b243e 100644 (file)
--- a/docs/development/devtools/devtools.rst
+++ b/docs/development/devtools/devtools.rst
@@ -239,7 +239,7 @@ Running the API component standalone
 ++++++++++++++++++++++++++++++++++++
 
 Assuming you have successfully built the codebase using the instructions above. The only requirement for the API
-component to run is a running MariaDb/Postgres database instance. The easiest way to do this is to run the docker
+component to run is a running Postgres database instance. The easiest way to do this is to run the docker
 image, please see the official documentation for the latest information on doing so. Once the database is up and
 running, a configuration file must be provided to the api in order for it to know how to connect to the database.
 You can locate the default configuration file in the packaging of the api component:
@@ -260,7 +260,7 @@ An example of running the api using a docker compose script is located in the Po
 Running the PAP component standalone
 ++++++++++++++++++++++++++++++++++++
 
-Once you have successfully built the PAP codebase, a running MariaDb/Postgres database and Kafka instance will also be
+Once you have successfully built the PAP codebase, a running Postgres database and Kafka instance will also be
 required to start up the application. To start database and Kafka, check official documentation on how to run an
 instance of each. After database and Kafka are up and running, a configuration file must be provided to the PAP
 component in order for it to know how to connect to the database and Kafka along with other relevant configuration

diff --git a/docs/development/devtools/smoke/api-smoke.rst b/docs/development/devtools/smoke/api-smoke.rst
index 8230f33..b2c81f8 100644 (file)
--- a/docs/development/devtools/smoke/api-smoke.rst
+++ b/docs/development/devtools/smoke/api-smoke.rst
@@ -11,7 +11,8 @@ Policy API Smoke Test
 ~~~~~~~~~~~~~~~~~~~~~
 
 The policy-api smoke testing is executed against a default ONAP installation as per OOM charts.
-This test verifies the execution of all the REST api's exposed by the component to make sure the contract works as expected.
+This test verifies the execution of all the REST api's exposed by the component to make sure the
+contract works as expected.
 
 General Setup
 *************

diff --git a/docs/development/devtools/smoke/db-migrator-smoke.rst b/docs/development/devtools/smoke/db-migrator-smoke.rst
index 74b8edd..c6d8fd0 100644 (file)
--- a/docs/development/devtools/smoke/db-migrator-smoke.rst
+++ b/docs/development/devtools/smoke/db-migrator-smoke.rst
@@ -8,415 +8,51 @@ Policy DB Migrator Smoke Tests
 Prerequisites
 *************
 
-Check number of files in each release
+- Have Docker and Docker compose installed
+- Some bash knowledge
 
-.. code::
-  :number-lines:
+Preparing the test
+==================
 
-    ls 0800/upgrade/*.sql | wc -l = 96
-    ls 0900/upgrade/*.sql | wc -l = 13
-    ls 1000/upgrade/*.sql | wc -l = 9
-    ls 0800/downgrade/*.sql | wc -l = 96
-    ls 0900/downgrade/*.sql | wc -l = 13
-    ls 1000/downgrade/*.sql | wc -l = 9
+The goal for the smoke test is to confirm the any upgrade or downgrade operation between different
+db-migrator versions are completed without issues.
 
-Upgrade scripts
-===============
+So, before running the test, make sure that there are different tests doing upgrade and downgrade
+operations to the latest version. The script with test cases is under db-migrator folder in `docker
+repository <https://github.com/onap/policy-docker/tree/master/policy-db-migrator/smoke-test>`_
 
-.. code::
-  :number-lines:
+Edit the `*-tests.sh` file to add the tests and also to check if the database variables (host,
+admin user, admin password) are set correctly.
 
-    /opt/app/policy/bin/prepare_upgrade.sh policyadmin
-    /opt/app/policy/bin/db-migrator -s policyadmin -o upgrade # upgrade to Jakarta version (latest)
-    /opt/app/policy/bin/db-migrator -s policyadmin -o upgrade -t 0900 # upgrade to Istanbul
-    /opt/app/policy/bin/db-migrator -s policyadmin -o upgrade -t 0800 # upgrade to Honolulu
+Running the test
+================
 
-.. note::
-   You can also run db-migrator upgrade with the -t and -f options
+The script mentioned on the step above is ran against the `Docker compose configuration
+<https://github.com/onap/policy-docker/tree/master/compose>`_.
 
-Downgrade scripts
-=================
+Change the `db_migrator_policy_init.sh` on db-migrator service descriptor in the docker compose file
+to the `*-test.sh` file.
 
-.. code::
-  :number-lines:
+Start the service
 
-    /opt/app/policy/bin/prepare_downgrade.sh policyadmin
-    /opt/app/policy/bin/db-migrator -s policyadmin -o downgrade -t 0900 # downgrade to Istanbul
-    /opt/app/policy/bin/db-migrator -s policyadmin -o downgrade -t 0800 # downgrade to Honolulu
-    /opt/app/policy/bin/db-migrator -s policyadmin -o downgrade -t 0 # delete all tables
+.. code-block:: bash
 
-Db migrator initialization script
-=================================
+    cd ~/git/docker/compose
+    ./start-compose.sh policy-db-migrator
 
-Update /oom/kubernetes/policy/resources/config/db_migrator_policy_init.sh with the appropriate upgrade/downgrade calls.
+To collect the logs
 
-The policy version you are deploying should either be an upgrade or downgrade from the current db migrator schema version.
+.. code-block:: bash
 
-Every time you modify db_migrator_policy_init.sh you will have to undeploy, make and redeploy before updates are applied.
+    docker compose logs
+    # or
+    docker logs policy-db-migrator
 
-1. Fresh Install
-****************
+To finish execution
 
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
+.. code-block:: bash
 
-   * - Number of files run
-     - 118
-   * - Tables in policyadmin
-     - 70
-   * - Records Added
-     - 118
-   * - schema_version
-     - 1000
+    ./stop-compose.sh
 
-2. Downgrade to Honolulu (0800)
-*******************************
-
-Modify db_migrator_policy_init.sh - remove any lines referencing upgrade and add the 2 lines under "Downgrade scripts" tagged as Honolulu
-
-Make/Redeploy to run downgrade.
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 13
-   * - Tables in policyadmin
-     - 73
-   * - Records Added
-     - 13
-   * - schema_version
-     - 0800
-
-3. Upgrade to Istanbul (0900)
-*****************************
-
-Modify db_migrator_policy_init.sh - remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts".
-
-Make/Redeploy to run upgrade.
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 13
-   * - Tables in policyadmin
-     - 75
-   * - Records Added
-     - 13
-   * - schema_version
-     - 0900
-
-4. Upgrade to Istanbul (0900) without any information in the migration schema
-*****************************************************************************
-
-Ensure you are on release 0800. (This may require running a downgrade before starting the test)
-
-Drop db-migrator tables in migration schema:
-
-.. code::
-  :number-lines:
-
-    DROP TABLE schema_versions;
-    DROP TABLE policyadmin_schema_changelog;
-
-Modify db_migrator_policy_init.sh - remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts".
-
-Make/Redeploy to run upgrade.
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 13
-   * - Tables in policyadmin
-     - 75
-   * - Records Added
-     - 13
-   * - schema_version
-     - 0900
-
-5. Upgrade to Istanbul (0900) after failed downgrade
-****************************************************
-
-Ensure you are on release 0900.
-
-Rename pdpstatistics table in policyadmin schema:
-
-.. code::
-
-    RENAME TABLE pdpstatistics TO backup_pdpstatistics;
-
-Modify db_migrator_policy_init.sh - remove any lines referencing upgrade and add the 2 lines under "Downgrade scripts"
-
-Make/Redeploy to run downgrade
-
-This should result in an error (last row in policyadmin_schema_changelog will have a success value of 0)
-
-Rename backup_pdpstatistic table in policyadmin schema:
-
-.. code::
-
-    RENAME TABLE backup_pdpstatistics TO pdpstatistics;
-
-Modify db_migrator_policy_init.sh - Remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts"
-
-Make/Redeploy to run upgrade
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 11
-   * - Tables in policyadmin
-     - 75
-   * - Records Added
-     - 11
-   * - schema_version
-     - 0900
-
-6. Downgrade to Honolulu (0800) after failed downgrade
-******************************************************
-
-Ensure you are on release 0900.
-
-Add timeStamp column to papdpstatistics_enginestats:
-
-.. code::
-
-    ALTER TABLE jpapdpstatistics_enginestats ADD COLUMN timeStamp datetime DEFAULT NULL NULL AFTER UPTIME;
-
-Modify db_migrator_policy_init.sh - remove any lines referencing upgrade and add the 2 lines under "Downgrade scripts"
-
-Make/Redeploy to run downgrade
-
-This should result in an error (last row in policyadmin_schema_changelog will have a success value of 0)
-
-Remove timeStamp column from jpapdpstatistics_enginestats:
-
-.. code::
-
-    ALTER TABLE jpapdpstatistics_enginestats DROP COLUMN timeStamp;
-
-The config job will retry 5 times. If you make your fix before this limit is reached you won't need to redeploy.
-
-Redeploy to run downgrade
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 14
-   * - Tables in policyadmin
-     - 73
-   * - Records Added
-     - 14
-   * - schema_version
-     - 0800
-
-7. Downgrade to Honolulu (0800) after failed upgrade
-****************************************************
-
-Ensure you are on release 0800.
-
-Modify db_migrator_policy_init.sh - remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts"
-
-Update pdpstatistics:
-
-.. code::
-
-    ALTER TABLE pdpstatistics ADD COLUMN POLICYUNDEPLOYCOUNT BIGINT DEFAULT NULL NULL AFTER POLICYEXECUTEDSUCCESSCOUNT;
-
-Make/Redeploy to run upgrade
-
-This should result in an error (last row in policyadmin_schema_changelog will have a success value of 0)
-
-Once the retry count has been reached, update pdpstatistics:
-
-.. code::
-
-    ALTER TABLE pdpstatistics DROP COLUMN POLICYUNDEPLOYCOUNT;
-
-Modify db_migrator_policy_init.sh - Remove any lines referencing upgrade and add the 2 lines under "Downgrade scripts"
-
-Make/Redeploy to run downgrade
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 7
-   * - Tables in policyadmin
-     - 73
-   * - Records Added
-     - 7
-   * - schema_version
-     - 0800
-
-8. Upgrade to Istanbul (0900) after failed upgrade
-**************************************************
-
-Ensure you are on release 0800.
-
-Modify db_migrator_policy_init.sh - remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts"
-
-Update PDP table:
-
-.. code::
-
-    ALTER TABLE pdp ADD COLUMN LASTUPDATE datetime NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER HEALTHY;
-
-Make/Redeploy to run upgrade
-
-This should result in an error (last row in policyadmin_schema_changelog will have a success value of 0)
-
-Update PDP table:
-
-.. code::
-
-    ALTER TABLE pdp DROP COLUMN LASTUPDATE;
-
-The config job will retry 5 times. If you make your fix before this limit is reached you won't need to redeploy.
-
-Redeploy to run upgrade
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 14
-   * - Tables in policyadmin
-     - 75
-   * - Records Added
-     - 14
-   * - schema_version
-     - 0900
-
-9. Downgrade to Honolulu (0800) with data in pdpstatistics and jpapdpstatistics_enginestats
-*******************************************************************************************
-
-Ensure you are on release 0900.
-
-Check pdpstatistics and jpapdpstatistics_enginestats are populated with data.
-
-.. code::
-  :number-lines:
-
-    SELECT count(*) FROM pdpstatistics;
-    SELECT count(*) FROM jpapdpstatistics_enginestats;
-
-Modify db_migrator_policy_init.sh - remove any lines referencing upgrade and add the 2 lines under "Downgrade scripts"
-
-Make/Redeploy to run downgrade
-
-Check the tables to ensure the number of records is the same.
-
-.. code::
-  :number-lines:
-
-    SELECT count(*) FROM pdpstatistics;
-    SELECT count(*) FROM jpapdpstatistics_enginestats;
-
-Check pdpstatistics to ensure the primary key has changed:
-
-.. code::
-
-    SELECT column_name, constraint_name FROM information_schema.key_column_usage WHERE table_name='pdpstatistics';
-
-Check jpapdpstatistics_enginestats to ensure id column has been dropped and timestamp column added.
-
-.. code::
-
-    SELECT table_name, column_name, data_type FROM information_schema.columns WHERE table_name = 'jpapdpstatistics_enginestats';
-
-Check the pdp table to ensure the LASTUPDATE column has been dropped.
-
-.. code::
-
-    SELECT table_name, column_name, data_type FROM information_schema.columns WHERE table_name = 'pdp';
-
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 13
-   * - Tables in policyadmin
-     - 73
-   * - Records Added
-     - 13
-   * - schema_version
-     - 0800
-
-10. Upgrade to Istanbul (0900) with data in pdpstatistics and jpapdpstatistics_enginestats
-******************************************************************************************
-
-Ensure you are on release 0800.
-
-Check pdpstatistics and jpapdpstatistics_enginestats are populated with data.
-
-.. code::
-  :number-lines:
-
-    SELECT count(*) FROM pdpstatistics;
-    SELECT count(*) FROM jpapdpstatistics_enginestats;
-
-Modify db_migrator_policy_init.sh - remove any lines referencing downgrade and add the 2 lines under "Upgrade scripts"
-
-Make/Redeploy to run upgrade
-
-Check the tables to ensure the number of records is the same.
-
-.. code::
-  :number-lines:
-
-    SELECT count(*) FROM pdpstatistics;
-    SELECT count(*) FROM jpapdpstatistics_enginestats;
-
-Check pdpstatistics to ensure the primary key has changed:
-
-.. code::
-
-    SELECT column_name, constraint_name FROM information_schema.key_column_usage WHERE table_name='pdpstatistics';
-
-Check jpapdpstatistics_enginestats to ensure timestamp column has been dropped and id column added.
-
-.. code::
-
-    SELECT table_name, column_name, data_type FROM information_schema.columns WHERE table_name = 'jpapdpstatistics_enginestats';
-
-Check the pdp table to ensure the LASTUPDATE column has been added and the value has defaulted to the CURRENT_TIMESTAMP.
-
-.. code::
-
-    SELECT table_name, column_name, data_type, column_default FROM information_schema.columns WHERE table_name = 'pdp';
-
-.. list-table::
-   :widths: 60 20
-   :header-rows: 0
-
-   * - Number of files run
-     - 13
-   * - Tables in policyadmin
-     - 75
-   * - Records Added
-     - 13
-   * - schema_version
-     - 0900
-
-.. note::
-   The number of records added may vary depending on the number of retries.
-
-With addition of Postgres support to db-migrator, these tests can be also performed on a Postgres version of database.
-In addition, scripts running the aforementioned scenarios can be found under `smoke-tests` folder on db-migrator code base.
 
 End of Document

diff --git a/docs/development/devtools/smoke/pap-smoke.rst b/docs/development/devtools/smoke/pap-smoke.rst
index a5f54c0..a17c8c6 100644 (file)
--- a/docs/development/devtools/smoke/pap-smoke.rst
+++ b/docs/development/devtools/smoke/pap-smoke.rst
@@ -11,7 +11,8 @@ Policy PAP Smoke Test
 ~~~~~~~~~~~~~~~~~~~~~
 
 The policy-pap smoke testing is executed against a default ONAP installation as per OOM charts.
-This test verifies the execution of all the REST api's exposed by the component to make sure the contract works as expected.
+This test verifies the execution of all the REST api's exposed by the component to make sure the
+contract works as expected.
 
 General Setup
 *************
@@ -28,7 +29,7 @@ The ONAP components used during the smoke tests are:
 
 - Policy API to perform CRUD of policies.
 - Policy DB to store the policies.
-- DMaaP for the communication between components.
+- Kafka for the communication between components.
 - Policy PAP to perform runtime administration (deploy/undeploy/status/statistics/etc).
 - Policy Apex-PDP to deploy & undeploy policies. And send heartbeats to PAP.
 - Policy Drools-PDP to deploy & undeploy policies. And send heartbeats to PAP.
@@ -66,4 +67,5 @@ Make sure to execute the delete steps in order to clean the setup after testing.
 
 Delete policies using policy-api
 --------------------------------
-Use the previously downloaded policy-api postman collection to delete the policies created for testing.
+Use the previously downloaded policy-api postman collection to delete the policies created for
+testing.

diff --git a/docs/development/devtools/smoke/xacml-smoke.rst b/docs/development/devtools/smoke/xacml-smoke.rst
index 61f3551..b57a306 100644 (file)
--- a/docs/development/devtools/smoke/xacml-smoke.rst
+++ b/docs/development/devtools/smoke/xacml-smoke.rst
@@ -10,8 +10,8 @@
 XACML PDP Smoke Test
 ~~~~~~~~~~~~~~~~~~~~
 
-The policy-xacml-pdp smoke testing can be executed against a kubernetes based policy framework installation,
-and/or a docker-compose set up similar to the one executed by CSIT tests.
+The policy-xacml-pdp smoke testing can be executed against a kubernetes based policy framework
+installation, and/or a docker-compose set up similar to the one executed by CSIT tests.
 
 General Setup
 *************
@@ -21,16 +21,21 @@ PF kubernetes Install
 
 For installation instructions, please refer to the following documentation:
 
-`Policy Framework K8S Install <https://docs.onap.org/projects/onap-policy-parent/en/latest/development/devtools/testing/csit.html>`_
+`Policy Framework K8S Install
+<https://docs.onap.org/projects/onap-policy-parent/en/latest/development/devtools/testing/csit.html>`_
 
-The script referred to in the above link should handle the install of the of microk8s, docker and other required components for the install of the policy framework and clamp components. The scripts are used by policy as a means to run the CSIT tests in Kubernetes.
+The script referred to in the above link should handle the install of the of microk8s, docker and
+other required components for the install of the policy framework and clamp components. The scripts
+are used by policy as a means to run the CSIT tests in Kubernetes.
 
 docker-compose based
 --------------------
 
-A smaller testing environment can be put together by replicating the docker-based CSIT test environment. Details are on the same page as K8s setup:
+A smaller testing environment can be put together by replicating the docker-based CSIT test
+environment. Details are on the same page as K8s setup:
 
-`Policy CSIT Test Install Docker <https://docs.onap.org/projects/onap-policy-parent/en/latest/development/devtools/testing/csit.html>`_
+`Policy CSIT Test Install Docker
+<https://docs.onap.org/projects/onap-policy-parent/en/latest/development/devtools/testing/csit.html>`_
 
 Testing procedures
 ******************

diff --git a/docs/development/devtools/testing/csit.rst b/docs/development/devtools/testing/csit.rst
index f5cd8ea..9151e16 100644 (file)
--- a/docs/development/devtools/testing/csit.rst
+++ b/docs/development/devtools/testing/csit.rst
@@ -42,12 +42,13 @@ Under the folder `~/git/policy/docker/csit`, there are two main scripts to run t
 Running CSIT in Docker environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If not familiar with the PF Docker structure, the detailed information can be found :ref:`here <docker-label>`
+If not familiar with the PF Docker structure, the detailed information can be found
+:ref:`here <docker-label>`
 
 Running tests to validate code changes
 --------------------------------------
 
-For *local* images, set `LOCAL_IMAGES=true`, located at the `get-versions.sh` script
+For *local* images, run the script with the `--local` flag.
 
 .. note::
    Make sure to do the same changes to any other components that are using locally built images.
@@ -59,7 +60,7 @@ Then use the `run-project-csit.sh` script to run the test suite.
 .. code-block:: bash
 
   cd ~/git/policy/docker
-  ./csit/run-project-csit.sh <component>
+  ./csit/run-project-csit.sh <component> --local
 
 
 The <component> input is any of the policy components available:
@@ -72,6 +73,7 @@ The <component> input is any of the policy components available:
  - drools-applications
  - xacml-pdp
  - clamp
+ - opa-pdp
 
 Keep in mind that after the Robot executions, logs from docker-compose are printed and
 test logs might not be available on console and the containers are teared down. The tests results
@@ -82,12 +84,14 @@ Running tests for learning PF usage
 -----------------------------------
 
 In that case, no changes required on docker-compose files, but commenting the tear down of docker
-containers might be required. For that, edit the file `run-project-csit.sh` script and comment the
-following line:
+containers might be required. For that, run the `run-project-csit.sh` script with `--no-exit` flag:
 
 .. code-block:: bash
 
-  # source_safely ${WORKSPACE}/compose/stop-compose.sh (currently line 36)
+  cd ~/git/policy/docker
+  ./csit/run-project-csit.sh <component> --local --no-exit
+  # or
+  ./csit/run-project-csit.sh <component> --no-exit # will download images from nexus3 server
 
 
 This way, the docker containers are still up and running for more investigation.
@@ -130,6 +134,7 @@ The <component> input is any of the policy components available:
  - drools-pdp
  - xacml-pdp
  - clamp
+ - opa-pdp
 
 
 Different from Docker usage, the microk8s installation is not removed when tests finish.