Developing and Debugging each Policy Component
**********************************************
-Running a MariaDb Instance
-++++++++++++++++++++++++++
-
-The Policy Framework requires a MariaDb instance running. The easiest way to do this is to run a docker image locally.
-
-One example on how to do this is to use the scripts used by the policy/api S3P tests.
-
-`Simulator Setup Script Example <https://gerrit.onap.org/r/gitweb?p=policy/api.git;a=tree;f=testsuites/stability/src/main/resources/simulatorsetup;h=9038413f67cff2e2a79d6345f198f96ee0c57de1;hb=refs/heads/master>`_
-
- .. code-block:: bash
-
- cd ~/git/onap/api/testsuites/stability/src/main/resources/simulatorsetup
- ./setup_components.sh
-
-Another example on how to run the MariaDb is using the docker compose file used by the Policy API CSITs:
-
-`Example Compose Script to run MariaDB <https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=scripts/policy/docker-compose-api.yml;h=e32190f1e6cb6d9b64ddf53a2db2c746723a0c6a;hb=refs/heads/master>`_
Running the API component standalone
++++++++++++++++++++++++++++++++++++
Assuming you have successfully built the codebase using the instructions above. The only requirement for the API
-component to run is a running MariaDb database instance. The easiest way to do this is to run the docker image, please
-see the mariadb documentation for the latest information on doing so. Once the mariadb is up and running, a
-configuration file must be provided to the api in order for it to know how to connect to the mariadb. You can locate
-the default configuration file in the packaging of the api component:
+component to run is a running MariaDb/Postgres database instance. The easiest way to do this is to run the docker
+image, please see the official documentation for the latest information on doing so. Once the database is up and
+running, a configuration file must be provided to the api in order for it to know how to connect to the database.
+You can locate the default configuration file in the packaging of the api component:
-`Default Policy API Configuration <https://gerrit.onap.org/r/gitweb?p=policy/api.git;a=blob;f=packages/policy-api-tarball/src/main/resources/etc/apiParameters.yaml;h=2c19199a8a889cb0ab203334182662fe15e1635e;hb=refs/heads/master>`_
+`Default Policy API Configuration <https://github.com/onap/policy-api/blob/master/packages/policy-api-tarball/src/main/resources/etc/apiParameters.yaml>`_
You will want to change the fields pertaining to "host", "port" and "databaseUrl" to your local environment settings
and start the policy-api springboot application either using your IDE of choice or using the run goal from Spring Boot
An example of running the api using a docker compose script is located in the Policy Integration CSIT test repository.
-`Policy CSIT API Docker Compose <https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=scripts/policy/docker-compose-api.yml;h=e32190f1e6cb6d9b64ddf53a2db2c746723a0c6a;hb=refs/heads/master>`_
+`Policy CSIT Docker Compose <https://github.com/onap/policy-docker/blob/master/compose/compose.yaml>`_
Running the PAP component standalone
++++++++++++++++++++++++++++++++++++
-Once you have successfully built the PAP codebase, a running MariaDb database and DMaaP instance will also be required
-to start up the application. For MariaDb instance, the easiest way is to run the docker image, please see the mariadb
-documentation for the latest information on doing so. For DMaaP, the easiest way during development is to run the DMaaP
-simulator which is explained in the below sections. Once the mariadb and DMaaP are running, a configuration file must
-be provided to the PAP component in order for it to know how to connect to the mariadb and DMaaP along with other
-relevant configuration details. You can locate the default configuration file in the packaging of the PAP component:
+Once you have successfully built the PAP codebase, a running MariaDb/Postgres database and Kafka instance will also be
+required to start up the application. To start database and Kafka, check official documentation on how to run an
+instance of each. After database and Kafka are up and running, a configuration file must be provided to the PAP
+component in order for it to know how to connect to the database and Kafka along with other relevant configuration
+details. You can locate the default configuration file in the packaging of the PAP component:
-`Default PAP Configuration <https://gerrit.onap.org/r/gitweb?p=policy/pap.git;a=blob;f=packages/policy-pap-tarball/src/main/resources/etc/papParameters.yaml;h=06dd45f4946fd0a11ed8ef859f8fc5bcf409a3f0;hb=HEAD>`_
+`Default PAP Configuration <https://github.com/onap/policy-pap/blob/master/packages/policy-pap-tarball/src/main/resources/etc/papParameters.yaml>`_
-Update the fields related to MariaDB, DMaaP and the RestServer for the application as per your local environment settings.
-Then to start the application, just run the Spring Boot application using IDE or command line.
+Update the fields related to database, Kafka and the RestServer for the application as per your local environment
+settings. Then to start the application, just run the Spring Boot application using IDE or command line.
Running the Smoke Tests
Running the Stability/Performance Tests
***************************************
-The following links contain instructions on how to run the S3P Stability and Performance tests. These may be helpful to developers to become
-familiar with the Policy Framework components and test any local changes.
+The following links contain instructions on how to run the S3P Stability and Performance tests. These may be helpful
+to developers to become familiar with the Policy Framework components and test any local changes.
.. toctree::
:maxdepth: 2
Running the Pairwise Tests
**************************
-The following links contain instructions on how to run the pairwise tests. These may be helpful to developers check that
-the Policy Framework works in a full ONAP deployment.
+The following links contain instructions on how to run the pairwise tests. These may be helpful to developers check
+that the Policy Framework works in a full ONAP deployment.
.. toctree::
:maxdepth: 1
Running Policy Components Locally
*********************************
-The following page outlines how to run the policy framework components locally using IntelliJ, Eclipse and the Command Line.
+The following page outlines how to run the policy framework components locally using IntelliJ,
+Eclipse and the Command Line.
.. toctree::
:maxdepth: 1
- The generated swagger.json can be accessed at: *https://service_IP:service_port/v2/api-docs*
- Swagger UI can be accessed at: *https://service_IP:service_port/swagger-ui/index.html*
-Running the DMaaP Simulator during Development
-**********************************************
-It is sometimes convenient to run the DMaaP simulator during development. You can run it from the command line using Maven or from within your IDE.
-
-Running on the Command Line
-+++++++++++++++++++++++++++
-1. Check out the policy models repository
-2. Go to the *models-sim/policy-models-simulators* subdirectory in the policy-models repo
-3. Run the following Maven command:
-
- .. code-block:: bash
-
- mvn exec:java -Dexec.mainClass=org.onap.policy.models.simulators.Main -Dexec.args="src/test/resources/simParameters.json"
-
-Running in Eclipse
-++++++++++++++++++
-1. Check out the policy models repository
-2. Go to the *models-sim/policy-models-simulators* module in the policy-models repo
-3. Specify a run configuration using the class *org.onap.policy.models.simulators.Main* as the main class
-4. Specify an argument of *src/test/resources/simParameters.json* to the run configuration
-5. Run the configuration
-
-Specifying a local configuration file
-+++++++++++++++++++++++++++++++++++++
-
-You may specify a local configuration file instead of *src/test/resources/simParameters.json* on the command line or as an argument in the run configuration in eclipse:
-
-.. code-block:: json
-
- {
- "dmaapProvider": {
- "name": "DMaaP simulator",
- "topicSweepSec": 900
- },
- "restServers": [
- {
- "name": "DMaaP simulator",
- "providerClass": "org.onap.policy.models.sim.dmaap.rest.DmaapSimRestControllerV1",
- "host": "localhost",
- "port": 3904,
- "https": false
- }
- ]
- }
Bringing up Strimzi-Kafka Deploment with Policy Framework
*********************************************************
-This page will explain how to setup a local Kubernetes cluster and minimal helm setup to run and deploy Policy Framework on a single host.
+This page will explain how to setup a local Kubernetes cluster and minimal helm setup to run and deploy
+Policy Framework on a single host.
-This is meant for a development purpose only as we are going to use microk8s in this page
+This is meant for a development purpose only as we are going to use microk8s in this page:
.. toctree::
:maxdepth: 1
.. code-block:: bash
- docker pull onap/policy-pdpd-cl:1.8.2
+ docker pull onap/policy-pdpd-cl:3.0.0
-At the time of this writing *1.8.2* is the latest version.
+At the time of this writing *3.0.0* is the latest version.
The *onap/policy-pdpd-cl* image extends the *onap/policy-drools* image with
the *usecases* controller that realizes the *control loop* application.
- **controlloop-management**: generic controller capabilities.
- **controlloop-usecases**: new *controller* introduced in the guilin release to realize the ONAP use cases.
-The following features are installed but disabled:
-
-- **controlloop-tdjam**: experimental java-only *controller* to be deprecated post guilin.
-- **controlloop-utils**: *actor* simulators.
Control Loops Transaction (controlloop-trans)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It relies on the new *Actor* model framework to carry out a policy's
execution.
-TDJAM Controller (controlloop-tdjam)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This is an experimental, java-only controller that will be deprecated after the
-guilin release.
Utilities (controlloop-utils)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DCAE_SERVERS=localhost
DCAE_CONSUMER_GROUP=dcae.policy.shared
- # Open DMaaP
-
- DMAAP_SERVERS=localhost
-
# AAI
AAI_HOST=localhost
Configuration
~~~~~~~~~~~~~
-noop.pre.sh
-"""""""""""
-
-In order to avoid the noise in the logs that relate to dmaap configuration, a startup script (*noop.pre.sh*) is added
-to convert *dmaap* endpoints to *noop* in the host directory to be mounted.
-
-.. code-block:: bash
-
- #!/bin/bash -x
-
- sed -i "s/^dmaap/noop/g" $POLICY_HOME/config/*.properties
-
features.pre.sh
"""""""""""""""
More specifically, these directories have examples of other PDP-D Control Loop configurations:
* `plans <https://github.com/onap/policy-docker/tree/master/compose>`__: startup & teardown scripts.
-* `scripts <https://github.com/onap/policy-docker/blob/master/compose>`__: docker-compose files.
+* `scripts <https://github.com/onap/policy-docker/blob/master/compose/compose.yaml>`__: docker-compose file.
* `tests <https://github.com/onap/policy-docker/blob/master/csit/resources/tests/drools-applications-test.robot>`__: test plan.
Additional information
~~~~~~~~~~~~
Check the *drools-pdp* `released versions <https://wiki.onap.org/display/DW/Policy+Framework+Project%3A+Component+Versions>`__ page for the latest versions.
-At the time of this writing *1.8.2* is the latest version.
+At the time of this writing *3.0.0* is the latest version.
.. code-block:: bash
- docker pull onap/policy-drools:1.8.2
+ docker pull onap/policy-drools:3.0.0
A container instantiated from this image will run under the non-priviledged *policy* account.
.. code-block:: bash
- docker run --rm -it nexus3.onap.org:10001/onap/policy-drools:1.8.2 -- bash
+ docker run --rm -it nexus3.onap.org:10001/onap/policy-drools:3.0.0 -- bash
Communication Endpoints
=======================
# of pdp-d that are necessary for <feature-name> to operate
# correctly.
# lib/feature the single feature jar that implements the feature.
- # [db] database directory, if the feature contains sql.
- # [db]/<db-name> database to which underlying sql scripts should be applied.
- # ideally, <db-name> = <feature-name> so it is easily to associate
- # the db data with a feature itself. In addition, since a feature is
- # a somewhat independent isolated unit of functionality,the <db-name>
- # database ideally isolates all its data.
- # [db]/<db-name>/sql directory with all the sql scripts.
- # [db]/<db-name>/sql/<sql-scripts> for this feature, sql
- # upgrade scripts should be suffixed with ".upgrade.sql"
- # and downgrade scripts should be suffixed with ".downgrade.sql"
# [artifacts] maven artifacts to be deployed in a maven repository.
# [artifacts]/<artifact> maven artifact with identifiable maven coordinates embedded
# in the artifact.
`feature-distributed-locking sql directory <https://git.onap.org/policy/docker/tree/policy-db-migrator/src/main/docker/config/pooling/sql>`__
for an example of upgrade/downgrade scripts.
-The following command will provide a report on the upgrade or downgrade activies:
-
-.. code-block:: bash
-
- db-migrator -s ALL -o report
-
-For example in the official guilin delivery:
-
-.. code-block:: bash
-
- policy@dev-drools-0:/tmp/policy-install$ db-migrator -s ALL -o report
- +---------+---------+
- | name | version |
- +---------+---------+
- | pooling | 1811 |
- +---------+---------+
- +-------------------------------------+-----------+---------+---------------------+
- | script | operation | success | atTime |
- +-------------------------------------+-----------+---------+---------------------+
- | 1804-distributedlocking.upgrade.sql | upgrade | 1 | 2020-05-22 19:33:09 |
- | 1811-distributedlocking.upgrade.sql | upgrade | 1 | 2020-05-22 19:33:09 |
- +-------------------------------------+-----------+---------+---------------------+
-
-In order to use the *db-migrator* tool, the system must be configured with a database.
-
-.. code-block:: bash
-
- SQL_HOST=mariadb
Maven Repositories
==================
The drools libraries in the PDP-D uses maven to fetch rules artifacts and software dependencies.
-The default *settings.xml* file specifies the repositories to search. This configuration
+The default *settings.xml* file specifies the repositories to search. This configuration
can be overriden with a custom copy that would sit in a mounted configuration
directory. See an example of the OOM override
`settings.xml <https://github.com/onap/oom/blob/master/kubernetes/policy/components/policy-drools-pdp/resources/configmaps/settings.xml>`_.
-s|--settings: custom settings.xml
-a|--artifact: file artifact (jar or pom) to deploy and/or install
-AAF
-===
-
-Policy can talk to AAF for authorization requests. To enable AAF set
-the following environment variables:
-
-.. code-block:: bash
-
- AAF=true
- AAF_NAMESPACE=org.onap.policy
- AAF_HOST=aaf-locate.onap
-
-By default AAF is disabled.
Policy Tool
===========
- *PDP-D* running status
- *features* applied
-- Data migration status on a per database basis.
The *start* and *stop* commands are useful for developers testing functionality on a docker container instance.
Telemetry Shell
===============
-*PDP-D* offers an ample set of REST APIs to debug, introspect, and change state on a running PDP-D. This is known as the
-*telemetry* API. The *telemetry* shell wraps these APIs for shell-like access using
+*PDP-D* offers an ample set of REST APIs to debug, introspect, and change state on a running PDP-D. This is known as the
+*telemetry* API. The *telemetry* shell wraps these APIs for shell-like access using
`http-prompt <http://http-prompt.com/>`__.
.. code-block:: bash
- ***.post.sh** scripts that will be executed after the PDP-D starts.
- **policy-keystore** to override the default PDP-D java keystore.
- **policy-truststore** to override the default PDP-D java truststore.
-- **aaf-cadi.keyfile** to override the default AAF CADI Key generated by AAF.
- ***.properties** to override or add any properties file for the PDP-D, this includes *controller*, *endpoint*,
*engine* or *system* configurations.
- **logback*.xml** to override the default logging configuration.
POLICY_PDP_PAP_API_KEY=
POLICY_PDP_PAP_API_SECRET=
- # DMaaP
-
- DMAAP_SERVERS=localhost
Note that *SQL_HOST*, and *REPOSITORY* are empty, so the PDP-D does not attempt
to integrate with those components.
Configuration
~~~~~~~~~~~~~
-In order to avoid the noise in the logs that relate to dmaap configuration, a startup script (*noop.pre.sh*) is added
-to convert *dmaap* endpoints to *noop* in the host directory to be mounted.
-
-noop.pre.sh
-"""""""""""
-
-.. code-block:: bash
-
- #!/bin/bash -x
-
- sed -i "s/^dmaap/noop/g" $POLICY_HOME/config/*.properties
-
active.post.sh
""""""""""""""
docker exec -it PDPD bash -c "/opt/app/policy/bin/telemetry"
docker exec -it PDPD bash -c "/opt/app/policy/bin/policy status"
- docker exec -it PDPD bash -c "/opt/app/policy/bin/db-migrator -s ALL -o report"
Controlled instantiation of the PDP-D
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
POLICY_PDP_PAP_API_KEY=
POLICY_PDP_PAP_API_SECRET=
- # DMaaP
-
- DMAAP_SERVERS=localhost
-
-prepare.pre.sh
-~~~~~~~~~~~~~~
-
-A pre-start script *config/prepare.pres.sh"can be added the custom config directory
-to prepare the PDP-D to activate the distributed-locking feature (using the database)
-and to use "noop" topics instead of *dmaap* topics:
-
-.. code-block:: bash
-
- #!/bin/bash
-
- bash -c "/opt/app/policy/bin/features enable distributed-locking"
- sed -i "s/^dmaap/noop/g" $POLICY_HOME/config/*.properties
active.post.sh
~~~~~~~~~~~~~~
More specifically, these directories have examples of other PDP-D configurations:
* `plans <https://github.com/onap/policy-docker/tree/master/compose>`__: startup & teardown scripts.
-* `scripts <https://github.com/onap/policy-docker/blob/master/compose>`__: docker-compose files.
+* `scripts <https://github.com/onap/policy-docker/blob/master/compose/compose.yaml>`__: docker-compose file.
* `tests <https://github.com/onap/policy-docker/blob/master/csit/resources/tests/drools-pdp-test.robot>`__: test plan.
Configuring the PDP-D in an OOM Kubernetes installation
**Step 1:** Set the containers location and project.
-For *local* images, set CONTAINER_LOCATION="", located at the `export-ports.sh` script
+For *local* images, set `LOCAL_IMAGES=true`, located at the `get-versions.sh` script
*You will need to build locally all the images using the steps in the previous section*
-For *remote* images set CONTAINER_LOCATION="nexus3.onap.org:10001/"
+Remote images are downloaded by default from "nexus3.onap.org:10001"
**Step 2:** Set gerrit branch
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.local.only=false \
-On `docker-compose.yml` compose file, add to the port section the mapping 5005.
+On `compose.yml` file, add to the port section the mapping 5005.
.. code-block:: yaml
image: ${CONTAINER_LOCATION}onap/policy-pap:${POLICY_PAP_VERSION}
container_name: policy-pap
depends_on:
- - mariadb
- simulator
- api
hostname: policy-pap
.. contents::
:depth: 3
-This page describes the API between the PAP and PDPs. The APIs in this section are implemented using `DMaaP
-API <https://wiki.onap.org/display/DW/DMaaP+API>`__ messaging. The APIs in this section are used for internal
-communication in the Policy Framework. The APIs are NOT supported for use by components outside the Policy Framework and
-are subject to revision and change at any time.
+This page describes the API between the PAP and PDPs. The APIs in this section are used for internal
+communication in the Policy Framework, using Kafka as messaging system. The APIs are NOT supported for
+use by components outside the Policy Framework and are subject to revision and change at any time.
There are three messages on the API:
1. PDP_STATUS: PDP→PAP, used by PDPs to report to the PAP
-2. PDP_UPDATE: PAP→PDP, used by the PAP to update the policies running on PDPs, triggers a PDP_STATUS message with
- the result of the PDP_UPDATE operation
+2. PDP_UPDATE: PAP→PDP, used by the PAP to update the policies running on PDPs, triggers a PDP_STATUS
+ message with the result of the PDP_UPDATE operation
-3. PDP_STATE_CHANGE: PAP→PDP, used by the PAP to change the state of PDPs, triggers a PDP_STATUS message with the result
- of the PDP_STATE_CHANGE operation
+3. PDP_STATE_CHANGE: PAP→PDP, used by the PAP to change the state of PDPs, triggers a PDP_STATUS message
+ with the result of the PDP_STATE_CHANGE operation
The fields in the table below are valid on API calls:
->(name) O M N/A The name of a TOSCA policy running on the PDP
->policy_type O M N/A The TOSCA policy type of the policyWhen a PDP starts,
it commences periodic sending of *PDP_STATUS*
- messages on DMaaP. The PAP receives these messages
+ messages. The PAP receives these messages
and acts in whatever manner is appropriate.
->policy_type_version O M N/A The version of the TOSCA policy type of the policy
->properties O M N/A The properties of the policy for the XACML, Drools,
==================
The purpose of this API is for PDPs to provide heartbeat, status, health, and statistical information to Policy
Administration. There is a single *PDP_STATUS* message on this API. PDPs send this message to the PAP using the
-*POLICY_PDP_PAP* DMaaP topic. The PAP listens on this topic for messages.
+*POLICY_PDP_PAP* topic. The PAP listens on this topic for messages.
-When a PDP starts, it commences periodic sending of *PDP_STATUS* messages on DMaaP. The PAP receives these messages and
+When a PDP starts, it commences periodic sending of *PDP_STATUS* messages. The PAP receives these messages and
acts in whatever manner is appropriate. *PDP_UPDATE* and *PDP_STATE_CHANGE* operations trigger a
*PDP_STATUS* message as a response.
name: xacml-23d33c2a-8715-43a8-ade5-5923fc0f185c
pdpGroup: defaultGroup
pdpSubgroup: xacml
- statistics:
- policyDeployCount: 0
- policyDeploySuccessCount: 0
- policyDeployFailCount: 0
- policyExecutedCount: 123
- policyExecutedSuccessCount: 122
- policyExecutedFailCount: 1
.. code-block:: yaml
deployment_instance_info:
node_address: drools_2_pod
# Other deployment instance info
- statistics:
- policyDeployCount: 3
- policyDeploySuccessCount: 3
- policyDeployFailCount: 0
- policyExecutedCount: 123
- policyExecutedSuccessCount: 122
- policyExecutedFailCount: 1
- policyUndeployCount: 0
- policyUndeploySuccessCount: 0
- policyUndeployFailCount: 0
response:
responseTo: 52117e25-f416-45c7-a955-83ed929d557f
responseStatus: SUCCESSSS
policies:
- name: onap.controllloop.operational.apex.bbs.EastRegion
version: 1.0.0
- statistics:
- policyExecutedCount: 0
- policyExecutedSuccessCount: 0
- policyExecutedFailCount: 0
- policyDeployCount: 1
- policyDeploySuccessCount: 1
- policyDeployFailCount: 0
- policyUndeployCount: 0
- policyUndeploySuccessCount: 0
- policyUndeployFailCount: 0
response:
responseTo: 679fad9b-abbf-4b9b-971c-96a8372ec8af
responseStatus: SUCCESS
name: xacml-23d33c2a-8715-43a8-ade5-5923fc0f185c
pdpGroup: onap.pdpgroup.Monitoring
pdpSubgroup: xacml
- statistics:
- policyDeployCount: 0
- policyDeploySuccessCount: 0
- policyDeployFailCount: 0
- policyExecutedCount: 123
- policyExecutedSuccessCount: 122
- policyExecutedFailCount: 1
2 PDP API for PAPs
==================
The purpose of this API is for the PAP to load and update policies on PDPs and to change the state of PDPs.
-The PAP sends *PDP_UPDATE* and *PDP_STATE_CHANGE* messages to PDPs using the *POLICY_PAP_PDP* DMaaP topic.
+The PAP sends *PDP_UPDATE* and *PDP_STATE_CHANGE* messages to PDPs using the *POLICY_PAP_PDP* topic.
PDPs listen on this topic for messages.
The PAP can set the scope of *PDP_STATE_CHANGE* message:
- upgrade/downgrade/report
* - schema
- -s
- - policyadmin/clampacm
+ - schema_name
* - to
- -t
- - 0800/0900
+ - target_version (latest is 1600)
* - from
- -f
- - 0800/0900
+ - origin_version (earliest is 0800)
The container also consists of several sql files which are used to upgrade/downgrade
the policy database.
* - SQL_HOST
- mariadb
* - SQL_DB
- - policyadmin/clampacm
+ - policyadmin clampacm etc [one or more schema separated by space]
* - SQL_USER
- policy_user
* - SQL_PASSWORD
* - SQL_HOST
- postgres
* - SQL_DB
- - policyadmin/clampacm
+ - policyadmin clampacm etc [one or more schema separated by space]
* - SQL_USER
- policy_user
* - SQL_PASSWORD
.. note::
It is advisable to take a backup of your database prior to running this utility.
- Please refer to the mariadb documentation on how to do this.
+ Please refer to the mariadb or postgres documentation on how to do this.
End of Document
Code Review / policy / parent.git / commitdiff