Merge "Added validation on matching required fields"
[policy/engine.git] / docs / platform / swarch_srm.rst
1
2 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
4
5 *************
6 Policy on OOM
7 *************
8
9 .. contents::
10     :depth: 3
11
12 The new Beijing release capabilities for OOM are described here.
13
14 Installation
15 ^^^^^^^^^^^^
16 Follow the OOM installation instructions at http://onap.readthedocs.io/en/latest/submodules/oom.git/docs/index.html
17
18 Overview of the running system
19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
20
21 Upon initialization, you should see the following pods, one instance for each Policy component: PAP, PDP-X, BRMSGW, PDP-D, policydb, and nexus.
22
23 Note the "-0" suffix for PDP-X and PDP-D components, which will be increased as they are scaled out to improve runtime performance and reliability.
24
25 .. code-block:: bash
26    :caption: verify pods
27
28     kubectl get pods --all-namespaces -o=wid
29      
30     onap    dev-brmsgw-5dbc4c8dc4-llk5s        1/1       Running   0     18m     10.42.120.43    k8sx
31     onap    dev-drools-0                       1/1       Running   0     18m     10.42.60.27     k8sx
32     onap    dev-nexus-7d96568f5f-qp5td         1/1       Running   0     18m     10.42.172.8     k8sx
33     onap    dev-pap-8587696769-vwj6k           2/2       Running   0     18m     10.42.19.137    k8sx
34     onap    dev-pdp-0                          2/2       Running   0     18m     10.42.144.218   k8sx
35     onap    dev-policydb-587d55bdff-4f5dz      1/1       Running   0     18m     10.42.12.242    k8sx
36
37
38 You will also see a service for every component:
39
40 .. code-block:: bash
41    :caption: verify services
42
43     kubectl get services --all-namespaces
44      
45     onap    brmsgw         NodePort    10.43.209.173   <none>     9989:30216/TCP                  24m
46     onap    drools         NodePort    10.43.27.92     <none>     6969:30217/TCP,9696:30221/TCP   24m
47     onap    nexus          NodePort    10.43.19.171    <none>     8081:30236/TCP                  24m
48     onap    pap            NodePort    10.43.9.166     <none>     8443:30219/TCP,9091:30218/TCP   24m
49     onap    pdp            ClusterIP   None            <none>     8081/TCP                        24m
50     onap    policydb       ClusterIP   None            <none>     3306/TCP                        24m
51
52 Config and Decision policy requests will be distributed across PDP-Xs through the *pdp* service.    PDP-X clients (such as DCAE) should configure their URLs to go through the *pdp* service.   Their requests will be distributed across the available PDP-X replicas.    
53 The PDP-Xs can be also accessed individually (dev-pdp-0, or dev-pdp-x if scaled out), but is preferable for PDP-X external clients to interface through the service.
54
55 PDP-Ds are also accessible on a group fashion by using the service IP, but DMaaP is the main means of communication with other ONAP components.
56
57
58 Healthchecks
59 ^^^^^^^^^^^^
60
61 Verify that the policy healthcheck passes by the robot framework:
62
63 .. code-block:: bash
64    :caption: robot healthcheck
65
66     ~/oom/kubernetes/robot/ete-k8s.sh onap health 2> /dev/null | grep PASS
67     ..
68     Basic Policy Health Check                                             | PASS |
69     ..
70
71
72 A policy healthcheck (with more detailed output) can be done directly by invoking the drools service in the policy VM.
73
74 .. code-block:: none
75    :caption: PDP-D service (more detailed) healthcheck
76
77     # Using default credentials for the healtcheck service.
78     # To change the default username and passwords for this service,
79     # please modify configuration pre-installation at:
80     # oom/kubernetes/policy/charts/drools/resources/config/opt/policy/config/drools/keys/feature-healthcheck.conf
81      
82     curl --silent --user 'healthcheck:zb!XztG34' -X GET http://localhost:30217/healthcheck | python -m json.tool
83     {
84         "details": [
85             {
86                 "code": 200,
87                 "healthy": true,
88                 "message": "alive",
89                 "name": "PDP-D",
90                 "url": "self"
91             },
92             {
93                 "code": 200,
94                 "healthy": true,
95                 "message": "",
96                 "name": "PAP",
97                 "url": "http://pap:9091/pap/test"
98             },
99             {
100                 "code": 200,
101                 "healthy": true,
102                 "message": "",
103                 "name": "PDP",
104                 "url": "http://pdp:8081/pdp/test"
105             }
106         ],
107         "healthy": true
108     }
109     
110
111 PDP-X Active/Active Pool
112 ^^^^^^^^^^^^^^^^^^^^^^^^
113
114 The policy engine UI (console container in the pap pod) can be used to check that the PAP and the PDP-Xs are synchronized.
115 The console URL is accessible at  ``http://<oom-vm>:30219/onap/login.htm``.   Select the *PDP* menu entry on the left side panel under *Policy*.
116
117     .. image:: srmPdpxPdpMgmt.png
118
119 After initialization, there will be no policies loaded into the policy subsystem.    This can be verified by accessing the Editor tab in the UI.
120
121
122 PDP-D Active/Active Pool
123 ^^^^^^^^^^^^^^^^^^^^^^^^
124
125 The PDP-Ds replicas will come up with the amsterdam controller installed in brainless mode (no maven coordinates) since the controller has not been associated with a set of drools rules to run (control loop rules).
126
127 The following command can be issued on each of the PDP-D replicas IPs:
128
129 .. code-block:: bash
130    :caption: Querying the rules association for a PDP-D replica 
131
132     # Using default credentials for the drools telemetry service.
133     # To change the default username and passwords for this service,
134     # please modify configuration pre-installation at:
135     # oom/kubernetes/policy/charts/drools/resources/config/opt/policy/config/drools/base.conf
136      
137      
138     curl --silent --user '@1b3rt:31nst31n' -X GET http://<drools-replica-ip>:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
139     {
140         "alive": false,
141         "artifactId": "NO-ARTIFACT-ID",
142         "brained": false,
143         "canonicalSessionNames": [],
144         "container": null,
145         "groupId": "NO-GROUP-ID",
146         "locked": false,
147         "recentSinkEvents": [],
148         "recentSourceEvents": [],
149         "sessionNames": [],
150         "version": "NO-VERSION"
151     }
152
153
154
155 Before Installing Policies
156 ^^^^^^^^^^^^^^^^^^^^^^^^^^
157
158 It has been experienced in large OOM k8s multi-node full ONAP installations that components DNS and connectivity problems across pods through services.   Eventually, the system becomes stable and ready to be used.   Single node, smaller installations, do not seem to have these issues.   Give the system enough time to make sure it has been initialized properly before pushing policies.
159
160 Make sure the policy subsystem is initialized by:
161
162     1. Verify that the "PDP Management" screen shows the 1 pooled PDP-X  "UP_TO_DATE".   If the PDP-X does not show the correct state, restart the faulty one to force re-synchronization with the pap.
163
164     .. code-block:: bash
165        :caption: Force re-synchronization of a PDP-X
166     
167         kubectl exec -it dev-pdp-0 --container pdp -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy.sh stop; policy.sh start"
168          
169         # bounce the BRMSGW as well since it synchronizes with PDP-Xs via websockets:
170          
171         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy.sh stop; policy.sh start"
172
173
174     2. Verify service name resolution is OK across policy components
175
176     .. code-block:: bash
177        :caption: Verify policy services connectivity
178
179         # pick any policy pod to run these tests from:
180         # kubectl get pods --all-namespaces -o=wide
181          
182         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "ping policydb"
183         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "ping pdp"
184         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "ping drools"
185         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "ping nexus"
186         kubectl exec -it dev-brmsgw-b877bc567-wbnbz -n onap -- bash -c "ping message-router"
187
188
189 Installing Policies
190 ^^^^^^^^^^^^^^^^^^^
191
192 The OOM default installation will come with no policies pre-configured.  There is a sample script used by integration teams to load policies to support all four use cases at:   */tmp/policy-install/config/push-policies.sh* in the pap container within the pap pod.   This script can be modified for your own particular installation, for example if only interested in vCPE use cases, remove those vCPE related API REST calls.   For the vFW use case, you may want to edit the encoded operational policy to point to the proper resourceID in your installation.
193
194 The above mentioned *push-policies.sh* script can be executed as follows:
195
196 .. code-block:: bash
197    :caption: Installing the default policies
198
199     # NOTE: If modifications are required to the /tmp/policy-install/config/push-policies.sh, it should be copied 
200     # to a different location, for example /tmp as /tmp/policy-install/config directory is read-only.
201      
202     kubectl exec -it dev-pap-8587696769-vwj6k -c pap -n onap -- bash -c "export PRELOAD_POLICIES=true; /tmp/policy-install/config/push-policies.sh"
203      
204     ..
205     Create BRMSParam Operational Policies
206     ..
207     Create BRMSParamvFirewall Policy
208     ..
209     Transaction ID: ef08cc65-9950-4478-a4ab-0f3bc2519f60 --Policy with the name com.Config_BRMS_Param_BRMSParamvFirewall.1.xml was successfully created.Create BRMSParamvDNS Policy
210     ..
211     Transaction ID: 52e33efe-ba66-47de-b404-8d441107d8a9 --Policy with the name com.Config_BRMS_Param_BRMSParamvDNS.1.xml was successfully created.Create BRMSParamVOLTE Policy
212     ..
213     Transaction ID: f13072b7-6258-4c16-99da-f908d29363ec --Policy with the name com.Config_BRMS_Param_BRMSParamVOLTE.1.xml was successfully created.Create BRMSParamvCPE Policy
214     ..
215     Transaction ID: 616f970a-b45e-40f7-88cd-d63000d22cca --Policy with the name com.Config_BRMS_Param_BRMSParamvCPE.1.xml was successfully created.Create MicroService Config Policies
216     Create MicroServicevFirewall Policy
217     ..
218     Transaction ID: 4c143a15-20af-408a-9285-bc7940261829 --Policy with the name com.Config_MS_MicroServicevFirewall.1.xml was successfully created.Create MicroServicevDNS Policy
219     ..
220     Transaction ID: 1e54ae73-509b-490e-bf62-1fea7989fd5f --Policy with the name com.Config_MS_MicroServicevDNS.1.xml was successfully created.Create MicroServicevCPE Policy
221     ..
222     Transaction ID: 32239868-bab2-4e12-9fd9-81a0ed4a6b1c --Policy with the name com.Config_MS_MicroServicevCPE.1.xml was successfully created.Creating Decision Guard policy
223     ..
224     Transaction ID: b43cb9d5-42c7-4654-aacf-d4898c4d13bb --Policy with the name com.Decision_AllPermitGuard.1.xml was successfully created.Push Decision policy
225     ..
226     Transaction ID: 3c1e4ae6-6991-415b-9f2d-c665a8c5a026 --Policy 'com.Decision_AllPermitGuard.1.xml' was successfully pushed to the PDP group 'default'.Pushing BRMSParam Operational policies
227     ..
228     Transaction ID: 58d26d03-b5b8-4fd3-b2df-1411a1c36420 --Policy 'com.Config_BRMS_Param_BRMSParamvFirewall.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamvDNS
229     ..
230     Transaction ID: 0854e54a-504b-4f06-bc2f-30f491cb9f5a --Policy 'com.Config_BRMS_Param_BRMSParamvDNS.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamVOLTE
231     ..
232     Transaction ID: d33c7dde-5c99-4dab-b4ff-9988473cd88d --Policy 'com.Config_BRMS_Param_BRMSParamVOLTE.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.BRMSParamvCPE
233     ..
234     Transaction ID: e8c8a73e-127c-4318-9e59-3cae9dcbe011 --Policy 'com.Config_BRMS_Param_BRMSParamvCPE.1.xml' was successfully pushed to the PDP group 'default'.Pushing MicroService Config policies
235     ..
236     Transaction ID: ec0429d7-e35f-4978-8a6c-40d2b5b3be61 --Policy 'com.Config_MS_MicroServicevFirewall.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.MicroServicevDNS
237     ..
238     Transaction ID: f7072f05-7b74-45b5-9bd3-99b7f8023e3e --Policy 'com.Config_MS_MicroServicevDNS.1.xml' was successfully pushed to the PDP group 'default'.pushPolicy : PUT : com.MicroServicevCPE
239     ..
240     Transaction ID: 6d47db63-7956-4f5f-ab34-aeb5a124a90d --Policy 'com.Config_MS_MicroServicevCPE.1.xml' was successfully pushed to the PDP group 'default'.
241
242 The policies pushed could be viewed eventually through the Policy UI:
243
244     .. image:: srmEditor.png
245
246 As part of the process pushing of policies through the policy, the brmsgw component will compose drools rules artifacts and publish them to the nexus respository at ``http://<oom-vm>:30236/nexus/``.
247
248     .. image:: srmNexus.png
249
250 At the same time each replica of the PDP-Ds will receive notifications for each new version of the policies to run for the Amsterdam controller.   The following command can be run to see how the amsterdam controller is associated with the latest rules version. 
251
252 The following command can be used for verifying each replica:
253
254
255 .. code-block:: none
256    :caption: Querying the rules association of a PDP-D replica
257
258     # Using default credentials for the drools telemetry service.
259     # To change the default username and passwords for this service,
260     # please modify configuration pre-installation at:
261     # oom/kubernetes/policy/charts/drools/resources/config/opt/policy/config/drools/base.conf
262      
263     curl --silent --user '@1b3rt:31nst31n' -X GET http://<replica-ip>:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
264
265     {
266         "alive": true,
267         "artifactId": "policy-amsterdam-rules",
268         "brained": true,
269         "groupId": "org.onap.policy-engine.drools.amsterdam",
270         "locked": false,
271         "modelClassLoaderHash": 1223551265,
272         "recentSinkEvents": [],
273         "recentSourceEvents": [],
274         "sessionCoordinates": [
275             "org.onap.policy-engine.drools.amsterdam:policy-amsterdam-rules:0.4.0:closedloop-amsterdam"
276         ],
277         "sessions": [
278             "closedloop-amsterdam"
279         ],
280         "version": "0.4.0"
281     }
282
283 Likewise, for verification purposes, each PDP-X replica can be queried directly to retrieve policy information.   
284
285 The following commands can be used to query a policy through the pdp service:
286
287
288 .. code-block:: bash
289    :caption: Querying the "pdp" service for the vFirewal policy
290
291     # Open a shell into the pap pod
292
293     ubuntu@k8sx:~$ kubectl exec -it dev-pap-8587696769-vwj6k -c pap -n onap bash
294
295     # In this example the vFirewall policy is queried.
296
297     policy@dev-pap-8587696769-vwj6k:/tmp/policy-install$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vFirewall.*"}' http://pdp:8081/pdp/api/getConfig | python -m json.tool
298     [
299         {
300             "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevFirewall\",\"description\":\"MicroService vFirewall Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"vFirewallBroadcastPackets\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"LESS_OR_EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ONSET\"},{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":700,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
301             "matchingConditions": {
302                 "ConfigName": "SampleConfigName",
303                 "Location": "SampleServiceLocation",
304                 "ONAPName": "DCAE",
305                 "service": "tca_policy",
306                 "uuid": "test"
307             },
308             "policyConfigMessage": "Config Retrieved! ",
309             "policyConfigStatus": "CONFIG_RETRIEVED",
310             "policyName": "com.Config_MS_MicroServicevFirewall.1.xml",
311             "policyType": "MicroService",
312             "policyVersion": "1",
313             "property": null,
314             "responseAttributes": {},
315             "type": "JSON"
316         },
317         {
318             "config":  ..... 
319             "matchingConditions": {
320                 "ConfigName": "BRMS_PARAM_RULE",
321                 "ONAPName": "DROOLS"
322             },
323             "policyConfigMessage": "Config Retrieved! ",
324             "policyConfigStatus": "CONFIG_RETRIEVED",
325             "policyName": "com.Config_BRMS_Param_BRMSParamvFirewall.1.xml",
326             "policyType": "BRMS_PARAM",
327             "policyVersion": "1",
328             "property": null,
329             "responseAttributes": {
330                 "controller": "amsterdam"
331             },
332             "type": "OTHER"
333         }
334     ]
335     
336
337 While the following commands could be used to query an specific PDP-X replica:
338
339
340 .. code-block:: bash
341    :caption: Querying PDP-X 0 for the vCPE policy
342
343     # open a shell into the pap pod
344  
345     ubuntu@k8sx:~$ kubectl exec -it dev-pap-8587696769-vwj6k -c pap -n onap bash
346  
347     # in this example the vCPE policy is queried.
348
349     curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.144.218:8081/pdp/api/getConfig | python -m json.tool
350     [
351         {
352             "config": ...,
353             "matchingConditions": {
354                 "ConfigName": "BRMS_PARAM_RULE",
355                 "ONAPName": "DROOLS"
356             },
357             "policyConfigMessage": "Config Retrieved! ",
358             "policyConfigStatus": "CONFIG_RETRIEVED",
359             "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
360             "policyType": "BRMS_PARAM",
361             "policyVersion": "1",
362             "property": null,
363             "responseAttributes": {
364                 "controller": "amsterdam"
365             },
366             "type": "OTHER"
367         },
368         {
369             "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
370             "matchingConditions": {
371                 "ConfigName": "SampleConfigName",
372                 "Location": "SampleServiceLocation",
373                 "ONAPName": "DCAE",
374                 "service": "tca_policy",
375                 "uuid": "test"
376             },
377             "policyConfigMessage": "Config Retrieved! ",
378             "policyConfigStatus": "CONFIG_RETRIEVED",
379             "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
380             "policyType": "MicroService",
381             "policyVersion": "1",
382             "property": null,
383             "responseAttributes": {},
384             "type": "JSON"
385         }
386     ]
387     
388 PDP-X Resiliency
389 ^^^^^^^^^^^^^^^^
390
391 A PDP-X container failure can be simulated by either:
392    a) performing a"policy.sh stop" operation within the PDP-X container, which in fact will shutdown the PDP-X service, and eventually will be detected by the liveness checks, or 
393    b) by plainly deleting the corresponding pod.    
394
395 In the following example, the PDP-X 0 is forced to fail.
396
397 .. code-block:: bash
398    :caption: Causing PDP-X 0 service to fail
399
400     # In these scenarios the liveness check will fail and recovery actions will take place.
401      
402      
403     # Alternative 1: In this scenario we shutdown the PDP-X 0 service, so the liveness monitored ports will be down 
404     # (but the pod is up) and corrective measures will be applied
405     ubuntu@k8sx:~$ kubectl exec -it dev-pdp-0 --container pdp -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy.sh stop;"
406         pdplp: STOPPING ..
407         pdp: STOPPING ..
408      
409     # Alternative 2: Brute force delete of the PDP-X 0 pod.
410     ubuntu@k8sx:~$ kubectl delete pod dev-pdp-0 -n onap
411         pod "dev-pdp-0" deleted
412
413 Upon detection of the service being down through the liveness check, the container will be restarted.   Note the **restart count** when querying the status of the pods:
414
415 .. code-block:: bash
416    :caption: Checking PDP-X 0 restart count
417
418     ubuntu@k8sx:~$ kubectl get pods --all-namespaces -o=wide
419      
420     NAMESPACE  NAME                             READY     STATUS    RESTARTS   AGE     IP              NODE
421
422     onap       dev-brmsgw-5dbc4c8dc4-llk5s      1/1       Running   0          3d      10.42.120.43    k8sx
423     onap       dev-drools-0                     1/1       Running   0          3d      10.42.60.27     k8sx
424     onap       dev-nexus-7d96568f5f-qp5td       1/1       Running   0          3d      10.42.172.8     k8sx
425     onap       dev-pap-8587696769-vwj6k         2/2       Running   0          3d      10.42.19.137    k8sx
426     onap       dev-pdp-0                        2/2       Running   0          3d      10.42.144.218   k8sx
427     onap       dev-policydb-587d55bdff-4f5dz    1/1       Running   0          3d      10.42.12.242    k8sx
428     
429
430 During the restart process, the PAP component, will detect that PDP-X 0 is down and therefore its state being reflected in the PDP-X screen:
431
432     .. image:: srmPdpxResiliencyPdpMgmt1.png
433
434 This screen will be updated to reflect PDP-X 0 is back alive, after PDP-X 0 synchronizes itself with the PAP. 
435
436     .. image:: srmPdpxResiliencyPdpMgmt2.png
437
438 At that point, the PDP-X is usable either directly or through the service to query for policies.
439
440
441 .. code-block:: bash
442    :caption: Query PDP-X 1 for vCPE policy
443
444     # in this example we perform the vCPE query from the OOM VM
445     # the default installation credentials are used for querying the vCPE policy
446
447     ubuntu@k8sx:~$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.233.111:8081/pdp/api/getConfig | python -m json.tool
448     [
449         {
450             "config": "..",
451             "matchingConditions": {
452                 "ConfigName": "BRMS_PARAM_RULE",
453                 "ONAPName": "DROOLS"
454             },
455             "policyConfigMessage": "Config Retrieved! ",
456             "policyConfigStatus": "CONFIG_RETRIEVED",
457             "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
458             "policyType": "BRMS_PARAM",
459             "policyVersion": "1",
460             "property": null,
461             "responseAttributes": {
462                 "controller": "amsterdam"
463             },
464             "type": "OTHER"
465         },
466         {
467             "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
468             "matchingConditions": {
469                 "ConfigName": "SampleConfigName",
470                 "Location": "SampleServiceLocation",
471                 "ONAPName": "DCAE",
472                 "service": "tca_policy",
473                 "uuid": "test"
474             },
475             "policyConfigMessage": "Config Retrieved! ",
476             "policyConfigStatus": "CONFIG_RETRIEVED",
477             "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
478             "policyType": "MicroService",
479             "policyVersion": "1",
480             "property": null,
481             "responseAttributes": {},
482             "type": "JSON"
483         }
484     ]
485
486 PDP-D Resiliency
487 ^^^^^^^^^^^^^^^^
488
489 A PDP-D container failure can be simulated by either:
490    a) performing a"policy stop" operation within the PDP-D pod, which in fact will shutdown the PDP-D service, and eventually will be detected by the liveness checks, or 
491    b) by plainly deleting the corresponding pod.    
492
493 In the following example, the PDP-D 0 is forced to fail.
494
495 .. code-block:: bash
496    :caption: Causing PDP-D 0 to fail
497
498     # In these scenarios the liveness check will fail and recovery actions will take place.
499
500     # Alternative 1: in this scenario we shutdown the PDP-D 0 policy process, so the liveness monitored ports 
501     # will be down (but the pod is up) and corrective measures will be applied
502
503     ubuntu@k8sx:~/oom/kubernetes$ kubectl exec -it dev-drools-0 --container drools -n onap -- bash -c "source /opt/app/policy/etc/profile.d/env.sh; policy stop"
504     [drools-pdp-controllers]
505     L []: Stopping Policy Management... Policy Management (pid=3284) is stopping... Policy Management has stopped.
506
507
508 Upon detection of the service being down through the liveness check, the container will be restarted.   Note the restart count when querying the status of the pods:
509
510 .. code-block:: bash
511    :caption: Checking PDP-D 0 restart count
512
513     ubuntu@k8sx:~$ kubectl get pods --all-namespaces -o=wide | grep drool
514     onap    dev-drools-0     0/1    Running   0    1d     10.42.10.21     k8sx
515     ..
516     ubuntu@k8sx:~$ kubectl get pods --all-namespaces -o=wide | grep drools
517     onap    dev-drools-0     1/1    Running   1    1d     10.42.10.21     k8sx  <-- note restart count
518
519 Verification that the restarted PDP-D 0 comes up with the appropriate policy loaded can be verified by checking its maven coordinates:
520
521 .. code-block:: bash
522    :caption: Verifying restarted PDP-D points to policies pre-failure.
523
524     ubuntu@k8sx:~$ curl --silent --user '@1b3rt:31nst31n' -X GET http://10.42.10.21:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
525     {
526         "alive": true,
527         "artifactId": "policy-amsterdam-rules",
528         "brained": true,
529         "groupId": "org.onap.policy-engine.drools.amsterdam",
530         "locked": false,
531         "modelClassLoaderHash": 189820624,
532         "recentSinkEvents": [],
533         "recentSourceEvents": [],
534         "sessionCoordinates": [
535             "org.onap.policy-engine.drools.amsterdam:policy-amsterdam-rules:0.4.0:closedloop-amsterdam"
536         ],
537         "sessions": [
538             "closedloop-amsterdam"
539         ],
540         "version": "0.4.0"
541     }
542
543
544 PDP-X Scaling
545 ^^^^^^^^^^^^^
546
547 To scale a new PDP-X, set the replica count appropriately.   
548
549 In our tests below, we are going to work with the OOM policy component in isolation.   In this exercise, we scale the PDP-X with 1 additional replica, PDP-X 1.  
550
551 .. code-block:: bash
552    :caption: Scaling a PDP-X
553
554     ubuntu@k8sx:~$ helm upgrade -i dev local/onap --namespace onap --set global.pullPolicy=IfNotPresent --set policy.pdp.replicaCount=2
555     Release "dev" has been upgraded. Happy Helming!
556     LAST DEPLOYED: Mon Jun  4 15:19:05 2018
557     NAMESPACE: onap
558     STATUS: DEPLOYED
559      
560     RESOURCES:
561     ==> v1/Service
562     NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                        AGE
563     dbc-pg-primary            ClusterIP  10.43.29.226   <none>       5432/TCP                       2d
564     dbc-pg-replica            ClusterIP  10.43.202.168  <none>       5432/TCP                       2d
565     dbc-postgres              ClusterIP  10.43.181.134  <none>       5432/TCP                       2d
566     dmaap-bc                  NodePort   10.43.254.230  <none>       8080:30241/TCP,8443:30242/TCP  2d
567     message-router-kafka      ClusterIP  10.43.69.159   <none>       9092/TCP                       2d
568     message-router-zookeeper  ClusterIP  None           <none>       2181/TCP                       2d
569     message-router            NodePort   10.43.123.102  <none>       3904:30227/TCP,3905:30226/TCP  2d
570     msb-consul                NodePort   10.43.27.77    <none>       8500:30285/TCP                 2d
571     msb-discovery             NodePort   10.43.178.20   <none>       10081:30281/TCP                2d
572     msb-eag                   NodePort   10.43.77.235   <none>       80:30282/TCP,443:30284/TCP     2d
573     msb-iag                   NodePort   10.43.221.196  <none>       80:30280/TCP,443:30283/TCP     2d
574     brmsgw                    NodePort   10.43.21.222   <none>       9989:30216/TCP                 2d
575     nexus                     NodePort   10.43.159.27   <none>       8081:30236/TCP                 2d
576     drools                    NodePort   10.43.233.67   <none>       6969:30217/TCP,9696:30221/TCP  2d
577     policydb                  ClusterIP  None           <none>       3306/TCP                       2d
578     pdp                       ClusterIP  None           <none>       8081/TCP                       2d
579     pap                       NodePort   10.43.110.50   <none>       8443:30219/TCP,9091:30218/TCP  2d
580     robot                     NodePort   10.43.172.248  <none>       88:30209/TCP                   2d
581      
582     ==> v1beta1/Deployment
583     NAME                          DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
584     dev-dmaap-bus-controller      1        1        1           1          2d
585     dev-message-router-kafka      1        1        1           1          2d
586     dev-message-router-zookeeper  1        1        1           1          2d
587     dev-message-router            1        1        1           1          2d
588     dev-kube2msb                  1        1        1           1          2d
589     dev-msb-consul                1        1        1           1          2d
590     dev-msb-discovery             1        1        1           1          2d
591     dev-msb-eag                   1        1        1           1          2d
592     dev-msb-iag                   1        1        1           1          2d
593     dev-brmsgw                    1        1        1           1          2d
594     dev-nexus                     1        1        1           1          2d
595     dev-policydb                  1        1        1           1          2d
596     dev-pap                       1        1        1           1          2d
597     dev-robot                     1        1        1           1          2d
598      
599     ==> v1beta1/StatefulSet
600     NAME        DESIRED  CURRENT  AGE
601     dev-dbc-pg  2        2        2d
602     dev-drools  1        1        2d
603     dev-pdp     2        2        2d
604      
605     ==> v1/PersistentVolumeClaim
606     NAME                          STATUS  VOLUME                        CAPACITY  ACCESS MODES  STORAGECLASS  AGE
607     dev-message-router-kafka      Bound   dev-message-router-kafka      2Gi       RWX           2d
608     dev-message-router-zookeeper  Bound   dev-message-router-zookeeper  2Gi       RWX           2d
609     dev-nexus                     Bound   dev-nexus                     2Gi       RWX           2d
610     dev-policydb                  Bound   dev-policydb                  2Gi       RWX           2d
611      
612     ==> v1/ConfigMap
613     NAME                                         DATA  AGE
614     dev-dmaap-bus-controller-config              1     2d
615     dev-message-router-cadi-prop-configmap       1     2d
616     dev-message-router-msgrtrapi-prop-configmap  1     2d
617     dev-msb-discovery                            1     2d
618     dev-msb-eag                                  1     2d
619     dev-msb-iag                                  1     2d
620     dev-brmsgw-pe-configmap                      2     2d
621     dev-drools-configmap                         6     2d
622     dev-drools-log-configmap                     1     2d
623     dev-drools-settings-configmap                1     2d
624     dev-policydb-configmap                       1     2d
625     dev-pdp-log-configmap                        1     2d
626     dev-pdp-pe-configmap                         3     2d
627     dev-pe-scripts-configmap                     1     2d
628     dev-filebeat-configmap                       1     2d
629     dev-pe-configmap                             1     2d
630     dev-pap-pe-configmap                         7     2d
631     dev-pap-sdk-log-configmap                    1     2d
632     dev-pap-log-configmap                        1     2d
633     dev-robot-resources-configmap                3     2d
634     dev-robot-lighttpd-authorization-configmap   1     2d
635     dev-robot-eteshare-configmap                 4     2d
636      
637     ==> v1/PersistentVolume
638     NAME                          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                              STORAGECLASS     REASON  AGE
639     dev-dbc-pg-data0              1Gi       RWO           Retain          Bound   onap/dev-dbc-pg-data-dev-dbc-pg-0  dev-dbc-pg-data  2d
640     dev-dbc-pg-data1              1Gi       RWO           Retain          Bound   onap/dev-dbc-pg-data-dev-dbc-pg-1  dev-dbc-pg-data  2d
641     dev-message-router-kafka      2Gi       RWX           Retain          Bound   onap/dev-message-router-kafka      2d
642     dev-message-router-zookeeper  2Gi       RWX           Retain          Bound   onap/dev-message-router-zookeeper  2d
643     dev-nexus                     2Gi       RWX           Retain          Bound   onap/dev-nexus                     2d
644     dev-policydb                  2Gi       RWX           Retain          Bound   onap/dev-policydb                  2d
645      
646     ==> v1beta1/ClusterRoleBinding
647     NAME          AGE
648     onap-binding  2d
649      
650     ==> v1/Pod(related)
651     NAME                                          READY  STATUS    RESTARTS  AGE
652     dev-dmaap-bus-controller-5bd859c7dc-blzdc     1/1    Running   0         2d
653     dev-message-router-kafka-748cdf7b9c-srv7l     1/1    Running   0         2d
654     dev-message-router-zookeeper-5b5969f6f-8rk9w  1/1    Running   0         2d
655     dev-message-router-b5bdc599c-5h56k            1/1    Running   0         2d
656     dev-kube2msb-579fc77c54-m84qx                 1/1    Running   0         2d
657     dev-msb-consul-7bc4fcc8-94gsc                 1/1    Running   0         2d
658     dev-msb-discovery-768547bcb-2hr7j             2/2    Running   0         2d
659     dev-msb-eag-5d95686c67-9lkzs                  2/2    Running   0         2d
660     dev-msb-iag-675b649848-pv2gh                  2/2    Running   0         2d
661     dev-brmsgw-5675f5877b-wv68s                   1/1    Running   0         2d
662     dev-nexus-7d96568f5f-m8c4l                    1/1    Running   0         2d
663     dev-policydb-587d55bdff-9gdjv                 1/1    Running   0         2d
664     dev-pap-678b44cd87-wxbww                      2/2    Running   0         2d
665     dev-robot-589c76bb6b-hrrdn                    1/1    Running   0         2d
666     dev-dbc-pg-0                                  1/1    Running   0         2d
667     dev-dbc-pg-1                                  1/1    Running   0         2d
668     dev-drools-0                                  1/1    Running   1         2d
669     dev-pdp-0                                     2/2    Running   1         2d
670     dev-pdp-1                                     0/2    Init:0/1  0         0s
671      
672     ==> v1/Secret
673     NAME                       TYPE                     DATA  AGE
674     dev-dbc-pg                 Opaque                   3     2d
675     dev-message-router-secret  Opaque                   1     2d
676     dev-drools-secret          Opaque                   2     2d
677     dev-policydb-secret        Opaque                   2     2d
678     onap-docker-registry-key   kubernetes.io/dockercfg  1     2d
679     
680
681 Check Policy Engine UI how the PDP-Xs are coming up and request policies to the PAP.
682
683 Eventually the new PDP-X will be connected and serving policies:
684
685     .. image:: srmPdpxScalingPdpMgmt1.png
686
687 The new PDP-X should be now ready to serve policies:
688
689 .. code-block:: bash
690    :caption: Check that the new PDP-X 3 and 4 can serve policies
691
692     ubuntu@k8sx:~/oom/kubernetes$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.183.0:8081/pdp/api/getConfig | python -m json.tool
693     [
694         {
695             "config": ..
696             "matchingConditions": {
697                 "ConfigName": "BRMS_PARAM_RULE",
698                 "ONAPName": "DROOLS"
699             },
700             "policyConfigMessage": "Config Retrieved! ",
701             "policyConfigStatus": "CONFIG_RETRIEVED",
702             "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
703             "policyType": "BRMS_PARAM",
704             "policyVersion": "1",
705             "property": null,
706             "responseAttributes": {
707                 "controller": "amsterdam"
708             },
709             "type": "OTHER"
710         },
711         {
712             "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
713             "matchingConditions": {
714                 "ConfigName": "SampleConfigName",
715                 "Location": "SampleServiceLocation",
716                 "ONAPName": "DCAE",
717                 "service": "tca_policy",
718                 "uuid": "test"
719             },
720             "policyConfigMessage": "Config Retrieved! ",
721             "policyConfigStatus": "CONFIG_RETRIEVED",
722             "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
723             "policyType": "MicroService",
724             "policyVersion": "1",
725             "property": null,
726             "responseAttributes": {},
727             "type": "JSON"
728         }
729     ]
730      
731      
732     ubuntu@k8sx:~/oom/kubernetes$ curl --silent -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'ClientAuth: cHl0aG9uOnRlc3Q=' --header 'Authorization: Basic dGVzdHBkcDphbHBoYTEyMw==' --header 'Environment: TEST' -d '{"policyName": ".*vCPE.*"}' http://10.42.137.241:8081/pdp/api/getConfig | python -m json.tool
733     [
734         {
735             "config": "{\"service\":\"tca_policy\",\"location\":\"SampleServiceLocation\",\"uuid\":\"test\",\"policyName\":\"MicroServicevCPE\",\"description\":\"MicroService vCPE Policy\",\"configName\":\"SampleConfigName\",\"templateVersion\":\"OpenSource.version.1\",\"version\":\"1.1.0\",\"priority\":\"1\",\"policyScope\":\"resource=SampleResource,service=SampleService,type=SampleType,closedLoopControlName=ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"riskType\":\"SampleRiskType\",\"riskLevel\":\"1\",\"guard\":\"False\",\"content\":{\"tca_policy\":{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}}}",
736             "matchingConditions": {
737                 "ConfigName": "SampleConfigName",
738                 "Location": "SampleServiceLocation",
739                 "ONAPName": "DCAE",
740                 "service": "tca_policy",
741                 "uuid": "test"
742             },
743             "policyConfigMessage": "Config Retrieved! ",
744             "policyConfigStatus": "CONFIG_RETRIEVED",
745             "policyName": "com.Config_MS_MicroServicevCPE.1.xml",
746             "policyType": "MicroService",
747             "policyVersion": "1",
748             "property": null,
749             "responseAttributes": {},
750             "type": "JSON"
751         },
752         {
753             "config": ...
754                 "ConfigName": "BRMS_PARAM_RULE",
755                 "ONAPName": "DROOLS"
756             },
757             "policyConfigMessage": "Config Retrieved! ",
758             "policyConfigStatus": "CONFIG_RETRIEVED",
759             "policyName": "com.Config_BRMS_Param_BRMSParamvCPE.1.xml",
760             "policyType": "BRMS_PARAM",
761             "policyVersion": "1",
762             "property": null,
763             "responseAttributes": {
764                 "controller": "amsterdam"
765             },
766             "type": "OTHER"
767         }
768     ]
769
770
771 PDP-D Scaling
772 ^^^^^^^^^^^^^
773
774 To scale a new PDP-D, set the replica count appropriately.   In our scenario below, we are going to scale the PDP-D service to add a new pod (2 active PDP-Ds).
775
776 .. code-block:: bash
777    :caption: Scaling a PDP-D
778
779     #   Note: we also set the PDP-X pool to 2 instances (matching the previous section)
780      
781      
782     ubuntu@k8sx:~$ helm upgrade -i dev local/onap --namespace onap --set global.pullPolicy=IfNotPresent --set policy.pdp.replicaCount=2 --set policy.drools.replicaCount=2
783     Release "dev" has been upgraded. Happy Helming!
784     LAST DEPLOYED: Mon Jun  4 15:52:46 2018
785     NAMESPACE: onap
786     STATUS: DEPLOYED
787      
788     RESOURCES:
789     ==> v1/ConfigMap
790     NAME                                         DATA  AGE
791     dev-dmaap-bus-controller-config              1     2d
792     dev-message-router-cadi-prop-configmap       1     2d
793     dev-message-router-msgrtrapi-prop-configmap  1     2d
794     dev-msb-discovery                            1     2d
795     dev-msb-eag                                  1     2d
796     dev-msb-iag                                  1     2d
797     dev-brmsgw-pe-configmap                      2     2d
798     dev-drools-configmap                         6     2d
799     dev-drools-log-configmap                     1     2d
800     dev-drools-settings-configmap                1     2d
801     dev-policydb-configmap                       1     2d
802     dev-pdp-pe-configmap                         3     2d
803     dev-pdp-log-configmap                        1     2d
804     dev-pe-scripts-configmap                     1     2d
805     dev-filebeat-configmap                       1     2d
806     dev-pe-configmap                             1     2d
807     dev-pap-pe-configmap                         7     2d
808     dev-pap-log-configmap                        1     2d
809     dev-pap-sdk-log-configmap                    1     2d
810     dev-robot-resources-configmap                3     2d
811     dev-robot-lighttpd-authorization-configmap   1     2d
812     dev-robot-eteshare-configmap                 4     2d
813      
814     ==> v1/PersistentVolume
815     NAME                          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                              STORAGECLASS     REASON  AGE
816     dev-dbc-pg-data0              1Gi       RWO           Retain          Bound   onap/dev-dbc-pg-data-dev-dbc-pg-0  dev-dbc-pg-data  2d
817     dev-dbc-pg-data1              1Gi       RWO           Retain          Bound   onap/dev-dbc-pg-data-dev-dbc-pg-1  dev-dbc-pg-data  2d
818     dev-message-router-kafka      2Gi       RWX           Retain          Bound   onap/dev-message-router-kafka      2d
819     dev-message-router-zookeeper  2Gi       RWX           Retain          Bound   onap/dev-message-router-zookeeper  2d
820     dev-nexus                     2Gi       RWX           Retain          Bound   onap/dev-nexus                     2d
821     dev-policydb                  2Gi       RWX           Retain          Bound   onap/dev-policydb                  2d
822      
823     ==> v1/PersistentVolumeClaim
824     NAME                          STATUS  VOLUME                        CAPACITY  ACCESS MODES  STORAGECLASS  AGE
825     dev-message-router-kafka      Bound   dev-message-router-kafka      2Gi       RWX           2d
826     dev-message-router-zookeeper  Bound   dev-message-router-zookeeper  2Gi       RWX           2d
827     dev-nexus                     Bound   dev-nexus                     2Gi       RWX           2d
828     dev-policydb                  Bound   dev-policydb                  2Gi       RWX           2d
829      
830     ==> v1beta1/ClusterRoleBinding
831     NAME          AGE
832     onap-binding  2d
833      
834     ==> v1beta1/Deployment
835     NAME                          DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
836     dev-dmaap-bus-controller      1        1        1           1          2d
837     dev-message-router-kafka      1        1        1           1          2d
838     dev-message-router-zookeeper  1        1        1           1          2d
839     dev-message-router            1        1        1           1          2d
840     dev-kube2msb                  1        1        1           1          2d
841     dev-msb-consul                1        1        1           1          2d
842     dev-msb-discovery             1        1        1           1          2d
843     dev-msb-eag                   1        1        1           1          2d
844     dev-msb-iag                   1        1        1           1          2d
845     dev-brmsgw                    1        1        1           1          2d
846     dev-nexus                     1        1        1           1          2d
847     dev-policydb                  1        1        1           1          2d
848     dev-pap                       1        1        1           1          2d
849     dev-robot                     1        1        1           1          2d
850      
851     ==> v1/Pod(related)
852     NAME                                          READY  STATUS    RESTARTS  AGE
853     dev-dmaap-bus-controller-5bd859c7dc-blzdc     1/1    Running   0         2d
854     dev-message-router-kafka-748cdf7b9c-srv7l     1/1    Running   0         2d
855     dev-message-router-zookeeper-5b5969f6f-8rk9w  1/1    Running   0         2d
856     dev-message-router-b5bdc599c-5h56k            1/1    Running   0         2d
857     dev-kube2msb-579fc77c54-m84qx                 1/1    Running   0         2d
858     dev-msb-consul-7bc4fcc8-94gsc                 1/1    Running   0         2d
859     dev-msb-discovery-768547bcb-2hr7j             2/2    Running   0         2d
860     dev-msb-eag-5d95686c67-9lkzs                  2/2    Running   0         2d
861     dev-msb-iag-675b649848-pv2gh                  2/2    Running   0         2d
862     dev-brmsgw-5675f5877b-wv68s                   1/1    Running   0         2d
863     dev-nexus-7d96568f5f-m8c4l                    1/1    Running   0         2d
864     dev-policydb-587d55bdff-9gdjv                 1/1    Running   0         2d
865     dev-pap-678b44cd87-wxbww                      2/2    Running   0         2d
866     dev-robot-589c76bb6b-hrrdn                    1/1    Running   0         2d
867     dev-dbc-pg-0                                  1/1    Running   0         2d
868     dev-dbc-pg-1                                  1/1    Running   0         2d
869     dev-drools-0                                  1/1    Running   1         2d
870     dev-drools-1                                  0/1    Init:0/1  0         1s
871     dev-pdp-0                                     2/2    Running   1         2d
872     dev-pdp-1                                     2/2    Running   0         33m
873      
874     ==> v1/Secret
875     NAME                       TYPE                     DATA  AGE
876     dev-dbc-pg                 Opaque                   3     2d
877     dev-message-router-secret  Opaque                   1     2d
878     dev-drools-secret          Opaque                   2     2d
879     dev-policydb-secret        Opaque                   2     2d
880     onap-docker-registry-key   kubernetes.io/dockercfg  1     2d
881      
882     ==> v1beta1/StatefulSet
883     NAME        DESIRED  CURRENT  AGE
884     dev-dbc-pg  2        2        2d
885     dev-drools  2        2        2d
886     dev-pdp     2        2        2d
887      
888     ==> v1/Service
889     NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                        AGE
890     dbc-postgres              ClusterIP  10.43.181.134  <none>       5432/TCP                       2d
891     dbc-pg-replica            ClusterIP  10.43.202.168  <none>       5432/TCP                       2d
892     dbc-pg-primary            ClusterIP  10.43.29.226   <none>       5432/TCP                       2d
893     dmaap-bc                  NodePort   10.43.254.230  <none>       8080:30241/TCP,8443:30242/TCP  2d
894     message-router-kafka      ClusterIP  10.43.69.159   <none>       9092/TCP                       2d
895     message-router-zookeeper  ClusterIP  None           <none>       2181/TCP                       2d
896     message-router            NodePort   10.43.123.102  <none>       3904:30227/TCP,3905:30226/TCP  2d
897     msb-consul                NodePort   10.43.27.77    <none>       8500:30285/TCP                 2d
898     msb-discovery             NodePort   10.43.178.20   <none>       10081:30281/TCP                2d
899     msb-eag                   NodePort   10.43.77.235   <none>       80:30282/TCP,443:30284/TCP     2d
900     msb-iag                   NodePort   10.43.221.196  <none>       80:30280/TCP,443:30283/TCP     2d
901     brmsgw                    NodePort   10.43.21.222   <none>       9989:30216/TCP                 2d
902     nexus                     NodePort   10.43.159.27   <none>       8081:30236/TCP                 2d
903     drools                    NodePort   10.43.233.67   <none>       6969:30217/TCP,9696:30221/TCP  2d
904     policydb                  ClusterIP  None           <none>       3306/TCP                       2d
905     pdp                       ClusterIP  None           <none>       8081/TCP                       2d
906     pap                       NodePort   10.43.110.50   <none>       8443:30219/TCP,9091:30218/TCP  2d
907     robot                     NodePort   10.43.172.248  <none>       88:30209/TCP                   2d
908     
909 Verify that the new PDP-D comes up with the latest policy coordinates:
910
911 .. code-block:: bash
912    :caption: Verify new PDP-D 2 comes up with policies loaded
913
914     ubuntu@k8sx:~$ curl --silent --user '@1b3rt:31nst31n' -X GET http://10.42.172.88:9696/policy/pdp/engine/controllers/amsterdam/drools | python -m json.tool
915     {
916         "alive": true,
917         "artifactId": "policy-amsterdam-rules",
918         "brained": true,
919         "groupId": "org.onap.policy-engine.drools.amsterdam",
920         "locked": false,
921         "modelClassLoaderHash": 1657760388,
922         "recentSinkEvents": [],
923         "recentSourceEvents": [],
924         "sessionCoordinates": [
925             "org.onap.policy-engine.drools.amsterdam:policy-amsterdam-rules:0.5.0:closedloop-amsterdam"
926         ],
927         "sessions": [
928             "closedloop-amsterdam"
929         ],
930         "version": "0.5.0"
931     }
932     
933
934 End of Document
935
936 .. SSNote: Wiki page ref. https://wiki.onap.org/display/DW/Policy+on+OOM
937 .. SSNote: Old Wiki page ref. https://wiki.onap.org/display/DW/Scalability%2C+Resiliency+and+Manageability
938
939