Fix issues blocking election handler thread 63/26263/1
authorKevin McKiou <km097d@att.com>
Wed, 13 Dec 2017 21:26:59 +0000 (15:26 -0600)
committerKevin McKiou <km097d@att.com>
Wed, 13 Dec 2017 21:27:31 +0000 (15:27 -0600)
commitbc8c8286645f74753d175eee7ca62d989555c96c
treea94d177a5e6ef1908cb56f1153864737b7b42c15
parent4884099eff44975eee57e6748823ff73f965e332
Fix issues blocking election handler thread

This bug tracks the AT&T bug 355533.  The symptom was that drools
pdps that were backing each other up were becoming stuck in a
standby state.  The cause was that the election handler thread was
being hung by a call to PolicyEngine.manager.deactivate which shuts
down the topic endpoints when the drools pdp operational state
transitions to disabled. Related problems were that the election
handler heartbeat was NOT blocked when the main thread was blocked
and the IntegrityMonitor forward progress counter was NOT blocked
from incrementing when the election handler thread was blocked.
This prevented the correct failover of the drools pdp to another
healthy one. This change fixes the two causes of the thread blockage,
moves the election handler heartbeat to the main thread and adds an
interface (AllSeemsWell) which is called when the election handler
has stalled/resumed. The AllSeemsWell interface will block forward
progress counter increments when ALLNOTWELL and will resume forward
progress counter increments when ALLSEEMSWELL. In addition, it reduces
the run time of the StandbyStateManagementTest from approximately 8
minutes to approximately 2 minutes. Since this changes classes also
changed by POLICY-444, this change must be merged before POLICY-444
can be merged.

Issue-ID: POLICY-501
Change-Id: I7b8180d11077ccf59b21b6484cb58b5522a3df8f
Signed-off-by: Kevin McKiou <km097d@att.com>
api-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeatureAPI.java
feature-active-standby-management/src/main/java/org/onap/policy/drools/activestandby/DroolsPdpsElectionHandler.java
feature-active-standby-management/src/test/java/org/onap/policy/drools/controller/test/AllSeemsWellTest.java [new file with mode: 0644]
feature-active-standby-management/src/test/java/org/onap/policy/drools/controller/test/StandbyStateManagementTest.java
feature-active-standby-management/src/test/resources/asw/feature-active-standby-management.properties [new file with mode: 0644]
feature-active-standby-management/src/test/resources/asw/feature-state-management.properties [new file with mode: 0644]
feature-active-standby-management/src/test/resources/feature-active-standby-management.properties
feature-active-standby-management/src/test/resources/feature-state-management.properties
feature-state-management/src/main/java/org/onap/policy/drools/statemanagement/DroolsPDPIntegrityMonitor.java
feature-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeature.java