Diameter Signaling Router (DSR) - How to Clear SDS Stuck Alarm Id10009

Asset ID:	1-71-2056747.1
Update Date:	2016-05-27
Keywords:

Solution Type Technical Instruction Sure

Solution 2056747.1 : Diameter Signaling Router (DSR) - How to Clear SDS Stuck Alarm Id10009 - Config and Prov DB not yet Synchronized

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 5.0 to DSR 7.1.0 [Release DSR 5.0 to DSR 7.0]
Information in this document applies to any platform.

Goal

How to clear SDS Critical Alarm ID 10009 "Config and Prov DB not yet synchronized" that may appear after NOAM switchover

Solution

This issue is due to some signal event timing internal to processes (apwSoapServer and Comcol) within SDS, and has been resolved in SDS 7.1 release under Bug 19460069. The issue does not occur upon every failover. When it is encountered and does not clear, if it is unaccompanied by other alarms it is likely in this 'stuck' state and will require manual intervention to resolve.

Instructions to Clear:

This Bug fails to register the alarm properly, so we have to manually re-assert the alarm such that it registers correctly before we can successfully clear it.

First, verify no other alarms are present. If additional alarms are present, engage Oracle TAC via MOS.

Next, look at the alarm to identify the INSTANCE information:

      TIMESTAMP: 2015-07-27 16:24:14.194 CEST
NETWORK_ELEMENT: DR_NO_SDS_OFR
         SERVER: noamx-sds
        SEQ_NUM: 842
   EVENT_NUMBER: 10009
       SEVERITY: CRITICAL
        PRODUCT: OAM
        PROCESS: apwSoapServer
           TYPE: REPL
       INSTANCE: noamx-sds    <===================== INSTANCE VALUE!
           NAME: Config and Prov db not yet synchronized
          DESCR: Config and Prov DB not yet synchronized
       ERR_INFO: replication not up to date
           SECS: 1438007054
          USECS: 194000
         CISECS: 1438007054
        CIUSECS: 194000
             ID: 0

Finally, apply the workaround using the INSTANCE captured above.

1- Log into the affected sds noam as root user

[root@noamx-sds ~]#

2- On the command line, verify the alarm by executing:

[root@noamx-sds ~]# iqt -E -finstance,raw -feventNumber,raw RecentAlarmEv.0 | grep "eventNumber=10009"

3- Re-assert the alarm by executing "alarm.put -e 10009 -s 1 -i <INSTANCE>" Example follows:

[root@noamx-sds ~]# alarm.put -e 10009 -s 1 -i noamx-sds

4- Wait 10 seconds, then execute the following (again, below uses our example alarm's INSTANCE value) to clear:

[root@noamx-sds ~]# alarm.put -e 10009 -s 5 -i noamx-sds

5 - Wait another 30 seconds and check at the command line to see if the alarm has cleared:

[root@noamx-sds ~]# iqt -E -finstance,raw -feventNumber,raw RecentAlarmEv.0 | grep "eventNumber=10009"

Note that it may take up to 5 minutes for the alarm to clear at the GUI and at the command line above. It should be faster, but if it is needed to pass an audit cycle it may take the extra time.

The commands above will not affect operation of the DSR or FABR, and are safe to execute as specified at any time.

References

<BUG:19460069> - STUCK ALARM 10009 - CONFIG AND PROV DB NOT YET SYNCHRONIZED

Attachments

This solution has no attachment