HLRR Ra.stat Output on SOAM shows Alarms for MP that are Not Present in the MP

Asset ID:	1-72-2344728.1
Update Date:	2018-01-16
Keywords:

Solution Type Problem Resolution Sure

Solution 2344728.1 : HLRR Ra.stat Output on SOAM shows Alarms for MP that are Not Present in the MP

Applies to:

Tekelec HLR Router - Version HLRR 4.0 and later
Tekelec

Symptoms

ra.stat -m -w output is showing transport and links down but the alarms have already cleared from the respective MP server.

On MP ra.stat shows no alarms:

[root@MP ~]# ra.stat
+ iqt -z -fsrcNode -fseverity,as=sev -fhits,as=ht,wid=2 -ftimeStamp,fmt=Time.M -feventNumber,as=event,wid=24,trunc,left -finstance,wid=12,left,over -ferrInfo,over,nonl RecentAlarmEv.1 by severity,timeStamp
srcNode sev ht timeStamp event instance errInfo
[root@MP ~]#

on SOAM, ra.stat -w -m shows multiple alarms for mp1-hstn:

         srcNode sev ht    timeStamp event                    instance        errInfo
MP *C 0 06:32:25.943 RSP/Destination Unavaila 209-102-063   GN_INFO/WRN RSP=209-102-063, PointCode=111-111-111 Reason: RSP status is Unavailable ^^ [2244:Srm.C:2323]
MP *C 0 06:32:25.943 RSP/Destination Unavaila 209-102-045   GN_INFO/WRN RSP=209-102-045, PointCode=111-111-111 Reason: RSP status is Unavailable ^^ [2244:Srm.C:2323]
MP *C 0 06:32:25.943 RSP/Destination Unavaila 209-102-062   GN_INFO/WRN RSP=209-102-062, PointCode=111-111-111 Reason: RSP status is Unavailable ^^ [2244:Srm.C:2323]
MP *C 0 06:32:25.944 RSP/Destination Unavaila 209-025-034   GN_INFO/WRN RSP=209-025-034, PointCode=111-111-111 Reason: RSP status is Unavailable ^^ [2244:Srm.C:2323]

Cause

inetmerge process syncs alarms between SOAM and the MPs. That may need a restart.

Alarms between SOAM and MP are synced but only in command line, some alarms are still appearing.

Solution

On Active SOAM:

iqt -E RecentAlarmEv.0
iqt -E RecentAlarmEv.1
ra.stat -m -w
On MP

iqt -E RecentAlarmEv.0
Make Active SOAM as Standby via GUI. Make sure that mate SOAM has taken role of ACTIVE.
On the SOAM that is put to standby:
- ps -ef | grep -i inetmerge
- pm.kill inetmerge
- wait for few minutes
- ps -ef | grep -i inetmerge
- Make sure that the inetmerge process has respawned with a new process ID.
On Standby SOAM:

iqt -E RecentAlarmEv.0
iqt -E RecentAlarmEv.1
ra.stat -m -w
On MP

iqt -E RecentAlarmEv.0

INTERNAL NOTES FOR ORACLE TAC:

If the issue does not clear after these steps, then SOAM will need to be rebooted but before that, gather following information and share with Oracle DSR FieldSupport for confirmation:

Output log of the steps 1-6 carried out.
Fresh Savelogs from SOAM and MP
inetmerge log from both SOAM and on MP
1. Please run: tr.cat inetmerge > /tmp/inetmerge_$HOSTNAME
2. This will create following file on soam: /tmp/inetmerge_soam
3. And following file on MP: /tmp/inetmerge_MP

Attachments

This solution has no attachment