After Upgrade to DSR R7.1.2 Inetmerge Process Stuck with Audit Status on Standby DR NO

Asset ID:	1-72-2197335.1
Update Date:	2016-10-27
Keywords:

Solution Type Problem Resolution Sure

Solution 2197335.1 : After Upgrade to DSR R7.1.2 Inetmerge Process Stuck with Audit Status on Standby DR NO

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 7.1.2 and later
Information in this document applies to any platform.

Symptoms

The inetmerge process is stuck with Audit status on the standby DR-NO Server after the upgrade to Release 7.1.2:

[admusr@xxdsrno02 ~]$ inetmstat -w
nodeId InetMerge State dir dSeq dTime updTime info
xxdsrno01 Standby To 0 0.00 09:39:49
xxdsrno01 Standby From 0 0.00 09:39:49
xxsrno02 Active To 0 0.00 09:39:49
xxdsrno02 Audit From 0 0.00 09:39:49
xxdsrno01 Standby To 0 0.00 09:39:49
xxdsrno01 Standby From 0 0.00 09:39:49
xxdsrso01 Standby From 0 0.00 09:39:49
xxdsrso02 Standby From 0 0.00 09:39:49
xxdsrso01 Standby From 0 0.00 09:39:49
xxdsrso02 Standby From 0 0.00 09:39:49

inetmerge log:

[root@xxdsrno02 ~]# tt inetmerge
+ cf.follow -10 /var/TKLC/rundb/run/proc/inetmerge/trace.dat
1018:184916.278 TR-V AUDIT [Receiver] chadsrno02: Audit Started. [12507/MergeReceiver.cxx:417]
1018:184916.292 TR-V AUDIT [Receiver] chadsrno02: Deleting fragment in table SNMPTrapLog [12507/MergeReceiver.cxx:559]
1018:184916.502 TR-V AUDIT [Receiver] chadsrno02: Audit Started. [12507/MergeReceiver.cxx:417]
1018:184916.516 TR-V AUDIT [Receiver] chadsrno02: Deleting fragment in table SNMPTrapLog [12507/MergeReceiver.cxx:559]
1018:184916.729 TR-V AUDIT [Receiver] chadsrno02: Audit Started. [12507/MergeReceiver.cxx:417]
1018:184916.742 TR-V AUDIT [Receiver] chadsrno02: Deleting fragment in table SNMPTrapLog [12507/MergeReceiver.cxx:559]
1018:184916.957 TR-V AUDIT [Receiver] chadsrno02: Audit Started. [12507/MergeReceiver.cxx:417]
1018:184916.970 TR-V AUDIT [Receiver] chadsrno02: Deleting fragment in table SNMPTrapLog [12507/MergeReceiver.cxx:559]
1018:184917.188 TR-V AUDIT [Receiver] chadsrno02: Audit Started. [12507/MergeReceiver.cxx:417]
1018:184917.202 TR-V AUDIT [Receiver] chadsrno02: Deleting fragment in table SNMPTrapLog [12507/MergeReceiver.cxx:559]

Changes

The upgrade of NOAM servers (2 NOAM servers and 2 DR-NOAM) completed from DSR 6.0.2 to DSR 7.1.2
SOAM and MP servers were still in DSR 6.0.2 and schedule for upgrade in next MTC.
The inetmerge process was seen stuck only on the standby DR NO Server

Cause

This appears to be a case of BUG 23743443. The part file for the SNMPTrapLog table is not being properly deleted from the file system, to make room for newer data to come in during the audit.

The issue can be identified by inetmerge trace of the form "Deleting fragment in table ..." for the same table repeated numerous times in quick succession.

Solution

Open a request to Oracle Support.

In this specific case for this table, the recovery/workaround would be the following procedure:

On the afflicted node:

Put the standby NOAM into Forced standby from the GUI
Bring COMCOL to DBdown state

prod.dbdown
Delete SNMPTraplog Part files from disk.

$ cd $DB/AppworksSNMPlog.0001
$ rm -f *
Verify the files are gone:

$ ls
Start COMCOL:

$ prod.start
Restore NOAM state from the GUI

References

<BUG:23743443> - INETMERGE: ENDLESS AUDIT LOOP BETWEEN ACTIVE & STANDBY NO

Attachments

This solution has no attachment