Diameter Signaling Router (DSR): Event_Number: 10200 – Remote Database Reinitialization In Progress

Asset ID:	1-72-2386617.1
Update Date:	2018-04-18
Keywords:

Solution Type Problem Resolution Sure

Solution 2386617.1 : Diameter Signaling Router (DSR): Event_Number: 10200 – Remote Database Reinitialization In Progress

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 7.3.0 and later
Tekelec

Symptoms

While doing the switchover of Active NOAM to DR NOAM, customer got following alarm on standby DR NOAM

Event_Number: 10200 – Remote Database Reinitialization In Progress

Changes

Cause

Standby DR NOAM server was rebuilt.
When it was inserted back in the server group 10200 - Remote Db Reinitialization Alarm Never Cleared.
ApwSoapServer which is responsible for Re initialization was having a watchdog failure and was restarting. below screenshot shows the watchdog failure

When a server gets added to the topology, apwsoapserver checks its connectivity. To check the connectivity it queries to cmsoapa process.
cmsoapa process was itself having thread watchdog failure, this can be verified as below

[admusr@noam1a proc]$ pwd
/var/TKLC/rundb/run/proc
[admusr@noam1a proc]$ tt cmsoapa
+ cf.follow -10 /var/TKLC/rundb/run/proc/cmsoapa/trace.dat

below screenshot of pl command shows that cmsoapa and apwSoapServer restarting (This can be verified by looking at column N).

cmsoapa does a name resolution from DNS server for checking the connectivity with each hosts in the topology.
Out of three named server first two in priority order was having latency that leads to delay in name resolution and cmsoapa gets time out in 10 second (this is internal timer of comcol).
Ping each of the nameserver above to determine the latency. In this case there was latency on first 2 nameservers.

[admusr@noam1a ~] $ cat /etc/resolv.conf

domain tekelec.com

nameserver 1.1.1.1

nameserver 2.2.2.2

nameserver 3.3.3.3

[admusr@noam1a ~]$

Solution

Change the priority in nameserver so that the third server has the highest priority. This server was in same site and connectivity was fast, this helped avoiding timeout of cmsoapa process.