Diameter Signaling Router (DSR): Event_Number - 31100

Asset ID:	1-72-2242964.1
Update Date:	2017-04-12
Keywords:

Solution Type Problem Resolution Sure

Solution 2242964.1 : Diameter Signaling Router (DSR): Event_Number - 31100 - DB Replication Fault

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 5.0 and later
Information in this document applies to any platform.

Symptoms

Database replication alarm (Event 31100) on both MP Server.
The Error info in alarm indicated that time on MP1 was 124sec behind of current time and time on MP2 was 124 sec ahead the current time

TIMESTAMP: 2017-03-12 05:38:32.113 GMT
NETWORK_ELEMENT: NOAM
SERVER: dsrNO
SEQ_NUM: 136503
EVENT_NUMBER: 31100
SEVERITY: MINOR
PRODUCT: Platform
PROCESS: inetrep
TYPE: SW
INSTANCE: mp1
NAME: DB Replication Fault
DESCR: The DB replication process is impaired by a s/w fault
ERR_INFO:
GN_WARNING/WRN mp1: time is 124 secs behind time here [SyncMaster.cxx:2378]
^^ [4360:SyncMaster.cxx:2385]

SECS: 1489297112
USECS: 113000
CISECS: 1489297112
CIUSECS: 113000
ID: 0
________________________________________________________________________________________________

TIMESTAMP: 2017-03-12 05:37:45.039 GMT
NETWORK_ELEMENT: NOAM
SERVER: mp
SEQ_NUM: 19518204
EVENT_NUMBER: 31100
SEVERITY: MINOR
PRODUCT: Platform
PROCESS: inetrep
TYPE: SW
INSTANCE: mp2
NAME: DB Replication Fault
DESCR: The DB replication process is impaired by a s/w fault
ERR_INFO:
GN_WARNING/WRN mp2: time is 124 secs ahead time here [SyncMaster.cxx:2378]
^^ [5462:SyncMaster.cxx:2385]

SECS: 1489297065
USECS: 39000
CISECS: 1489297065
CIUSECS: 39000
ID: 1

Cause

There was a ntp daemon not synchronized alarm (Event 32509) on MP1 server causing incorrect time on the server.

This in turn was causing the Database replication alarm 31100 on MP servers.

  TIMESTAMP: 2017-03-11 17:09:56.262 GMT
NETWORK_ELEMENT: NOAM
  SERVER: mp1
  SEQ_NUM: 19480753
  EVENT_NUMBER: 32509
  SEVERITY: MINOR
  PRODUCT: TPD
  PROCESS: cmplatalarm
  TYPE: PLAT
  INSTANCE:
  NAME: Server NTP Daemon Not Synchronized
  DESCR: Server NTP Daemon Not Synchronized
  ERR_INFO:
  GN_WARNING/WRN Platform detected an error condition [cmplatalarm.cxx:319]
  ^^ Additional details captured in /var/TKLC/log/syscheck/fail_log (timestamp: 1489252196) [cmplatalarm.cxx:323]
  ^^ [5441:cmplatalarm.cxx:325]

  SECS: 1489252196
  USECS: 262000
  CISECS: 1489252196
  CIUSECS: 262000
  ID: 7

Solution

Do the following on the server reporting the ntp alarm to clear both NTP issue and database replication alarms

# ntpq -pn

# sudo service ntpd status

# sudo service ntpd stop

# sudo ntpdate <IP from remote column of ntpq command> --> do the ntpdate command 3 times, until offset is/or is close to zero

# sudo service ntpd start

# sudo service ntpd status

# ntpq -pn

Warning:Sudden database timestamp jump may temporarily affect the system, so it is best that given server is take out of service (Active NOAM GUI: Main Menu--> Status & Mange --> HA-->Edit)

Attachments

This solution has no attachment