SDS SOAM server missing from NOAM Status & Manage->Server View

Asset ID:	1-72-2326404.1
Update Date:	2018-03-06
Keywords:

Solution Type Problem Resolution Sure

Solution 2326404.1 : SDS SOAM server missing from NOAM Status & Manage->Server View

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 5.0 and later
Information in this document applies to any platform.

Symptoms

SDS SOAM server missing from NOAM Status & Manage->Server View

OAM Max HA Role showed OOS in NOAM GUI Main Menu: Status & Manage -> HA, even after reboot.

ha.mystate and pl commands gave error indicated database tables were not correctly populated:

[root@SOAM ~]# ha.mystate
ha.mystate: Cannot open NodeInfo at /opt/comcol/prod/bin/ha.mystate line 75.
[root@sdsSO-dsr3wtc2-b ~]# pl

10/23/2017 16:14:36 pm.wakeup#31000{}
** IDB_DBDOWN/FTL database not loaded [IdbDict.cxx:585]
^^ shmget(key=0xdb0000, size=1, shmflg=0) [CoreD.cxx:348]
^^ IprFragment::attach(key=0xdb0000, mode=0) [IprFrag.cxx:305]
^^ E_NOENT cannot shmget [CoreD.cxx:346]
^^ IdbDictMap::loadTbls() [IdbDict.cxx:607]
^^ PrGnTbl::getPrGnTbl(name=PmState) [PrGnTbl.cxx:237]
^^ IddTable::ddictBind(tblName=PmState, tblSig=0x77bf07c4) [IddTable.cxx:187]
^^ IdbReadIter::ddictBind() [IdbIter.cxx:602]
^^ IdbIter::open(spec="PmState", sig=0x77bf07c4) [IdbIter.cxx:2024]
^^ PmCtl::getRec() [PmCtl.cxx:37]
^^ PmCtl::getPmPid() [PmCtl.cxx:221]
^^ PmCtl::wakeUpPm() [PmCtl.cxx:351]

Changes

Cause

prod.log showed the following:

idbsync: writing PdbExceptionLog 0..1
idbsync: writing DiagnosticLogPart 0..1
idbsync: writing DiagnosticSharedPart
idbsync: writing DiagnosticLocalPart
idbsync: writing ComAgentConfig
idbsync: writing ComAgentAdmin
idbsync: writing ComAgentMerge 0..1
idbsync: writing ComAgentLocal
idbsync: writing UpdateLog
+ iqt -IiddToXML =DataDictPart > /var/TKLC/rundb/run/db/DataDictPart/20170922.143823.0210.DataDictPart.tmp
+ edd.op --install --must-eq-current /var/TKLC/rundb/run/db/DataDictPart/20170922.143823.0210.DataDictPart.tmp
created: 20170922.143823.0210.DataDictPart.xml
removed: 20140916.110911.8680.DataDictPart.xml
...removing existing IPC resources...
+ md_ipcrm ... 200 resources
...Removing UpSyncLog (transient update log)...

==== 2017-09-22 08:20:10 ====
...prod.dbup (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.dbup ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-09-22 15:20:10 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

==== 2017-09-22 08:20:10 ====
...prod.start (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.start ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-09-22 15:20:11 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

==== 2017-10-05 09:40:37 ====
...prod.dbdown (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

==== 2017-10-05 10:05:21 ====
...prod.dbup (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.dbup ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-10-05 17:05:21 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

==== 2017-10-05 10:05:22 ====
...prod.start (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.start ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-10-05 17:05:22 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

==== 2017-10-05 10:05:54 ====
...prod.dbdown (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

==== 2017-10-05 10:42:36 ====
...prod.dbup (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.dbup ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-10-05 17:42:36 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

==== 2017-10-05 10:42:36 ====
...prod.start (RUNID=00)...
...getting current state...
Current state: DbDown (database on disk but not loaded)

************** !!!!!!!!!!!!!!!!!!! *******************
***
*** prod.start ABORTING: bad system date.
Last started: 2014-09-16 12:18:31 UTC
Current: 2017-10-05 17:42:37 UTC
*** NOTE: manual recovery may be required
*** + A potentially problematic date change such as time going backwards
*** or time going far into the future was detected.
***
************** !!!!!!!!!!!!!!!!!!! *******************

Solution

The timestamp file is created/updated (DB Creation/Loading timestamp)
whenever the COMCOL DB is newly created or loaded from the disk. And
there is error check being done to make sure that the DB which is being
loaded is not more that 3 years old or if there is any unusual timestamp
jump.
So in this case the DB was up for more than 3 years and after the system
is rebooted we are encountering this issue.

To fix this issue we have two options
1. We can re.timestamp which updates the timestamp to current time. This can be followed by a reboot of the server
2. We can delete the file $RUN/timestamp. This file will be recreated when prod.dbup/prod.start is run during a reboot of the server.

For this specific case, i.e., when this document was initially written, option 1 was used to clear the issue.

Attachments

This solution has no attachment