A "Message Feeder Output Congestion" Alarm Raised On The DIH Server.

Asset ID:	1-72-2387747.1
Update Date:	2018-04-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2387747.1 : A "Message Feeder Output Congestion" Alarm Raised On The DIH Server.

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 5.0 and later
Information in this document applies to any platform.

Symptoms

"Message Feeder Output Congestion" alarm raised on the DIH server.

Changes

Cause

The build process of the session was generating following error message in the build process log (tr.tail -10 IxpBuildxxxx).

hMem::createFragShMem(isBig=0, useHugePages=0, fragSizeK=2024, tblCnt=7, subtblCnt=4)
  ^^ IdbDict::mkEmptyShFrag(partName=DtsBlockPart, subpart=88, fragInst=0xffffffff)
  ^^ IdbPart::initEmptyFrag(part=DtsBlockPart, shmKey=-1)
  ^^ IdbPart::getNewFrag(begTime=1559388472923, endTime=1559388472923)
  ^^ IdbFragIter::switchFrag(newFrag=0, doCopyAutoIncr=1)
  ^^ current frag=0xbffa 20180404.102938.9080.DtsBlockPart.0088,23
  ^^ IdbIter::addRecSelect(void *xRec)
  , Stream : 16728 [26829/DtsProducerBuffer.cpp:538]
0404:201159.061 IXP3 Send alarm: xDR Loss (major event) Stream = xx_xx_xxxx_xx [26829/DmAlarm.cpp:293]
0404:201159.061 IXP-V Unable to send alarm: ERROR: E_NOMEM: cannot shmat
  ^^ shmat(shmid=14909748, shmaddr=0, shmflg=0)
  ^^ ShMemMap::attach(shmid=14909748, addr=0, shmflg=0)
  ^^ segNum=1 nSegments=2
  ^^ IprFragShMem::attach(shMemId=0xe38134, addr=0xa3d63000, key=0xdb91ce, mode=1)
  ^^ IdbPart::loadFrag(key=679:20180404.110959.4110.IxpLocalPart[0])
  ^^ IdbPart::getFrag()
  ^^ IdbPart::getPrevFrag()
  ^^ openLast(): getPrevFrag(NULL) [get-last-frag] failed
  ^^ IdbFragIter::_open()
  ^^ subpart=IxpLocalPart
  ^^ IdbIter::open(spec="Alarm", sig=0x1005c8a8)
  [26829/DmAlarm.cpp:312]
0404:201159.061 BLD2 BldXdrStorageThread::ProcessData - cannot send alarm xDR lost [26829/BldXdrStorageThread.h:197]
0404:201159.062 IXP3 Send alarm: xDR Loss clear (clear event) Stream = xx_xx_xxxx_xx [26829/DmAlarm.cpp:293]
0404:201159.062 IXP-V Unable to send alarm: ERROR: E_NOMEM: cannot shmat

Solution

Since build process was restarting, PMF was not able to send data to build process. pduServer0 process log was showing that PMF was observing congestion while sending data to destination.

0404:201249.716 TR-V Congestion alarm raised. [8784/DsLink.C:238]
0404:201317.540 TR-V xxxxx_pmf-xx_ixp0001_xxxxx - congestion happened before timeout, restarting the timer. [8784/DsLink.C:244]
0404:201411.408 TR-V xxxxx-xx_pmf-xx_ixp0001_xxxxx - congestion happened before timeout, restarting the timer. [8784/DsLink.C:244]
0404:201507.464 TR-V DsChildLink::disconnectCb: xxxxx-xx_pmf-xx_ixp0001_xxxxx, getCd() = -1 [8784/DsServerLink.C:80]
0404:201507.464 TR-V Scheduling thread removal for 0x99cbf54: DataDest name:ip:port=xxxx-xx_pmf-xx_ixp0001_17070-xxxx:40633 state=RUN IP/Port=xx.xx.xx.xx:xxxx [8784/ServerFlowMgr.C:490]
0404:201507.464 TR-V ds xxxxx_pmf-xx_ixp0001_xxxx: state = 2, SigCtl::toExit() = 0 [8784/DataDest.C:661]
0404:201507.465 TR-V Process exit for DataDest name:ip:port=xxxxx_pmf-xx_ixp0001_xxxx-xxxx state=STOP IP/Port=xx.xx.xx.xx:xxxx [8784/DataDest.C:879]
0404:201507.981 TR-V chkDSRemoval - shutdown 0x99cbf54 [6035/ServerFlowMgr.C:510]
0404:201507.981 TR-V shutdown_wait(5) done for DataDest name:ip:port=xxxx-DF_pmf-xx_ixp0001_xxxx-xxxx state=DONE IP/Port=xx.xx.xx.xx:xxxx [6035/DataDest.C:854]
0404:201521.693 TR-V Change state : 7 [6035/DataDest.C:351]
0404:201521.693 TR-V Change state : 7 [6035/DataDest.C:351]
0404:201521.693 TR-V Change state :

It will fix automatically when PMF will be able to connect to build process running on ixp server.If not fixed automatically in few minutes, perform following steps :

Restart Build process by executing command:- pm.set off IXP_xxxx & pm.set on IXP_xxxx
pduServer0 process by executing command pm.set off pduServer0 & pm.set on pduServer0.
Refer KM 2062525.1, 2091707.1.

References

<NOTE:2180507.2> - Oracle Communications Life Cycle Notifications for Tekelec-Based Products

Attachments

This solution has no attachment