Diameter PMF Stopped Processing Traffic Due To "System Out Of Shared Memory" (E

Asset ID:	1-72-2191876.1
Update Date:	2017-03-16
Keywords:

Solution Type Problem Resolution Sure

Solution 2191876.1 : Diameter PMF Stopped Processing Traffic Due To "System Out Of Shared Memory" (E_NOMEM) In PmiaMonitor Trace

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 10.2.0 and later
Information in this document applies to any platform.

Symptoms

PMF stopped processing incoming traffic with pmiaMonitor generating an alarm "SystemOutOfSharedMemory alarm set".

Changes

Cause

The number of available segment available in the IDB MsuBlock has been exhausted and PmiaMonitor can no longer store the new MSUs for processing. This is why the traffic can no longer be processed and sent to IXP.

pmiaMonitor log contains the following:

0918:172045.322 TR-V DbMsuBlock13 DBMsuBlock::add Failed to add a new block: ERROR: E_NOMEM: segment count limit exceeded (3000) [ShMemLimit.cxx:177]
  ^^ IprFragShMem::createShSeg(size=4194304, readOnly=0, useHugePages=0) [IprFragShMem.cxx:2876]
  ^^ IprFragShMem::createFragShMem(isBig=0, useHugePages=0, fragSizeK=4096, tblCnt=7, subtblCnt=4) [IprFragShMem.cxx:698]
  ^^ IdbDict::mkEmptyShFrag(partName=MsuPart, subpart=13, fragInst=0xffffffff) [mkEmpFrag.cxx:78]
  ^^ IdbPart::initEmptyFrag(part=MsuPart, shmKey=-1) [IdbPart.cxx:1944]
  ^^ IdbPart::getNewFrag(begTime=1509618938945, endTime=1509618938945) [IdbPart.cxx:1675]
  ^^ IdbFragIter::switchFrag(newFrag=0, doCopyAutoIncr=1)

Solution

pmiaMonitor is using lots of share memory due to IP statistical management. Per default the pmiaMonitor keeps track of the IP associations to compute statistics and it uses a lot of shared memory.

These statistics are not mandatory and disabling them will reduce the shared memory usage.

Here is how to proceed:

Logon as cfguser to the PMF
Disable the Statistics:

#iset -f _statEnabled=0 DbIpLink where _statEnabled=1
Restart pmiaMonitor:
$pm.kill pmiaMonitor
Switch to root user and modify ccm-config.xml to make the changes permanent:
$ su -
$ sed -i 's/1 as/0 as/' /opt/TKLCjmxagent/in/ccm-config.xml

Attachments

This solution has no attachment