IMF Server Instability and Process Restarts Due to Oom-Killer on PIC 9.0.x

Asset ID:	1-72-2084552.1
Update Date:	2016-05-30
Keywords:

Solution Type Problem Resolution Sure

Solution 2084552.1 : IMF Server Instability and Process Restarts Due to Oom-Killer on PIC 9.0.x

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 7.5 to 9.0.4 [Release 7.0 to 9.0]
Information in this document applies to any platform.

Symptoms

Processes on IMF are randomly killed by oom-killer mechanism.

Changes

Server were migrated to new hardware with high amount of memory (24Gbits or more) and is running PIC 9.0.x which is 32 bits OS.

Cause

Amount of low memory (free -lm) is too low and cause oom-killer mechanism to be triggered.

Following message can be found in /var/log/messages:

Nov 18 00:12:11 cust-1c kernel: fcMonitor invoked oom-killer: gfp_mask=0xc0d0, order=3, oomkilladj=0
Nov 18 00:12:11 cust-1c kernel: [<c045b5d3>] out_of_memory+0x72/0x1a3
Nov 18 00:12:11 cust-1c kernel: [<c045cb56>] __alloc_pages+0x24e/0x2cf
Nov 18 00:12:11 cust-1c kernel: [<c045cbfe>] __get_free_pages+0x27/0x33
Nov 18 00:12:11 cust-1c kernel: [<f93be3f7>] pmia_set_ring+0x140/0x2da [af_pmia]
Nov 18 00:12:11 cust-1c kernel: [<f93bc0cb>] pmia_setsockopt+0x68/0xec [af_pmia]
Nov 18 00:12:11 cust-1c kernel: [<c05bae00>] sys_setsockopt+0x76/0x95
Nov 18 00:12:11 cust-1c kernel: [<c05bc2bf>] sys_socketcall+0x15f/0x1b7
Nov 18 00:12:11 cust-1c kernel: [<c0404ee1>] sysenter_past_esp+0x56/0x79

Solution

Following solutions can be applied (alternatively).

Disable some unused processes

Fast Monitor process:
pm.set off fcMonitor
Q752 processes:
pm.set off q752Gen q752Feeder

Reduce amount of memory allocated to fcMonitor process

If there are fast copy links, it is possible to reduce amount of memory allocated to fcMonitor process:

iset -fvalue=2500 LongParam where "name='PmBlockNum'"

Prefered solution

Upgrade to 64 bits operating system (PIC 10.1 or more).

References

<BUG:19109426> - [233726]OOM_KILLER IS TRIGGERED BY OS ON IMF 32BITS FOR HP GEN8

Attachments

This solution has no attachment