ILOM Produces "Warning: /coredump Is Nearly Full" on X5-2(L) and X4-2(L) Platforms When Using OHMP 2.3.1

Asset ID:	1-72-2051536.1
Update Date:	2018-03-26
Keywords:

Solution Type Problem Resolution Sure

Solution 2051536.1 : ILOM Produces "Warning: /coredump Is Nearly Full" on X5-2(L) and X4-2(L) Platforms When Using OHMP 2.3.1

Applies to:

Oracle Server X5-2L - Version All Versions to All Versions [Release All Releases]
Oracle Server X5-2 - Version All Versions to All Versions [Release All Releases]
Sun Server X4-2L - Version All Versions to All Versions [Release All Releases]
Sun Server X4-2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

The SP/ILOM may become unreachable over ILOM Web-BUI and over ssh.

Some ILOM processes may be logging core dumps and /coredump directory reported as nearly full.

-rw-r--r--    1 root     root        32852 Aug 11 10:39 /coredump/_core-snapshot_XXXXX.lumain#lumain.core.965.tgz
-rw-r--r--    1 root     root       449555 Aug 11 10:49 /coredump/XXXXX.lumain#lumain.core.965.gz
-rw-r--r--    1 root     root       704512 Aug 11 10:25 /coredump/XXXXX.pod#evtmgr_rcv.core.20166.gz
-rw-r--r--    1 root     root       467941 Aug 11 10:42 /coredump/XXXX.pod#evtmgr_rcv.core.20329.gz

-> show -script /X/logs/event/list

Event
ID     Date/Time                 Class     Type      Severity
----- ------------------------ -------- -------- --------
1663   Tue Aug 11 10:36:12 2015 Fault     Warning   minor
       Warning: /coredump is nearly full

_SEL (System Event Log):

62451 Wed Sep 23 08:40:31 2015 Fault Warning minor
Warning: /coredump is nearly full
62423 Wed Sep 23 08:30:31 2015 Fault Warning minor
Warning: /coredump is nearly full
62395 Wed Sep 23 08:10:30 2015 Fault Warning minor
Warning: /coredump is nearly full
62340 Wed Sep 23 07:54:00 2015 Fault Warning minor
Warning: /coredump is nearly full
62312 Wed Sep 23 07:41:00 2015 Fault Warning minor
Warning: /coredump is nearly full
62284 Wed Sep 23 07:28:00 2015 Fault Warning minor
Warning: /coredump is nearly full

Also, there are many cores generated:

_ls_-l_@coredump@*core*.out

-rw-r--r-- 1 root root 1088684 Oct 6 12:14 /coredump/ILOM.pod#evtmgr_rcv.core.14316.gz
-rw-r--r-- 1 root root 1107688 Oct 6 12:26 /coredump/ILOM.pod#evtmgr_rcv.core.14444.gz
-rw-r--r-- 1 root root 1149023 Oct 6 12:39 /coredump/ILOM.pod#evtmgr_rcv.core.14577.gz
-rw-r--r-- 1 root root 1132678 Oct 6 12:52 /coredump/ILOM.pod#evtmgr_rcv.core.14705.gz
-rw-r--r-- 1 root root 1089019 Oct 6 13:04 /coredump/ILOM.pod#evtmgr_rcv.core.14832.gz
-rw-r--r-- 1 root root 1087092 Oct 6 13:18 /coredump/ILOM.pod#evtmgr_rcv.core.14954.gz

Changes

ILOM may become unresponsive on Sun Server X4-2(L) and Oracle Sever X5-2(L) caused by OHMP 2.3.1.

Cause

Issue caused in combination with OHMP 2.3.1 and OS network interfaces which are in bonding mode.

Solution

Workarounds:

1. Stop/remove OHMP hwmgmt service at OS

2. Don't configure OS network interface to bonding mode.

3. Temporally apply the workaround of stopping hwmgmtd:

Stop OHMP2.3.1 hwmgmt service in OS: # stop /etc/init.d/hwmgmtd

Solution:

To fix this issue, we have to upgrade the OHMP to version 2.3.3 or newer.

To clear /coredump directory, please have a look on MOS document 2130881.

ILOM shows /coredump is almost full On Servers With Service Processors Using ILOM Firmware prior to 3.2.5.x (Doc ID 2130881.1)

References

<NOTE:1009698.1> - How to Perform Platform Configuration, Management, and Data Collection Tasks with ipmitool on Oracle X86 Systems [Video]
<BUG:20315857> - ILOM POD#EVTMGR/LUMAIN COREDUMP WITH OHMP FISHWRAP XML DUPLICATED SUN-ID
<NOTE:1448069.1> - How to run an ILOM Snapshot on a Sun/Oracle X86 System
<NOTE:2130881.1> - ILOM shows /coredump is almost full On Servers With Service Processors Using ILOM Firmware prior to 3.2.5.x

Attachments

This solution has no attachment