Diameter Signaling Router (DSR): MP Rebooted And Syscheck Shows Error – “ Kdump: FAILURE:: MINOR::5000000020000000 -- Server Kernel Dump File Detected”

Asset ID:	1-72-2390825.1
Update Date:	2018-04-25
Keywords:

Solution Type Problem Resolution Sure

Solution 2390825.1 : Diameter Signaling Router (DSR): MP Rebooted And Syscheck Shows Error – “ Kdump: FAILURE:: MINOR::5000000020000000 -- Server Kernel Dump File Detected”

Applies to:

BNS Platform Hardware - Version DSR 7.3.0 and later
Tekelec

Symptoms

MP rebooted and following error is reported in syscheck

Running modules in class system...
* kdump: FAILURE:: MINOR::5000000020000000 -- Server Kernel Dump File Detected
One or more module in class "system" FAILED

syscheck -v shows

kdump: Checking for kernel dump files.

kdump: There are kernel dump files on the system:

kdump: KDUMP: /var/TKLC/kdump/127.0.0.1-2018-03-10-23:29:58/vmcore-dmesg.txt

kdump: KDUMP: /var/TKLC/kdump/127.0.0.1-2018-03-10-23:29:58/vmcore-incomplete

* kdump: FAILURE:: MINOR::5000000020000000 -- Server Kernel Dump File Detected

mem: Skipping expected memory check.

mem: Minimum expected memory found.

mem: 67519582208 bytes (~64392 Mb) of RAM installed.

One or more module in class "system" FAILED

Changes

Cause

This is a hardware issue.

The vmcore-dmesg.txt file indicates the kernel panic was due to some CPU machine check failures.

[Hardware Error]: CPU 31: Machine Check Exception: 0 Bank 9: cc000206000800c1
[Hardware Error]: TSC 0 ADDR ad3af0000 MISC 900004000400c8c
[Hardware Error]: PROCESSOR 0:206d7 TIME 1513238405 SOCKET 1 APIC 2f microcode 1808
[Hardware Error]: Run the above through 'mcelog --ascii'
[Hardware Error]: CPU 15: Machine Check Exception: 0 Bank 9: cc000206000800c1
[Hardware Error]: TSC 0 ADDR ad3cf0000 MISC 900004000400c8c
[Hardware Error]: PROCESSOR 0:206d7 TIME 1513238405 SOCKET 1 APIC 2e microcode 1808
[Hardware Error]: Run the above through 'mcelog --ascii'
[Firmware Warn]: ERST: Firmware does not respond in time
[Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 1: bf80000000000115
[Hardware Error]: RIP Unable to render embedded object: File (INEXACT) not found. 10:
{schedule+0x286/0x3a0}
[Hardware Error]: TSC d1ac407cf31db7 ADDR 466f6380 MISC 86
[Hardware Error]: PROCESSOR 0:206d7 TIME 1520724525 SOCKET 0 APIC e microcode 1808
[Hardware Error]: Run the above through 'mcelog --ascii'
[Hardware Error]: CPU 23: Machine Check Exception: 5 Bank 1: bf80000000000115
[Hardware Error]: RIP Unable to render embedded object: File (INEXACT) not found. 10:
{intel_idle+0xd1/0x1b0}
[Hardware Error]: TSC d1ac407cf31dd0 ADDR 466f6380 MISC 86
[Hardware Error]: PROCESSOR 0:206d7 TIME 1520724525 SOCKET 0 APIC f microcode 1808
[Hardware Error]: Run the above through 'mcelog --ascii'
[Hardware Error]: CPU 3: Machine Check Exception: 5 Bank 0: 0000000000000000
[Hardware Error]: RIP Unable to render embedded object: File (INEXACT) not found. 10: {intel_idle+0xd1/0x1b0}
[Hardware Error]: TSC d1ac407cf32479
[Hardware Error]: PROCESSOR 0:206d7 TIME 1520724525 SOCKET 0 APIC 6 microcode 1808
[Hardware Error]: Run the above through 'mcelog --ascii'
[Hardware Error]: Some CPUs didn't answer in synchronization
[Hardware Error]: Machine check: Invalid
Kernel panic - not syncing: Fatal machine check on current CPU

Solution

It is recommended to replace the blade

References

<NOTE:1682110.1> - BNS Platform Hardware Replacement Template

Attachments

This solution has no attachment