Kernel Crashes Cause an X6-2 System to Reboot Repeatedly.

Asset ID:	1-72-2241139.1
Update Date:	2017-12-10
Keywords:

Solution Type Problem Resolution Sure

Solution 2241139.1 : Kernel Crashes Cause an X6-2 System to Reboot Repeatedly.

Applies to:

Oracle Server X6-2 - Version All Versions and later
Linux OS - Version Enterprise Linux 3.0 and later
Exadata X6-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

An X6-2 server crashes and reboots with varying stack traces which do not match any known fault.

This is one example of a stack trace caused by this issue, however it is not necessarily representative:

PID: 231378 TASK: ffff8822fd14c480 CPU: 70 COMMAND: "automount"

#0 [ffff880dffd07ab0] machine_kexec at ffffffff8103ac19
#1 [ffff880dffd07b20] crash_kexec at ffffffff810b97e3
#2 [ffff880dffd07bf0] oops_end at ffffffff8150f958
#3 [ffff880dffd07c20] die at ffffffff8101878b
#4 [ffff880dffd07c50] do_general_protection at ffffffff8150f7a2
#5 [ffff880dffd07c80] general_protection at ffffffff8150eea5
[exception RIP: __ticket_spin_lock+9]
RIP: ffffffff81040fd9 RSP: ffff880dffd07d38 RFLAGS: 00010286
RAX: 0000000000010000 RBX: ffff882f21050540 RCX: 0000000000000028
RDX: dead000000100070 RSI: ffff883fbdda0ec0 RDI: dead0000001000cc
RBP: ffff880dffd07d38 R8: ffffffff819a04a8 R9: 0000000000000040
R10: 000000000001f970 R11: 0000000000000246 R12: dead0000001000cc
R13: ffff882bead849c0 R14: ffff883fbdda0f6
0 R15: ffff8830bb6038b8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#6 [ffff880dffd07d40] _raw_spin_lock at ffffffff8150e70e
#7 [ffff880dffd07d50] get_next_positive_subdir at ffffffffa0582ae9 [autofs4]
#8 [ffff880dffd07da0] autofs4_expire_indirect at ffffffffa05831bb [autofs4]
#9 [ffff880dffd07e10] autofs4_do_expire_multi at ffffffffa05834c5 [autofs4]
#10 [ffff880dffd07e60] autofs4_expire_multi at ffffffffa058352a [autofs4]
#11 [ffff880dffd07e90] autofs4_root_ioctl_unlocked at ffffffffa0580d78
[autofs4]
#12 [ffff880dffd07ed0] autofs4_root_ioctl at ffffffffa0580f51 [autofs4]
#13 [ffff880dffd07ee0] do_vfs_ioctl at ffffffff8117fd2c
#14 [ffff880dffd07f30] sys_ioctl at ffffffff81180091
#15 [ffff880dffd07f80] system_call_fastpath at ffffffff81516ba2
RIP: 00007fa109972a77 RSP: 00007fa1082dd610 RFLAGS: 00000216
RAX: 0000000000000010 RBX: ffffffff81516ba2 RCX: 00007fa1082ddab0
RDX: 00007fa1082e2d0c RSI: 0000000040049366 RDI: 0000000000000009
RBP: 0000000000000000 R8: 00007fa10b120900 R9: 00007fa1082e2d0c
R10: 000000000001f970 R11: 0000000000000246 R12: 00007fa1082e2d0c
R13: 0000000040049366 R14: 0000000000000002 R15: 0000000000000009
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b

This issue only applies to X6-2 server hardware systems.

Cause

There is a known microcode issue with some releases of X6-2 server which can cause unpredictable kernel crashes.

See Bug 25447921 and Bug 24926423

Solution

1. Backup the OS.
2. Install the latest microcode_ctl package from ULN.
2. Reboot the server.
3. Check if the issue is fixed.

Exadata database servers facing this issue should update to 12.1.2.3.5 or 12.2.1.1.1, both of which contain the fix to this issue.

References

<BUG:25392839> - UPDATE INTEL MICROCODE TO 20161104 TO FIX ISSUES IN HASWELL/BROADWELL
<BUG:24811462> - BUG: SCHEDULING WHILE ATOMIC: KWORKER - RIP TICK_NOHZ_STOP_SCHED_
<BUG:19906976> - BUG: SCHEDULING WHILE ATOMIC: KWORKER/0:1/0/0X10000100
<NOTE:1448073.1> - Engineered Systems Upgrade/Supportability Reference Doc (INTERNAL ONLY)
<BUG:25447921> - SCHEDULING WHILE ATOMIC WITH RIP FINISH_TASK_SWITCH+0X5C/0X100

Attachments

This solution has no attachment