Kernel Panic at "nfs_do_writepage+0xc9/0xf4"

Asset ID:	1-72-2003303.1
Update Date:	2018-01-08
Keywords:

Solution Type Problem Resolution Sure

Solution 2003303.1 : Kernel Panic at "nfs_do_writepage+0xc9/0xf4"

Applies to:

Linux OS - Version Oracle Linux 5.10 with Unbreakable Enterprise Kernel [2.6.32] to Oracle Linux 5.9 with Unbreakable Enterprise Kernel [2.6.32] [Release OL5U10 to OL5U9]
Oracle Exalogic Elastic Cloud X2-2 Hardware
Linux x86-64
Linux x86

Symptoms

Oracle Linux with Unbreakable Enterprise Kernel (UEK) crashes with kernel panic messages below:

...
Jun 17 17:08:53 server1 kernel: kernel BUG at fs/nfs/write.c:271!
Jun 17 17:08:53 server1 kernel: invalid opcode: 0000 [#1] SMP
Jun 17 17:08:53 server1 kernel: last sysfs file: /sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map
Jun 17 17:08:53 server1 kernel: CPU 12
Jun 17 17:08:53 server1 kernel: Modules linked in: ip_vs(U) autofs4(U) ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) hidp(U) nfs(U) fscache(U) nfs_acl(U) auth_rpcgss(U) rfcomm(U) l2cap(U) bluetooth(U) rfkill(U) lockd(U) sunrpc(U) cpufreq_ondemand(U) acpi_cpufreq(U) freq_table(U) bonding(U) be2iscsi(U) ib_iser(U) iscsi_tcp(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) libcxgbi(U) cxgb3(U) mdio(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6(U) ib_uverbs(U) ib_umad(U) mlx4_vnic(U) mlx4_vnic_helper(U) mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) mlx4_core(U) video(U) output(U) sbs(U) sbshc(U) parport_pc(U) lp(U) parport(U) joydev(U) igb(U) snd_seq_dummy(U) snd_seq_oss(U) snd_seq_midi_event(U) snd_seq(U) snd_seq_device(U) snd_pcm_oss(U) snd_mixer_oss(U) snd_pcm(U) i2c_i801(U) i2c_core(U) snd_timer(U) snd(U) soundcore(U) snd_page_alloc(U) pcspkr(U) ioatdma(U) dca(U) ahci(U) shpchp(U) megaraid_sas(U) [last unloaded: microcode]
Jun 17 17:08:53 server1 kernel: Pid: 16087, comm: java Not tainted 2.6.32-400.26.3.el5uek #1 SUN FIRE X4170 M2 SERVER
Jun 17 17:08:53 server1 kernel: RIP: 0010:[<ffffffffa04f2712>] [<ffffffffa04f2712>] nfs_do_writepage+0xc9/0xf4 [nfs]
Jun 17 17:08:53 server1 kernel: RSP: 0018:ffff8815c1b75a58 EFLAGS: 00010202
Jun 17 17:08:53 server1 kernel: RAX: 0000000000000001 RBX: ffff88057406c1c0 RCX: ffff8809c4419800
Jun 17 17:08:53 server1 kernel: RDX: 0000000000010000 RSI: 000000000000000b RDI: 0000000000000286
Jun 17 17:08:53 server1 kernel: RBP: ffff8815c1b75a78 R08: ffff880000048d00 R09: 0000000000000000
Jun 17 17:08:53 server1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffea001e558e10
Jun 17 17:08:53 server1 kernel: R13: 0000000000000000 R14: ffff8815c1b75bd8 R15: 0000000000000000
Jun 17 17:08:53 server1 kernel: FS: 000000004126d940(0063) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Jun 17 17:08:53 server1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 17 17:08:53 server1 kernel: CR2: 00000006df160000 CR3: 00000003c4413000 CR4: 00000000000006e0
Jun 17 17:08:53 server1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 17 17:08:53 server1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 17 17:08:53 server1 kernel: Process java (pid: 16087, threadinfo ffff8815c1b74000, task ffff881220230780)
Jun 17 17:08:53 server1 kernel: Stack:
Jun 17 17:08:53 server1 kernel: ffffea001e558e10 ffffea001e558e10 7fffffffffffffff ffff88159366b350
Jun 17 17:08:53 server1 kernel: <0> ffff8815c1b75a98 ffffffffa04f2b17 ffffea001e558e10 ffff8815c1b75c78
Jun 17 17:08:53 server1 kernel: <0> ffff8815c1b75bb8 ffffffff810df419 ffff8815c1b75bd8 ffffffffa04f2b03
Jun 17 17:08:53 server1 kernel: Call Trace:
Jun 17 17:08:53 server1 kernel: [<ffffffffa04f2b17>] nfs_writepages_callback+0x14/0x25 [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffff810df419>] write_cache_pages+0x23a/0x35e
Jun 17 17:08:53 server1 kernel: [<ffffffffa04f2b03>] ? nfs_writepages_callback+0x0/0x25 [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffffa04f2abb>] nfs_writepages+0x9c/0xe4 [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffff810128ee>] ? apic_timer_interrupt+0xe/0x20
Jun 17 17:08:53 server1 kernel: [<ffffffffa04f39be>] ? nfs_flush_one+0x0/0xc3 [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffff810df588>] do_writepages+0x21/0x2a
Jun 17 17:08:53 server1 kernel: [<ffffffff810d89af>] __filemap_fdatawrite_range+0x64/0x77
Jun 17 17:08:53 server1 kernel: [<ffffffff810d911c>] __filemap_fdatawrite+0x1c/0x1e
Jun 17 17:08:53 server1 kernel: [<ffffffff810d9143>] filemap_fdatawrite+0x13/0x15
Jun 17 17:08:53 server1 kernel: [<ffffffff810d9163>] filemap_write_and_wait+0x1e/0x3d
Jun 17 17:08:53 server1 kernel: [<ffffffffa04e7a19>] nfs_sync_mapping+0x2d/0x3e [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffffa04e7ef7>] nfs_revalidate_mapping+0x8f/0xbb [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffffa04e5838>] nfs_file_mmap+0x6d/0x76 [nfs]
Jun 17 17:08:53 server1 kernel: [<ffffffff810f83bf>] mmap_region+0x273/0x437
Jun 17 17:08:53 server1 kernel: [<ffffffff8104328b>] ? need_resched+0x23/0x2d
Jun 17 17:08:53 server1 kernel: [<ffffffff810f8a72>] do_mmap_pgoff+0x2b0/0x315
Jun 17 17:08:53 server1 kernel: [<ffffffff810ecff1>] sys_mmap_pgoff+0x1bb/0x1e6
Jun 17 17:08:53 server1 kernel: [<ffffffff810aa26f>] ? audit_syscall_entry+0x103/0x12f
Jun 17 17:08:53 server1 kernel: [<ffffffff810166a7>] sys_mmap+0x22/0x24
Jun 17 17:08:53 server1 kernel: [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
Jun 17 17:08:53 server1 kernel: Code: 8b 15 f3 45 02 00 48 ff c0 c1 fa 02 48 63 d2 48 39 d0 7e 11 48 8d 79 48 31 f6 e8 ba bc bf e0 eb 04 0f 0b eb fe f6 43 48 02 74 04 <0f> 0b eb fe 48 89 de 4c 89 f7 e8 1c c1 ff ff 85 c0 75 0c 48 89
Jun 17 17:08:53 server1 kernel: RIP [<ffffffffa04f2712>] nfs_do_writepage+0xc9/0xf4 [nfs]
Jun 17 17:08:53 server1 kernel: RSP <ffff8815c1b75a58>
Jun 17 17:08:53 server1 kernel: ---[ end trace 09e1e8a60fa8c586 ]---
Jun 17 17:08:53 server1 kernel: Kernel panic - not syncing: Fatal exception
Jun 17 17:08:53 server1 kernel: Pid: 16087, comm: java Tainted: G D 2.6.32-400.26.3.el5uek #1
Jun 17 17:08:53 server1 kernel: Call Trace:
Jun 17 17:08:53 server1 kernel: [<ffffffff81057982>] panic+0xa5/0x162
Jun 17 17:08:53 server1 kernel: [<ffffffff81012a8e>] ? call_function_single_interrupt+0xe/0x20
Jun 17 17:08:53 server1 kernel: [<ffffffff8105764f>] ? print_oops_end_marker+0x23/0x25
Jun 17 17:08:53 server1 kernel: [<ffffffff8145a9e6>] oops_end+0xb7/0xc7
Jun 17 17:08:53 server1 kernel: [<ffffffff8101569d>] die+0x5a/0x63
Jun 17 17:08:53 server1 kernel: [<ffffffff8145a2bc>] do_trap+0x115/0x124
Jun 17 17:08:53 server1 kernel: [<ffffffff81013966>] do_invalid_op+0x9c/0xa5
Jun 17 17:08:53 server1 kernel: [<ffffffffa04f2712>] ? nfs_do_writepage+0xc9/0xf4 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffff8123891b>] ? __lookup_tag+0xb7/0x11e
Jun 17 17:08:54 server1 kernel: [<ffffffff81012bbb>] invalid_op+0x1b/0x20
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f2712>] ? nfs_do_writepage+0xc9/0xf4 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f26bc>] ? nfs_do_writepage+0x73/0xf4 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f2b17>] nfs_writepages_callback+0x14/0x25 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffff810df419>] write_cache_pages+0x23a/0x35e
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f2b03>] ? nfs_writepages_callback+0x0/0x25 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f2abb>] nfs_writepages+0x9c/0xe4 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffff810128ee>] ? apic_timer_interrupt+0xe/0x20
Jun 17 17:08:54 server1 kernel: [<ffffffffa04f39be>] ? nfs_flush_one+0x0/0xc3 [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffff810df588>] do_writepages+0x21/0x2a
Jun 17 17:08:54 server1 kernel: [<ffffffff810d89af>] __filemap_fdatawrite_range+0x64/0x77
Jun 17 17:08:54 server1 kernel: [<ffffffff810d911c>] __filemap_fdatawrite+0x1c/0x1e
Jun 17 17:08:54 server1 kernel: [<ffffffff810d9143>] filemap_fdatawrite+0x13/0x15
Jun 17 17:08:54 server1 kernel: [<ffffffff810d9163>] filemap_write_and_wait+0x1e/0x3d
Jun 17 17:08:54 server1 kernel: [<ffffffffa04e7a19>] nfs_sync_mapping+0x2d/0x3e [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffffa04e7ef7>] nfs_revalidate_mapping+0x8f/0xbb [nfs]
Jun 17 17:08:54 server1 kernel: [<ffffffffa04e5838>] nfs_file_mmap+0x6d/0x76 [nfs]
Jun 17 17:08:55 server1 kernel: [<ffffffff810f83bf>] mmap_region+0x273/0x437
Jun 17 17:08:55 server1 kernel: [<ffffffff8104328b>] ? need_resched+0x23/0x2d
Jun 17 17:08:55 server1 kernel: [<ffffffff810f8a72>] do_mmap_pgoff+0x2b0/0x315
Jun 17 17:08:55 server1 kernel: [<ffffffff810ecff1>] sys_mmap_pgoff+0x1bb/0x1e6
Jun 17 17:08:55 server1 kernel: [<ffffffff810aa26f>] ? audit_syscall_entry+0x103/0x12f
Jun 17 17:08:55 server1 kernel: [<ffffffff810166a7>] sys_mmap+0x22/0x24
Jun 17 17:08:55 server1 kernel: [<ffffffff81011df2>] system_call_fastpath+0x16/0x1b
Jun 17 16:39:08 server1 syslogd 1.4.1: restart.
...

Cause

This is a kernel defect (Bug 19011470) at nfs_flush_incompatible(), which is fixed with UEK version 2.6.32-400.34.1.el5uek and higher.

Solution

Please upgrade the UEK kernel to 2.6.32-400.34.1.el5uek or higher.

References

<BUG:19011470> - KERNEL PANIC AT NFS_DO_WRITEPAGE+0XC9/0XF4

Attachments

This solution has no attachment