Oracle ZFS Storage: Panic BAD TRAP: type=e (#pf Page fault) rp=ffffff03d5885f90 addr=30 occurred in module "zfs" due to a NULL pointer dereference seen on 2013 code

Asset ID:	1-72-1641141.1
Update Date:	2015-08-12
Keywords:

Solution Type Problem Resolution Sure

Solution 1641141.1 : Oracle ZFS Storage: Panic BAD TRAP: type=e (#pf Page fault) rp=ffffff03d5885f90 addr=30 occurred in module "zfs" due to a NULL pointer dereference seen on 2013 code

Applies to:

Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

The following type errors are seen on the array:

A Kernel Panic on the Node. "The system has rebooted after a kernel panic."

https://<nas-ip>:215/#maintenance/logs=fltlog
The system has rebooted after a kernel panic.

https://<nas-ip>:215/#maintenance/logs=alert

Description	The system has rebooted after a kernel panic.
Type	Major Defect
Impact	There may be some performance impact while the panic is copied to the savecore directory. Disk space usage by panics can be substantial.
Automated response	The failed system image has been preserved for analysis.
Recommended action	Generate a Support Bundle and contact you support provider with the id of that bundle.
Event time	2014-3-06 06:20:00
Unique Identifier	5886c184-ba89-c1f1-93d5-a1cca7b3080d
Status	This alert is associated with an active problem on the system. For more information specific to this alert, see the 'Active Problems' page.

The following panic type stack trace is seen on the array this is from the debug.sys log file:

Mar 6 06:02:00 nasarray01 ^Mpanic[cpu58]/thread=ffffff03d5886c40: Mar 6 06:02:00 nasarray01 genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ffffff03d5885f90 addr=30 occurred in module "zfs" due to a NULL pointer dereference
Mar 6 06:02:00 nasarray01 unix: [ID 100000 kern.notice] Mar 6 06:02:00 nasarray01 unix: [ID 839527 kern.notice] nfsd_kproc:
Mar 6 06:02:00 nasarray01 unix: [ID 753105 kern.notice] #pf Page fault Mar 6 06:02:00 nasarray01 unix: [ID 532287 kern.notice] Bad kernel fault at addr=0x30 Mar 6 06:02:00 nasarray01 unix: [ID 243837 kern.notice] pid=6247, pc=0xfffffffff7a0fadd, sp=0xffffff03d5886080, eflags=0x10282
Mar 6 06:02:00 nasarray01 unix: [ID 211416 kern.notice] cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f8<xmme,fxsr,pge,mce,pae,pse,de>
Mar 6 06:02:00 nasarray01 unix: [ID 624947 kern.notice] cr2: 30
Mar 6 06:02:00 nasarray01 unix: [ID 625075 kern.notice] cr3: 71df000
Mar 6 06:02:00 nasarray01 unix: [ID 625715 kern.notice] cr8: 0
Mar 6 06:02:00 nasarray01 unix: [ID 100000 kern.notice]
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         rdi: fffff6e96c08b208 rsi: fffff601c5f1a500 rdx: ffffff03d58862a0
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         rcx: ffffee32e6158000 r8:                0 r9: fffff60179d75a00
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         rax: ffffee32e6158000 rbx: fffff6e96c08b208 rbp: ffffff03d5886110
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         r10: fffffffffb85d88c r11:                0 r12: ffffff03d58862a0
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         r13:     0 r14: fffff601c5f1a500 r15: ffffff03d58862a0
Mar 6 06:02:00 nasarray01 unix: [ID 592667 kern.notice]         fsb:     0 gsb: fffff600cf57e040 ds:               4b
Mar 6 06:02:01 nasarray01 unix: [ID 592667 kern.notice]          es:    4b fs:                0 gs:              1c3
Mar 6 06:02:01 nasarray01 unix: [ID 592667 kern.notice]         trp:     e err:                0 rip: fffffffff7a0fadd
Mar 6 06:02:01 nasarray01 unix: [ID 592667 kern.notice]          cs:    30 rfl:            10282 rsp: ffffff03d5886080
Mar 6 06:02:01 nasarray01 unix: [ID 266532 kern.notice]          ss:    38
Mar 6 06:02:01 nasarray01 unix: [ID 100000 kern.notice]
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5885eb0 unix:die+105 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5885f80 unix:trap+152b ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5885f90 unix:cmntrap+e7 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886110 zfs:l2arc_read+45 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d58861f0 zfs:arc_read+229 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886250 zfs:dsl_read+34 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886370 zfs:dbuf_read_impl+186 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d58863d0 zfs:dbuf_read+108 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886490 zfs:dmu_buf_hold_array_by_dnode+1d9 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886520 zfs:dmu_buf_hold_array+6f ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886590 zfs:dmu_read_uio+4b ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886630 zfs:zfs_read+267 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d58866a0 genunix:fop_read+a9 ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886830 nfssrv:rfs3_read+4dd ()
Mar 6 06:02:01 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886950 nfssrv:common_dispatch+52b ()
Mar 6 06:02:02 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886970 nfssrv:rfs_dispatch+2d ()
Mar 6 06:02:02 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886a50 rpcmod:svc_process_request+184 ()
Mar 6 06:02:02 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886aa0 rpcmod:stp_dispatch+b9 ()
Mar 6 06:02:02 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886b30 genunix:taskq_d_thread+c1 ()
Mar 6 06:02:02 nasarray01 genunix: [ID 655072 kern.notice] ffffff03d5886b40 unix:thread_start+8 ()

Cause

This is a known issue.

The root cause is Bug 16804978 - l2arc_cached should handle b_cookie safely

Solution

Upgrade array to the current version of code 2013.1.1.5 or higher this issue is fixed in this version or above.

Attachments

This solution has no attachment