Exalogic: ZFS Storage Appliance (ZFSSA) Software May Panic On "avl

Asset ID:	1-72-2301338.1
Update Date:	2017-12-01
Keywords:

Solution Type Problem Resolution Sure

Solution 2301338.1 : Exalogic: ZFS Storage Appliance (ZFSSA) Software May Panic On "avl_remove" in July 2017 PSU

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.3.170718 to 2.0.6.3.170718
Exalogic Elastic Cloud X4-2 Quarter Rack
Linux x86-64
Oracle Solaris on x86-64 (64-bit)
Oracle Virtual Server [x86-64]

Symptoms

In Exalogic July 2017 PSU, the ZFS Storage Appliance (ZFSSA) Heads may panic and crash, during NDMP backups or restores. This can occur regardless of DMA (Data Management Application) and backup types (local or remote).

Should this issue occur, it will lead to a BAD TRAP system panic, with a panic stack similar to one of the following as its signature:

unix:trap+3b2 ()
unix:cmntrap+e7 ()
genunix:avl_remove+68 ()
zfs:dlim_onexit_cleanup+2d ()
genunix:taskq_d_thread+c1 ()
unix:thread_start+8 ()

avl_last+0x1b()
dnode_sync+0x265()
dmu_objset_sync_dnodes+0x7f()
dmu_objset_sync+0x1fa()
dsl_dataset_sync+0x63()
dsl_pool_sync+0xfa()
spa_sync+0x438()
txg_sync_thread+0x244()
thread_start+8()

Changes

In July 2017 PSU, ZFSSA 2013.1.7.2 released.

Cause

This is a known ZFSSA Defect in Filesystems:

BUG 26019969 - Both heads in ZFSSA cluster are panicking on avl_remove

For further details on this, please review ZFS Storage Appliance (ZFSSA) Software may Panic During NDMP Backups and Restores (Doc ID 2264889.1)

Solution

The Bug 26019969 fix is included in the Exalogic October 2017 PSU release. It is highly recommended to upgrade to the latest PSU release to take advantage of this fix.

In case if the upgrade is not possible immediately, please contact Oracle Exalogic Support via a new Service Request to obtain the an interim one-off patch for v2.0.6.3.170718.

2.0.6.3.170718 (July 2017 PSU): 26445423

NOTE

This Patch 26445423 Upgrades the ZFS Storage Appliance (ZFSSA) to v2013.1.7.7.

Please contact Oracle Exalogic Support with a Service Request to obtain the patch for your version of Exalogic if you believe you are hitting this BUG on 2.0.6.3.170718 (July 2017 PSU).

For Exalogic Releases, please review , Exalogic Patch Set Updates (PSU) Master Note (Doc ID 1314535.1)

Firmware version details can be found at Exalogic Infrastructure Physical and Virtual Releases/PSUs – Software and Firmware Version Information (Doc ID 1530781.1)

NOTE TO SUPPORT

This is Stand-Alone-ZFSSA-Upgrade.

Prior to delivering this Patch 26445423 to customers, obtain approval from Exalogic Development.

References

<BUG:26019969> - BOTH HEADS IN ZFSSA CLUSTER ARE PANICKING ON AVL_REMOVE
<BUG:26445423> - ZFS SOFTWARE 2013.1.7.7 UPGRADE PATCH FOR EXALOGIC
<NOTE:2264889.1> - ZFS Storage Appliance (ZFSSA) Software may Panic During NDMP Backups and Restores
<NOTE:2286560.1> - Known Issues for Exalogic 2.0.6.3.X Patch Set Updates

Attachments

This solution has no attachment