Oracle ZFS Storage Appliance: Head failed to join cluster after upgrade to 2013 firmware

Asset ID:	1-72-1951795.1
Update Date:	2018-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1951795.1 : Oracle ZFS Storage Appliance: Head failed to join cluster after upgrade to 2013 firmware

Applies to:

Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-BA - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Head B stuck at Joining Cluster.

Head A CLI dropped to emergency shell.

Changes

Upgraded from 2011 firmware to 2013 firmware

Cause

Head A Appliance Kit Daemon (AKD) is stuck with the following stack, while Head B is trying to join cluster.

> a::findstack -v
stack pointer for thread a: fcd8de68
[ fcd8de68 libc_hwcap1.so.1`__lwp_park+0x19() ]
fcd8de98 libc_hwcap1.so.1`cond_wait_queue+0x63(cc074ec, cc074bc, 0, feefd920)
fcd8ded8 libc_hwcap1.so.1`__cond_wait+0x89(cc074ec, cc074bc, fcd8def8, feefd9e1)
fcd8def8 libc_hwcap1.so.1`cond_wait+0x27(cc074ec, cc074bc, fcd8df38, feefda27)
fcd8df18 libc_hwcap1.so.1`pthread_cond_wait+0x24(cc074ec, cc074bc, f0059674, f0149a31)
fcd8df58 libak.so.1`ak_taskq_dispatch+0x49(cc07488, f0059674, ce1cc9c, f0059711)
fcd8df88 libak.so.1`ak_ad_autojoin_upon_failback+0x4d(8136c08, f01dda78, fcd8eeb8, f005b724)
fcd8eeb8 libak.so.1`ak_ad_rm_update+0x27a(8136c08, f01dda78, aebab47, 120e7530, d512b88, aebab47)
fcd8ef18 libak.so.1`ak_rm_elem_update+0x231(8136c08)
fcd8ef98 libak.so.1`ak_rm_update_thread+0x360(8136c08, 0, 0, f0149fd9)
fcd8efc8 libak.so.1`ak_thread_start+0x6a(8582c60, fefa2000, fcd8efe8, fef03719)
fcd8efe8 libc_hwcap1.so.1`_thrp_setup+0x9d(fcd93140)
fcd8eff8 libc_hwcap1.so.1`_lwp_start(fcd93140, 0, 0, 0, 0, 0)
>

This is related to the following bug:

BUG 19324720 - Cannot join cluster after upgrade to 2013.1.2.0 IDR if multiple stashes preexist

Solution

Please contact Oracle Support for recovery.

Fix is included in Appliance Firmware Release 2013.1.4.0

Shutdown head B.
Stop akd on head A, move the Active Directory stash object out of the way and re-enable akd.
Start Head B.

After Head B joined cluster. Re-configure Active Directory

References

<BUG:19324720> - CANNOT JOIN CLUSTER AFTER UPGRADE TO 2013.1.2 IF MULTI-AD STASH OBJECTS PREEXIST

Attachments

This solution has no attachment