Oracle ZFS Storage Appliance: Hardware Updates Failed With Error

Asset ID:	1-72-2379092.1
Update Date:	2018-04-18
Keywords:

Solution Type Problem Resolution Sure

Solution 2379092.1 : Oracle ZFS Storage Appliance: Hardware Updates Failed With Error - "expander unresponsive"

Applies to:

Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage ZS5-2 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS5-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS5-2 - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

After the AK code upgrade, the 'hardware updates' failed with an 'expander unresponsive' error as below:

<active_node1>:maintenance system updates firmware> ls
Properties:
upgrades_pending = 35
upgrades_failed = 1
upgrades_in_progress = 0

Status:
Pending
Component Current Version Status
Chassis 1520NMT009 (Expander 0) 0018 some disks only have 1 path(s) (2 paths required)
Chassis 1520NMT005 (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT005 (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT02N (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT02N (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT02R (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT02R (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT008 (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT008 (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT004 (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT004 (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT02M (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT02M (Expander 0) 0018 waiting for chassis snapshot
Chassis 1520NMT02P (Expander 1) 0018 waiting for chassis snapshot
Chassis 1520NMT02P (Expander 0) 0018 waiting for chassis snapshot
Disk <unknown> HDD 4 A122 scheduled
Disk <unknown> HDD 20 A122 scheduled
Disk <unknown> HDD 22 A122 scheduled
Disk <unknown> HDD 20 A122 scheduled
Disk <unknown> HDD 3 A122 scheduled
Disk <unknown> HDD 22 A122 scheduled
Disk <unknown> HDD 21 A122 scheduled
Disk <unknown> HDD 23 A122 scheduled
Disk <unknown> HDD 23 A122 scheduled
Disk <unknown> HDD 21 A122 scheduled
Disk <unknown> HDD 2 A122 scheduled
Disk <unknown> HDD 23 A122 scheduled
Disk <unknown> HDD 21 A122 scheduled
Disk <unknown> HDD 21 A122 scheduled
Disk <unknown> HDD 23 A122 scheduled
Disk <unknown> HDD 5 A122 scheduled
Disk <unknown> HDD 20 A122 scheduled
Disk <unknown> HDD 22 A122 scheduled
Disk <unknown> HDD 20 A122 scheduled
Disk <unknown> HDD 22 A122 scheduled

Failed
Component Current Version Status
Chassis 1520NMT009 (Expander 1) 0018 expander unresponsive

In Progress
No Updates in Progress

Changes

AK firmware upgrade of the ZS3-2 cluster nodes to Appliance Firmware Release 2013.1.7.13 (OS8.7.13).

Cause

2 chassis were reporting missing path to the tray :

chassis status:
chassis-000 [s/n] Oracle ZFS Storage ZS3-2 ok
chassis-001 [s/n] Oracle Storage DE2-24C path = 2 ok
chassis-002 [s/n] Oracle Storage DE2-24C path = 2 ok
chassis-003 [s/n] Oracle Storage DE2-24C path = 2 ok
chassis-004 [s/n] Oracle Storage DE2-24C path = 2 >>> FAULT <<<
chassis-005 [s/n] Oracle Storage DE2-24C path = 2 ok
chassis-006 [s/n] Oracle Storage DE2-24C path = 2 ok
chassis-007 [s/n] Oracle Storage DE2-24C path = 1 <<<< missing path to tray ok
chassis-008 [s/n] Oracle Storage DE2-24C path = 1 <<<< missing path to tray ok

Solution

1) It was seen that the chassis-007 [s/n] had single path via IOM0. A reseat & replace of the IOM1 of this chassis did not resolve the issue. The same error persisted :

<active node1>:maintenance system updates firmware> ls
Properties:
upgrades_pending = 35
upgrades_failed = 1
upgrades_in_progress = 0

Status:
Pending
Component Current Version Status
Chassis 007 <s/n> (Expander 0) 0018 some disks only have 1 path(s) (2 paths required)

Failed
Component Current Version Status
Chassis 007 <> 0018 expander unresponsive

2) Observation from Peer Node - All chassis had 2 paths to all disks.

3) A reboot of the active node was done. The peer head took over the resources and completed the hardware updates successfully :

<active_node2>:> maintenance system updates firmware show
Properties:
upgrades_pending = 0
upgrades_failed = 0
upgrades_in_progress = 0

Status:
Pending
No Pending Updates

Failed
No Failed Updates

In Progress
No Updates in Progress

4) IOM & disk updates were successful. All the chassis reported both the paths now from both the heads.

References

<NOTE:1394900.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Firmware Upgrade Problems

Attachments

This solution has no attachment