Sun Storage 7000 Unified Storage System: Hardware Firmware Upgrades are not applied on Cluster node after Upgrade

Asset ID:	1-72-1553165.1
Update Date:	2018-05-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1553165.1 : Sun Storage 7000 Unified Storage System: Hardware Firmware Upgrades are not applied on Cluster node after Upgrade

Applies to:

Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

Problem Description: There are 4 remaining hardware upgrades on one of the cluster nodes (the passive one - node B) after upgrade. For this example, the upgrade was to 2011.1.6.0

No error message, but the "maintenance system updates" shows the following:

<node_B>:maintenance system updates> show
  Updates:
  UPDATE DATE STATUS
  ak-nas@2011.04.24.5.0,1-1.33 2012-12-13 21:28:50 previous
  ak-nas@2011.04.24.6.0,1-1.36 2013-4-22 22:15:56 current
  Hardware Updates: 4 remaining
  Deferred updates:
  The appliance is currently configured as part of a cluster. The cluster peer
  may have shared resources for which deferred updates are available. After all
  updates are completed, check both cluster peers for any deferred updates.

From CLI, it can be seen that there is no progress on the hardware upgrades.

After the upgrade to 2011.1.6.0, the hardware upgrades seem to be hanging, no progress has been made for more than 3 hours.

This is the output from CLI:

<node_B>:maintenance system updates> show
Updates:
UPDATE DATE STATUS
ak-nas@2011.04.24.5.0,1-1.33 2012-12-13 21:28:50 previous
ak-nas@2011.04.24.6.0,1-1.36 2013-4-22 22:15:56 current
Hardware Updates: 4 remaining
Deferred updates:
The appliance is currently configured as part of a cluster. The cluster peer
may have shared resources for which deferred updates are available. After all
updates are completed, check both cluster peers for any deferred updates.

Checking the logs/rm.ak.txt log from the support bundle, it can be seen that the drives are not getting upgraded, which explains why there is no progress shown in CLI:

Sun May 12 06:25:08 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:10 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:13 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:15 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:15 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:16 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:16 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:16 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:16 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:17 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:17 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:17 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:18 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:18 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:18 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:19 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:19 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:20 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:20 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:20 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e
Sun May 12 06:25:21 2013: update_remote not propagating update of ak:/object/ca936330-27b1-cbf4-8ef1-9dbc1a273e8e

Changes

Cause

The cause of this problem is the fact that passive node (node B) has no resources.

All the resources are owned by the master peer (node A).

As long as the passive node has no resources, it cannot upgrade any hardware and the 4 remaining hardware updates are not applied

Solution

Please ensure that the correct cluster upgrade procedure has been followed - See Document 1447284.1 (Sun Storage 7000 Unified Storage System: How to upgrade a clustered system).

This issue has not yet been fully investigated, or the root cause found.

Please open a Service Request for assistance from Oracle NAS Support.

Oracle NAS Support Engineer: Please engage RPE by opening a new bug and provide assistance in analysing any system suffering from this issue

Other References:

Customer Service Manual System Updates Section

Official FISHworks Software Updates wiki Page

Check for relevancy - 10-May-2018

References

<NOTE:1394900.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Firmware Upgrade Problems
<NOTE:1447284.1> - Oracle ZFS Storage Appliance: How to upgrade a clustered system

Attachments

This solution has no attachment