Oracle ZFS Storage Appliance: Pending Firmware Updates are Terminated after Microcode Upgrade

Asset ID:	1-72-1924365.1
Update Date:	2018-05-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1924365.1 : Oracle ZFS Storage Appliance: Pending Firmware Updates are Terminated after Microcode Upgrade

Applies to:

Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Oracle's 7000/ZS3/ZS4 NAS Appliance is reporting a pending number of firmware updates. The number of Update(s) Remaining has stopped decreasing over time.

In this example, we can see in the "Maintenance System" view of the BUI, that 13 updates never completed.

Firmware Updates
13 Update(s) Remaining

Similarly, all CLI utilities report that 13 updates remain.

ZFS-Head-1:> maintenance system updates show
UPDATE                                   DATE                      STATUS
ak-nas@2013.06.05.1.6,1-1.1              2014-2-26 23:06:37        previous
ak-nas@2013.06.05.2.2,1-1.1              2014-7-8 19:00:16         current

Hardware Updates: 13 remaining

Deferred updates:
The appliance is currently configured as part of a cluster. The cluster peer
may have shared resources for which deferred updates are available. After all
updates are completed, check both cluster peers for any deferred updates.

ZFS-Head-1:> raw rm.updateStatus()
result = {
      done: 0,
      failed: 0,
      total: 13
}

It may assist in troubleshooting, to determine which component never completed. Typically, it is the IOM in the disk shelf.

From the ./logs/fmtopo.txt file in the bundle, we can see that only 5 of the 18 controllers updated from revision 0010 to revision 0018, with 13 remaining.

Some methods to look at the firmware revisions of IOM's in the DE2-24P/C and 4400 Disk Trays.

    "Warning : The following command(s)/instruction(s) may result in changes to appliance settings/configuration, and could therefore result in serious damage
                if not fully understood and implemented correctly. These steps must be carried out only by a highly experienced Oracle TSC or Field Support
                engineer, trained to work at the Solaris Shell on the ZFS Storage Appliance.
               If you are at all unsure whether to proceed, please stop and consult your manager or supervisor immediately.".

# fmtopo -V | grep '^hc.*4410.*controller=.$'
# fmtopo -V | grep '^hc.*DE2.*controller=.$'

DE2-24P and DE2-24C only

# ls /dev/scsi/ses
c0t50800200014A3BFEd0 c0t50800200014A91FEd0 c0t5080020001658E7Ed0 c0t508002000167A3BEd0

# de2cli -d /dev/scsi/ses/c0t50800200014A91FEd0

Device: /dev/scsi/ses/c0t50800200014A91FEd0
Vendor: ORACLE
Product: DE2-24C (Wasabi)
Revision: 0018
Canister: IOM 00

# grep '^hc.*DE2.*controller=.$' fmtopo.txt
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02N:
      fru-serial=IMS0976147G09HE:fru-part=7043628:fru-revision=0010/ses-enclosure=0/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02N:
      fru-serial=IMS0976147G0CW0:fru-part=7043628:fru-revision=0010/ses-enclosure=0/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+24+1:chassis-serial=1347NMT034:
      fru-serial=IMS0976147G0EAB:fru-part=7043628:fru-revision=0010/ses-enclosure=1/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+24+1:chassis-serial=1347NMT034:
      fru-serial=IMS0976147G0E9L:fru-part=7043628:fru-revision=0010/ses-enclosure=1/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02K:
      fru-serial=IMS0976147G0EA4:fru-part=7043628:fru-revision=0010/ses-enclosure=2/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02K:
      fru-serial=IMS0976147G0E8X:fru-part=7043628:fru-revision=0010/ses-enclosure=2/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02J:
      fru-serial=IMS0976147G0E9T:fru-part=7043628:fru-revision=0010/ses-enclosure=3/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02J:
      fru-serial=IMS0976147G0E8G:fru-part=7043628:fru-revision=0010/ses-enclosure=3/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02M:
      fru-serial=IMS0976147G0EFJ:fru-part=7043628:fru-revision=0010/ses-enclosure=4/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02M:
      fru-serial=IMS0976147G0EHE:fru-part=7043628:fru-revision=0010/ses-enclosure=4/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02L:
      fru-serial=IMS0976147G0E7X:fru-part=7043628:fru-revision=0010/ses-enclosure=5/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02L:
      fru-serial=IMS0976147G0E7A:fru-part=7043628:fru-revision=0010/ses-enclosure=5/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02T:
      fru-serial=IMS0976147G0E9G:fru-part=7043628:fru-revision=0010/ses-enclosure=6/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02T:
      fru-serial=IMS0976147G0E8H:fru-part=7043628:fru-revision=0018/ses-enclosure=6/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02P:
      fru-serial=IMS0976147G0DA6:fru-part=7043628:fru-revision=0018/ses-enclosure=7/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02P:
      fru-serial=IMS0976147G0DA0:fru-part=7043628:fru-revision=0018/ses-enclosure=7/controller=1
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02R:
      fru-serial=IMS0976147G0CVE:fru-part=7043628:fru-revision=0018/ses-enclosure=8/controller=0
hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=32312292+25+1:chassis-serial=1347NMT02R:
      fru-serial=IMS0976147G0CV2:fru-part=7043628:fru-revision=0018/ses-enclosure=8/controller=1

Changes

Typically the problem manifests itself after a microcode upgrade.

Cause

The most probable cause is that the Clustered Appliance was returned to its AKCS_CLUSTERED state before all firmware upgrades were allowed to complete.

The "rule based" upgrade will not proceed when the nodes are reporting as AKCS_CLUSTERED.

You have to return the cluster to a failover state, (AKCS_OWNER/AKCS_STRIPPED), to trigger the continuance of the upgrades.

Solution

The solution is to put the cluster back into a failover state.

There are many ways to do this. It may be performed on either node.

In this example, the cli reboot option is used from node 1.

Once the cluster returns to an AKCS_OWNER/AKCS_STRIPPED state, the firmware updates will continue (pending updates will decrease to 0)

ZFS-Head-1:> configuration cluster show
Properties:
               state = AKCS_CLUSTERED
               description = Active
               peer_asn = 8aaf1551-cf41-c158-ec6b-c53a5dcea3ab
               peer_hostname = ZFS-Head-2
               peer_state = AKCS_CLUSTERED
               peer_description = Active

ZFS-Head-1> maintenance system reboot
Continuing will immediately take over the resources assigned to the cluster
peer. This may result in clients experiencing a slight delay in service.
Are you sure? (Y/N)Y

ZFS-Head-1:>:> configuration cluster show
Properties:
               peer_state = AKCS_STRIPPED
peer_description = Ready (waiting for failback)
               description = Active (takeover completed)
               peer_asn = 8aaf1551-cf41-c158-ec6b-c53a5dcea3ab
               peer_hostname = ZFS-Head-2
               peer_state = AKCS_OWNER
             peer_description = Ready (takeover completed)

You can now monitor the pending updates.

Check for relevancy - 10-May-2018

Attachments

This solution has no attachment