Sun Storage 7000 Unified Storage System: Appliance Firmware Update Health checks

Asset ID:	1-75-1401522.1
Update Date:	2017-10-11
Keywords:

Solution Type Troubleshooting Sure

Solution 1401522.1 : Sun Storage 7000 Unified Storage System: Appliance Firmware Update Health checks

Applies to:

Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7720 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Purpose

This document discusses general Health Check failures faced as a part of pre-firmware upgrade.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Troubleshooting Steps

Symptoms:

Health Check are performed automatically as a pre-upgrade task to prevent any issues during firmware upgrade.

If the health check fails then a message will be such as:

7000:> maintenance system updates select ak-nas@2010.08.17.4.1,1-1.37 upgrade

The selected software update requires a system reboot in order to take effect.

The system will automatically reboot at the end of the update process. The update will take several minutes. At any time during this process, you can cancel the update with [Control-C].

Are you sure? (Y/N)
error: System is not in an upgradeable state: prerequisite healthcheck reports
problems. See alert log for more.

In this case please check the alert logs to identify and isolate the issue. To check the Alert log:

BUI:

Maintenance - LOGS - ALERTS

CLI:

S7000> maintenance logs alerts show

Resolutions:

** Please note SIM re-seat procedure mentioned below in the Resolution section may lead to loss of data availability and should only be done if issue is seen only with one SIM. In case issue is seen with more than one SIM or if you are not sure how to proceed further please contact Oracle for further assistance.

ID	Healthcheck Failure	Alert Log Message	Resolution
B1	Any Disk Shelf SAS Interface Module (SIM) slot is absent.	"System software update cannot proceed: Slot <label> in disk shelf <name> is reported as absent"	1) Check whether physical status of SIM is showing Blue or white LED 2) Perform the Disk Shelf Cable Check Procedure 3) If the issue is still present, perform the Disk Shelf SIM Re-seat Procedure 4) If the issue is still present, contact Oracle Support for service or to replace the Disk Shelf SIM
B2	Any Disk Shelf SIM slot is faulted.	"System software update cannot proceed: Slot <label> in disk shelf <name> is faulted"	1) Check whether SIM is visible on other head in case of cluster. Ensure and compare the SIM firmware version is same. 2) Check whether physical status of SIM is showing amber LED 3) Perform the Disk Shelf Cable Check Procedure 4) If the issue is still present, perform the Disk Shelf SIM Re-seat Procedure 5) If the issue is still present, contact Oracle Support for service or to replace the Disk Shelf SIM
C1	Any Disk Shelf SIM slot is missing firmware revision information.	"System software update cannot proceed: Some slots of disk shelf <name> have no firmware revision information"	1) Perform the Disk Shelf SIM Re-seat Procedure 2) If the issue is still present, contact Oracle Support to update SIM Firmware or to replace the Disk Shelf SIM
C2	Disk Shelf SIM slots report different part numbers.	"System software update cannot proceed: The slots of disk shelf <name> have non-uniform part numbers"	1) Perform the Disk Shelf SIM Re-seat Procedure 2) If the issue is still present, contact Oracle Support for service or to replace the Disk Shelf SIM
C5	Disk Shelf SIM slots report different firmware revisions.	"System software update cannot proceed: The slots of disk shelf <name> have mixed firmware revisions <rev1> and <rev2>"	1) Contact Oracle Support to update SIM Firmware or to replace the Disk Shelf SIM
E1	Any Disk Shelf does not have two paths.	"System software update cannot proceed: Disk shelf <name> has <just one path \| zero paths>"	1) Check whether SIM is visible on other head in case of cluster. Ensure and compare the SIM firmware version is same. 2) Check whether physical status of SIM is showing amber LED 3) Perform the Disk Shelf Cable Check Procedure 4) If the issue is still present, perform the Disk Shelf SIM Re-seat Procedure 5) If the issue is still present, contact Oracle Support for service or to replace the Disk Shelf SIM
E2	Any Disk Shelf path is not online.	"System software update cannot proceed: Disk shelf <name> path <pathname> is <state>"	1) Check whether SIM is visible on other head in case of cluster. Ensure and compare the SIM firmware version is same. 2) Check whether physical status of SIM is showing amber LED 3) Perform the Disk Shelf Cable Check Procedure 4) If the issue is still present, perform the Disk Shelf SIM Re-seat Procedure 5) If the issue is still present, contact Oracle Support for service or to replace the Disk Shelf SIM
E3	Any Disk or Log device that is configured in a pool does not have two paths.	"System software update cannot proceed: Pool <data \| log> disk <label> in disk shelf <name> has <just one path \| zero paths>"	1) Check whether the SIM FW 3525 2) Perform the Disk Re-seat Procedure Note: Do not reseat if system is running below firmware AK2011.1.8 and involve Oracle Support. 3) If the issue is still present, contact Oracle Support for service or to replace the Disk

In case any other hardware is shown as faulted please raise a service request with the latest support bundle to get the hardware diagnosed and replaced. To generate and upload a support bundle please see

Document 1019887.1 Sun Storage 7000 Unified Storage System: How to collect a supportbundle using the BUI or CLI

Reseating Disk may lead to system hang as mentioned in Bug: 16193307/18641650 if system is running below firmware AK2011.1.8

Further Information:

Read the Release Notes for the appropriate version of the appliance software, the links to the Release Notes for each of the versions can be found here:

MOS Document ID 2021771.1 - Oracle ZFS Storage Appliance: Software Updates

General Information on installing system updates can be found here:

Oracle ZFS Storage Appliance Customer Service Manual (Release OS8.7.x) - http://docs.oracle.com/cd/E79446_01/pdf/E79459.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release OS8.7.0) - http://docs.oracle.com/cd/E78901_01/pdf/E78914.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release OS8.6.x) - http://docs.oracle.com/cd/E76475_01/pdf/E76485.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release OS8.6.0) - http://docs.oracle.com/cd/E71909_01/pdf/E71921.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release 2013.1.5.0) - http://docs.oracle.com/cd/E69262_01/pdf/E69256.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release 2013.1.4.0) - http://docs.oracle.com/cd/E56047_01/pdf/E56078.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release 2013.1.3.0) - http://docs.oracle.com/cd/E56021_01/pdf/E55850.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release 2013.1.2.0) - http://docs.oracle.com/cd/E51475_01/pdf/E52874.pdf

Oracle ZFS Storage Appliance Customer Service Manual (Release 2013.1.0) - http://docs.oracle.com/cd/E27998_01/pdf/E48492.pdf

Sun ZFS Storage 7120, 7320, and 7420 Appliance Customer Service Manual (Release 2011.1.5) - http://docs.oracle.com/cd/E28317_01/pdf/E38247.pdf

Sun Storage 7000 Unified Storage System Service Manual - http://docs.oracle.com/cd/E19548-01/pdf/821-1388.pdf

Back to Document 1394900.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot Firmware Upgrade problems.

Before following any of the below Resolution procedures it is still recommended to contact Oracle Support:

Resolution Procedures (Example) :

1. Disk Shelf Cable Check Procedure

Step 1. Check each SAS cable connection on the back of each Disk Shelf and make sure the corresponding green Port LED is illuminated. It can be stable or flashing.
Step 2. If green Port LED is off, re-seat both ends of the SAS cable as needed until the green Port LED is illuminated.
Step 3. Replace cable if needed.

2. SIM Verification Procedure

Step 1. Use the AK BUI or the following AK CLI command to identify which JBOD has the issue.

7000:> maintenance hardware select chassis-001 get
name = 1101FMD02T
faulted = false
manufacturer = Sun Microsystems, Inc.
model = Sun Disk Shelf (SAS-2)
part = Storage-J4410
serial = 1101FMD02T
revision = 3524
type = storage
path = 2
locate = false

Step 2. Use the AK BUI or the following AK CLI commands to identify which SIM slot has the issue. For example, in this case SIM 0 is faulted and has the wrong revision number.

7000:> maintenance hardware select chassis-001 select slot select slot-000 show
Properties:
label = SIM 0
present = true
faulted = true
manufacturer = Sun Microsystems, Inc.
part = 3753633
serial = 2029QTF-1014QC10FA
revision = 3520
locate = false

7000:> maintenance hardware select chassis-001 select slot select slot-001 show
Properties:
label = SIM 1
present = true
faulted = false
manufacturer = Sun Microsystems, Inc.
part = 3753633
serial = 2029QTF-1014QC10D5
revision = 3524
locate = false

3. Disk Shelf SIM Re-seat Procedure

Step 1. Perform the SIM Verification Procedure to find the SIM slot that has the issue.
Step 2. Remove the SIM unit from the Disk Shelf.
Step 3. Wait two minutes.
Step 4. Insert the SIM unit into the Disk Shelf.
Step 5. Wait for green SIM LED on back of Disk Shelf to illuminate.

4. Disk Re-seat Procedure

Step 1. Remove the disk from the Disk Shelf.
Step 2. Wait 30 seconds.
Step 3. Insert the disk into the Disk Shelf.
Step 4. Wait for green LED on front of disk to illuminate. It can be stable or flashing.

End

References

<BUG:16193307> - SYSTEM IS SLUGGISH AFTER PEER REBOOT WITH "UNEXPECTEDLY LOST SAS ZONE LOCK"
<BUG:18641650> - DRIVE REPLACEMENT DID NOT HAPPEN WITH BAD PHY
<NOTE:1019887.1> - Sun Storage 7000 Unified Storage System: How to Collect a Support Bundle using the BUI or CLI
<NOTE:1021661.1> - J4400 SIM cards randomly failing due to heartbeat timeout.
<NOTE:1307224.1> - FAB: Standard: Reactive: RW2 SIM heartbeat timeout (aka White Light Special) scenario with RW2, which will cause one of the SIMs to go offline and show up as one of the paths to the JBOD removed.
<NOTE:1307373.1> - Dual port SAS Logzilla device will show only a single SAS port operational in some cases in slot 23 of an Oracle DiskShelf JBOD.
<NOTE:1394900.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Firmware Upgrade Problems

Attachments

This solution has no attachment