Sun Storage 7000 Unified Storage System: Mismatched SIM Firmware (FW)

Asset ID:	1-72-1542500.1
Update Date:	2016-09-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1542500.1 : Sun Storage 7000 Unified Storage System: Mismatched SIM Firmware (FW)

Applies to:

Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

Firmware version of SIM cards is brought-up as a possible issue for SAS-2 JBODS J4410.

There are two symptoms being reported: healthcheck warnings and chassis misconfiguration.

Please read the information provided to help clarify this issue.

The data disks and log disks are on the tray and the chassis misconfiguration is related to all the output scenarios posted in this document.

Look closely to firmware revision of the chassis compared to other chassis and the log disk firmware compared to other log disks and the data disk firmware compared to other data disk.

The chassis misconfiguration is related to all three possible firmware mismatch, but will report a generic fault event through Fault Management Architecture (FMA) .

Changes

Software updates are delivered as opaque binary downloads that contain some or all of:

Management and system software.
Firmware for internal components such as HBAs and network devices.
Firmware for disks and flash devices.
Firmware for external storage enclosure components.

Administrators should verify that the system is in a healthy state prior to applying the update. The details are described below in the preconditions section.

External documentation for upgrading the Appliance Kit code on a Sun ZFS Storage Appliance http://docs.oracle.com/cd/E28317_01/html/E38247/maintenance__system__updates.html#scrolltoc

The new code that is uploaded and in status 'waiting' will allow you to run a healthcheck

CLI:> maintenance system updates select ak-nas@<code version> check

You have requested to run checks associated with waiting upgrade media. This
will execute the same set of checks as will be performed as part of any upgrade
attempt to this media, and will highlight conditions that would prevent
successful upgrade. No actual upgrade will be attempted, and the checks
performed are of static system state and non-invasive. Do you wish to continue?

Are you sure? (Y/N)

Healthcheck completed. There are no issues at this time which would cause an upgrade to this media to be aborted.

Preconditions:

You can see the following with mixed firmware revisions on the SIMS below. This will not prevent a upgrade of the Sun ZFS Storage Appliance with SAS-2 J4410 Arrays.

Healthcheck warnings: C5 "System software update cannot proceed: The slots of disk shelf <name> have mixed firmware revisions <rev1> and <rev2>." SIMs report different firmware revisions.

BUI:

Maintenance - LOGS - ALERTS

CLI:

S7000> maintenance logs alerts show

Description: The slots of disk shelf <name> include revisions that are foreign to the current system software
        parts: 3753633, 3753633
        revisions: 3527, 3527
        foreign: 3527

Description: Firmware of in-use STEC model ZeusIOPs G3 disks are not uniform (E125, E12B)
chassis-<name>: Disk Shelf <name> has 2 disks at rev E125 (HDD ##, HDD ##)
chassis-<name>: Disk Shelf <name> has 4 disks at rev E12B (HDD ##, HDD ##, HDD ##, HDD

Cause

Earlier versions of the Appliance Firmware Release code reported an issue with 3527 related to a Fault Management (FMA) code = AK-8000-H7 that is not programmed in the Appliance Kit code to manage this fault message (MSG ID) for SIM firmware revisions 3527 and 3529.

New parts from manufacturing shipped by Oracle logistics with newer firmware code installed on the SIM card is the reason the Healthcheck reports as a 'Healthcheck warnings'.

This report is not fatal or cause for a downgrade of the SIM firmware.

Manufacturer	Sun Microsystems, Inc.
Model	Sun Disk Shelf (SAS-2)
Part	7041262
Serial	<name>
Revision	3527
Speed	7200rpm
Type	Storage

NOTE: This FMA event code may also be generated when the JBOD SIMs/IOMs are actually at different firmware versions.

The firmware 3527 and 3529 may also report Fault Management code = AK-8000-H7 as a misconfigured chassis, this means the chassis may have a bad disk in one of the bays reported by S.M.A.R.T disk.

Any hardware failures will prevent a upgrade of the Appliance Kit code look like this:

This issue could be related to a bad SIM

s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33> check
You have requested to run checks associated with waiting upgrade media. This
will execute the same set of checks as will be performed as part of any upgrade
attempt to this media, and will highlight conditions that would prevent
successful upgrade. No actual upgrade will be attempted, and the checks
performed are of static system state and non-invasive. Do you wish to continue?

Are you sure? (Y/N)
Healthcheck running ... / |

Healthcheck completed. Conditions were reported which would cause an attempted update to this media to abort; see the alert log for details.

s7000:maintenance logs alert> ls

entry-098 2013-4-23 15:57:20 0c7c603d-a7f8-c527-de41-997c5beb2467, System
                               software update cannot proceed: Some slots of disk shelf <name> have no firmware revision information, Minor alert
entry-099 2013-4-23 15:57:20 dc7f5849-1526-cd3f-82fa-a5ec7d57ca5e, System
                               software update cannot proceed: Disk shelf <name> has just one path, Minor alert

s7000:maintenance logs alert> select entry-098 ls
Properties:
                     timestamp = 2013-4-23 15:57:20
                          uuid = 0c7c603d-a7f8-c527-de41-997c5beb2467
                   description = System software update cannot proceed: Some slots of disk shelf <name> have no firmware revision information
                          type = Minor alert

s7000:maintenance logs alert> select entry-099 ls
Properties:
                     timestamp = 2013-4-23 15:57:20
                          uuid = dc7f5849-1526-cd3f-82fa-a5ec7d57ca5e
                   description = System software update cannot proceed: Disk shelf <name> has just one path
                          type = Minor alert

s7000:maintenance logs alert>

This issue could be related to a bad disk

CLI:> maintenance problems select problem-### ls or BUI: https://<nas-ip>:215/#maintenance/problems=<uuid>
--------------- ------------------------------------ -------------- ---------
TIME EVENT-ID MSG-ID SEVERITY
--------------- ------------------------------------ -------------- ---------
Mar 22 19:44:22 265445a8-b49d-c9ac-f32e-bfb35a6d834e AK-8000-H7 Major

Host : hosname-s7000
Platform : SUN-FIRE-X4470-SERVER Chassis_id : <nas-serial number>
Product_sn : <nas-serial number>

Fault class : defect.ak.xmlrpc.hardware.misconfigured_chassis
Affects : hc://:product-id=SUN-Storage-J4410:chassis-id=<name>:serial=<name>:part=3753659:revision=3527/ses-enclosure=1 faulted but still in service
Problem in : hc://:product-id=SUN-Storage-J4410:chassis-id=<name>:serial=<name>:part=3753659:revision=3527/ses-enclosure=1 faulted but still in service
FRU : hc://:product-id=SUN-Storage-J4410:chassis-id=<name>:serial=<name>:part=3753659:revision=3527/ses-enclosure=1 faulty
Description : The JBOD '<name>' is configured incorrectly. Refer to http://sun.com/msg/AK-8000-H7 for more information.
Response : None.

Solution

When upgrading the Appliance Kit code (from earlier versions) you may see the following warning:

C5 "System software update cannot proceed: The slots of disk shelf <name> have mixed firmware revisions <rev1> and <rev2>." SIMs report different firmware revisions.

C5 issue originated in Appliance Kit code 2010.Q3.4.2 revision reporting a problem with older SIM firmware revisions 3520 and 3524 when newer 3525 was installed on the SIMs.

This used to result in not being able to upgrade the Appliance Kit code and required a Oracle Support Tech to manually upgrade older SIM firmware revisions.

In the current SIM firmware revision 3525 associated with the Appliance Kit code in combination with the manufacturers newer 3527 and 3529 will not halt an upgrade.

For systems that have SIMs/IOMs at different firmware versions, proceed to upgrade to the latest the Appliance Firmware Release version.

(If the system is already at the latest version, please contact Oracle Support via a Service Request for assistance)

Please proceed with normal health check and upgrade the Appliance kit code. Use the 'upgrade -f' CLI command to 'force' the upgrade.

s7000:maintenance system updates> ls
Updates:

UPDATE                                   DATE                      STATUS
ak-nas@2010.08.17.4.2,1-1.37             2011-9-19 17:37:58        previous
ak-nas@2011.04.24.3.0,1-1.19             2012-5-7 19:26:43         current
ak-nas@2011.04.24.5.0,1-1.33             2012-12-13 21:28:50       waiting

s7000:maintenance system updates> select ak-nas@2011.04.24.5.0,1-1.33
s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33>
abort       date        exit        help        script      sleep       tree
check       deny        get         nslookup    shell       time        upgrade
confirm     done        getent      ping        show        traceroute

s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33> check
You have requested to run checks associated with waiting upgrade media. This
will execute the same set of checks as will be performed as part of any upgrade
attempt to this media, and will highlight conditions that would prevent
successful upgrade. No actual upgrade will be attempted, and the checks
performed are of static system state and non-invasive. Do you wish to continue?

Are you sure? (Y/N) y

Healthcheck running ... |
Healthcheck completed. There are no issues at this time which would cause an upgrade to this media to be aborted.

s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33> cd /
s7000:> maintenance logs select alert
s7000:maintenance logs alert> ls

entry-098 2013-4-23 15:36:46 e45d4c55-2f03-48cf-b7bd-e4106ab652b4, System
                               The slots of disk shelf <name> include revisions that are foreign to the current system software, Minor alert

s7000:maintenance logs alert>select entry-098 ls
Properties:
                     timestamp = 2013-4-23 09:16:29
                          uuid = e45d4c55-2f03-48cf-b7bd-e4106ab652b4
                   description = The slots of disk shelf <name> include revisions that are foreign to the current system software
                          type = Minor alert

s7000:maintenance chassis-001> ls
Properties:
                          name = 1206FMD02C
                       faulted = false
                  manufacturer = Sun Microsystems, Inc.
                         model = Sun Disk Shelf (SAS-2)
                          part = 7041262
                        serial = <name>
                      revision = 3527
                          type = storage
                           rpm = 7200
                          path = 2
                        locate = false

For issues that 'prevent an upgrade' or report 'AK-8000-H7 as a misconfigured chassis' please open a Service Request (SR) http://support.oracle.com and generate a support bundle.

http://<hostname-ip>:215
From BUI:
Maintenance -> System -> click on + sign next to Support Bundles
in the pop-up windows for Service Number put 3-xxxxxxxxx (when the support bundle completes building and uploading this SR will update automatically update the SR data has arrived.)

From CLI:
s7000:> maintenance system sendbundle 3-xxxxxxxxx (when the support bundle completes building and uploading this SR will update automatically update the SR that data has arrived.)

_______________________________________________
Manual Download/Upload of support bundle is the same
process through the BUI but the upload will fail then click the down arrow to the right of the ak.uuid.tar.gz file name, in the pop up window on the desktop select the directory you want to download the file to then goto https://support.oracle.com and upload the Support Bundle to the SR

Healthcheck running ... done
error: System is not in an upgradeable state: Healthcheck framework usage error
- healthcheck scripts exited abnormally. See alert log for more.

s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33> cd /
s7000:> maintenance logs select alert
s7000-tvp540-a-h2:maintenance logs alert> ls
entry-099 2013-9-4 11:37:52 a04faad1-f4ec-6535-f00b-a3aa5d761fff, The slots of disk shelf <name> have mixed firmware revisions 3525 and 3529, Major Fault

s7000:> maintenance system updates select ak-nas@2011.04.24.5.0,1-1.33
s7000:maintenance system updates ak-nas@2011.04.24.5.0,1-1.33> upgrade -f

References

<NOTE:1491956.1> - Sun Storage 7000 Unified Storage System: "The JBOD '{Serial Number}' is configured incorrectly" is reported
<NOTE:1535372.1> - Sun Storage 7000 Unified Storage System: Avoid configuring zpools with mixed disk sizes
<NOTE:1163816.1> - Sun Storage 7000 Unified Storage System: SAS Interconnect Module (SIM) failure with blue LED
<NOTE:1021661.1> - J4400 SIM cards randomly failing due to heartbeat timeout.
<NOTE:1433238.1> - Oracle ZFS Storage Appliance: How To Manually Upgrade SIM/IOM Firmware On SAS-1 Or SAS-2 Attached JBODS
<NOTE:1394900.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Firmware Upgrade Problems
<NOTE:1401522.1> - Sun Storage 7000 Unified Storage System: Appliance Firmware Update Health checks
<NOTE:1391405.1> - Sun Storage 7000 Unified Storage System: SIM shows unknown as model and unknown for serial number
<NOTE:1378512.1> - How to Replace A J4XXX SIM Module In A Oracle ZFS Storage ZS3-4 & Sun Storage 7000 Series :ATR:1378512.1:1
<NOTE:1307373.1> - Dual port SAS Logzilla device will show only a single SAS port operational in some cases in slot 23 of an Oracle DiskShelf JBOD.

Attachments

This solution has no attachment