Description

A known issue with vxWorks RAID controller firmware for Sun StorageTek arrays (as listed in Section 2) may cause drives associated with host/IO volumes to experience write failures when the controllers reboot. This issue can occur after approximately 828.5 days of uptime, when vxWorks (by default) is scheduled for a simultaneous auto-reboot of the controllers.

Occurrence

This issue can occur on the following platforms:

• Sun StorageTek 2510 Array
• Sun StorageTek 2530 Array
• Sun StorageTek 2540 Array
• Sun StorageTek 6140 Array
• Sun StorageTek 6180 Array
• Sun StorageTek 6540 Array
• Sun StorageTek 6580 Array
• Sun StorageTek 6780 Array
• Sun StorageTek Flexline 380 Array

running vxWorks Controller Firmware versions 6.70.xx or earlier.

To determine the version of firmware on the controller, please view the Common Array Manager (CAM) Storage System Summary page of the CAM host managing the array.

There is a timer in vxWorks (vxAbsTicks) that is a double word long 0x00000000 (a 32 bit number). cfgMonitorTask monitors this offset to avoid drive failure during IO to the disk, and reboots the controller once the vxAbsTicks reaches 0xff000000. When this timer rolls over from 0xffffffff to 0x00000000 (approximately 828.5 days) there is a possibility that if host I/O volumes exist, the associated drives will be failed with a write failure.

Symptoms

RAID arrays using software mirroring to mirror data between the two arrays perform an unscheduled simultaneous reboot at nearly the same time (approximately 828.5 days uptime), causing a write failure.

Workaround

To avoid the (unscheduled) controller auto-reboot, alternately reboot the controllers anytime between 1 day uptime and 800 days uptime to restart the counter prior to vxAbsTicks rollover. With a proper failover environment, there should be no interruption of service.

This issue is resolved in the firmware 07.35.10.10 or later for the 25xx series and 07.15.11.12 or later for the 6xxx series. The fix changes the reboot schedule to different times for each controller without the need for a manual reboot. (Reboot date/time can still be scheduled manually).

NOTE: This Alert describes a specific problem in 6.xx firmware where both controllers can reboot SIMULTANEOUSLY. The fix changes the reboot schedule to different times for each controller without the need for a manual reboot. (Reboot date/time can still be scheduled manually).

See <Document:1569641.1> for additional information and clarification on this issue.

The firmware matrix are available in the following documents:

<Document:1021780.1> Sun StorageTek 2510 (iSCSI) Firmware Matrix
<Document:1005365.1> Sun StorageTek 2530 (SAS) Firmware Matrix
<Document:1017877.1> Sun StorageTek 2540 (FC) Firmware Matrix
<Document:1011474.1> Sun StorageTek 6140 Firmware Matrix
<Document:1022296.1> Sun Storage 6180 Firmware Matrix
<Document:1009934.1> Sun StorageTek 6540 Firmware Matrix
<Document:1022298.1> Sun Storage 6580/6780 Firmware Matrix
<Document:1011551.1> Sun StorageTek Flexline 380 (FLX380) with Common Array Manager (CAM) Firmware Matrix

History

18-Jun-2010: Document created, issue is Resolved
01-Jun-2010: Updated for minor formatting issues
15-Dec-2011: Update "Notes" in Workaround section
19-Dec-2011: Update Workaround section for additional information
15-Nov-2012: Maintenance update; no change in content, migrate BugID
29-May-2013: Updated to include additional reference to document 1455525.1
31-Jul-2013: Updated to include additional reference to document 1569641.1

vxWorks Detail
What is it? - There is a timer in the firmware, specifically
in vxWorks, called vxAbsTicksthat is only a double word long
0x0000 0000.  When this timer rolls over from 0xffff ffff
to 0x0000 0000 (approximately 828.5 days) there is the possibility
that if there is host I/O to volumes, the associated drives will
be failed with a write failure.  Thiswas discovered in 2003,
CR# 68447 was opened against the issue.  The CR# put a function
in the controller firmware called 'cfgMonitorTask' that will
reboot the controllerif the vxAbsTicks value is within 12 days
of 828 days.  This has been in the firmware from 03.xx up to
06.60 firmware.

You can monitor this using the following shell
command:
% vxAbsTicks
vxAbsTicks = 0x2e5540: value = 227183 = 0x3776f

What Happened? - When the conversion from RC1 to RC2 was completed
The functionality in cfgMonitorTask was not ported into 07.xx CFW.
Therefore, this reintroduced the ungraceful vxAbsTicks timer rollover
at approximately 828.5days with the possibility that if there is
host I/O to volumes, the associated drives will be failed with a
write failure.

Where was it fixed? - CR 138248 was added to RC2 trunk prior to
Emerald/Exmoor and is in all subsequent releases which adds the
proactive reboot of the controllers prior to the ungraceful vxAbsTicks
timer rollover.

Questions regarding this document should be addressed to:
sunalertpublication_us_grp@oracle.com

References

This solution has no attachment