Sun Storage 6180 Array
 
Sun Storage 6580 Array
 
Sun Storage 2510 Array
 
Sun Storage 2540 Array
 
Sun Storage 2540-M2 Array
 
Sun Storage 6780 Array
 
Sun Storage 6140 Array
 
Sun Storage 2530 Array
 
Sun Storage 2530-M2 Array
 
Sun Storage 6540 Array
 
Sun Storage 6130 Array
 

PLA-Support>Sun Systems>DISK>Arrays>SN-DK: 6130
 

In this Document

Purpose

Troubleshooting Steps

Appendix:

Collecting Support Information

How to get event lists from the user interface:

References

Applies to:

Purpose

The purpose of this document is to help troubleshoot disk impending drive failures for Sun Storage Arrays.

Impending drive failures can occur in two ways:

1) The disk drive itself keeps track of errors on the disk and will report to the subsystem. If too many errors have occurred on the drive, the drive will be flagged. This is known as a Drive Reported Predictive Failure Alert (PFA).
2) The system keeps track of the number errors a drive has in communicating with the RAID controllers. If too many events occur in given period, the drive will be flagged. This is known as Synthesized PFA.

PFA's are NOT drive failures. They are an early warning service supplied by the array and drives in order to give the administrator a warning that the drive may or is about to fail. This document is designed to help clarify when a drive should be replaced, if this fault should be present on an array.

Symptoms Include:

• SANtricity Storage Manager shows an alert for Impending Drive Failure Low Risk
• Common Array Manager shows an alarm for Impending Drive Failure Risk Low (xx.66.1026)
• SANtricity Storage Manager shows an alert for Impending Drive Failure Medium Risk
• Common Array Manager shows an alarm for Impending Drive Failure Risk Medium (xx.66.1025)
• SANtricity Storage Manager shows an alert for Impending Drive Failure High Risk
• Common Array Manager shows an alarm for Impending Drive Failure Risk High (xx.66.1024)

If you have been replacing a lot of drives, especially due to these faults, you will want to go through this exercise to identify the reason for the failure.

Troubleshooting Steps

Please validate that each troubleshooting step below is true for your environment. Each step will provide instructions via a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Please do not skip a step.

1. Verify the Critical Fault by the Array.

Use the user interface to verify the list of critical faults, and the details of each fault, along with the firmware and NVSRAM of your system

Reference: <Document 1021057.1> Sun Storage Common Array Manager (CAM): How to Verify Critical Faults for Sun Storage 2500, 2500-M2, 6000 and J4000 Arrays.
Reference: <Document 1021067.1> How to Verify Sun Storage Array Firmware Using Sun Storage Common Array Manager or SANtricity.

Use the table below to match up your faults to the firmware and NVSRAM on the array. 

• If there is no critical fault in as referenced in the symptoms above, there are no drives that have this condition.  If the array isn't reporting the Impending Failure, then there should be no alarm. No further troubleshooting is required.
• If you cannot match up your configuration and fault in the above list, then you are exposed to bug 15481652, 15529046 and/or 15726317, which can cause false impending failure faults.  Go to Step 2.
  <Bug 15481652> <Bug 15529046> <Bug 15726317>

 2. Identify the number of affected volumes.

From the alarm details, you will be able to see the volume group/Virtual Disk ID(VDisk) and the Drive location. Using the VDisk ID,
get the number of volumes in the vdisk by looking at the VDisk details.

CAM Browser

1. Log into Common Array Manager
2. Click on your array name
3. Click on Virtual Disks in the left hand tree
4. In the large pane, click on your VDisk ID
5. In the section marked "Related Information" there is an entry for Volumes. The value between the "( )" is the number of volumes.

SANtricity

1. Launch SANtricity ES
2. Double Click on your Array Name to launch the Array Management Window
3. Click on the Logical tab
4. In the left pane of the logical tab there is an expandable tree under your array name. This will list the Volume Groups by name. Expand the Volume Group identified in the fault, and count the number of volumes. Note that "Free Space" is NOT a volume

• If there are more than one volumes in the VDisk/Volume Group, go to Step 3.
• If there is only a single volume or no volumes in the VDisk/Volume Group, the system is correctly identifying the drive fault, and should be replaced.

3. View the volume details and identify whether the volumes are owned by a single controller or split between two controllers.

NOTE: You only need to go as far as identifying at least two of the volumes are owned by different controllers in the affected VDisk/Volume Group.

CAM Browser

1. Log into Common Array Manager
2. Click on your array name
3. Click on Virtual Disks in the left hand tree
4. In the large pane, click on your VDisk ID
5. In the section marked "Related Information" there is an entry for Volumes. Click on it to get the list of volume names, and copy them down.
6. Click on the first volume in the list, and look at the Preferred Controller field, and copy that down.
7. Click on the Volumes Option in the left pane
8. Click on the next volume in your list, identify the preferred controller field.
9. Repeat steps 6 through 8 until you identify two volumes that have different preferred controllers, or have reached the end of the list and concluded that all volumes are owned by the same controller.

CAM sscs

Use the list of volumes from:

to run against:

There is a field for preferred owner. Review all volumes until you have identified whether there are two or more volumes owned by different controllers, or the end of the list is reached.

SANtricity

1. Launch SANtricity ES
2. Double Click on your Array Name to launch the Array Management Window
3. Click on the Logical tab
4. In the left pane of the logical tab there is an expandable tree under your array name. This will list the Volume Groups by name. Expand the Volume Group identified in the fault, and count the number of volumes. Note that "Free Space" is NOT a volume.
5. Right Click on each volume name in the list, and click properties(6.xx firmware) or just click on the volume and the right pane will populate with the volume details.
6. Identify the preferred owner of the volume
7. Repeat steps 5 and 6 until you identify two volumes that have different preferred controllers, or have reached the end of the list and concluded that all volumes are owned by the same controller.
   
   

• If two or more volumes have preferred ownership by separate controllers, go to Step 4.
• If all volumes have the same preferred owner, go the system is correctly identifying the drive fault, and should be replaced.

4. Save a stateCaptureData.dmp file from the array, and look for the Synthetic PFA flag for each affected drive.

This can be performed by collecting a support data through normal methods and unzipping/unpacking the file.

Reference <Document 1002514.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] Common Array Manager
Reference <Document 1014074.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] SANtricity Storage Manager

The file can also be collected through

CAM CLI

'service' is located under:
Solaris: /opt/SUNWsefms/bin
Windows: C:\Program Files\Sun\Common Array Manager\Component\fms\bin
Linux: /opt/sun/cam/private/fms/bin

SANtricity GUI

• Launch SANtricity ES
• Double Click on your Array Name to launch the Array Management Window
• Open the Advanced Menu
• Select the Troubleshooting Sub Menu
• Select Capture State Information and save the file off on your client system for review(this is a text file).

Within the file, there is an output called "getObjectGraph_MT(4,0,0,0,0,0,0,0,0)". In that output use the Tray and Slot location from the alarm to identify the drive.

• For 6.x firmware, convert the drive location "tray/slot" from Hex to Decimal to identify the drive. For example, "55/f" represents "85/15"
• For 7.x firmware, convert the "tray ref" to "Tray ID" as follows and identify the drive:
  
  Here is a SAMPLE output for drive Tray.0.Drive.10:
  
  Executing getObjectGraph_MT(4,0,0,0,0,0,0,0,0,0) on controller A:
  OBJECT GRAPH - 0x2a7264b8
  cfgGeneration : 0x6739
  DRIVE - 0x2b073930
  Off/GHS/Rmvd/Avlb : 0/0/0/0
  uncertified : 0
  Pfa : 0
  drivePfaReason : 0x1 NONE
  status : 0x1 OPTIMAL
  cause : 0x1
  vgIndex : 3
  inquiry info : SEAGATE / ST373554FSUN72G / 0409
  serialNumber : 337163KX 3KP163KX
  blkSize : 0x200
  raw/usable caps : 0x1117732400 / 0x10f7732400
  interfaceType : Fibre 0x5 0x7 Fibre 0x1 0x7
  slot/tray ref : 10/0e 10 00 00 a0 b8 4d 91 0b 00 00 00 00 00 00 00 00 00 00 00
  In the example above, the "tray ref" is "10 00 00 a0 b8 4d 91 0b". Use "ionShow" output in the same file to convert this "tray ref" to TrayID. Note that if "tray ref" is zeros, it refers to Controller Tray.
  
  The example below refers to TrayID 0:
  
  Executing  ionShow(12,0,0,0,0,0,0,0,0,0):
  
  
  Initiator Driver Logical Units:
  Encl Disc Timeout   0
  Expected Enclosures
  
  Type Devnum   ItnCnt Online Id               Location State      Capacity BlkSz
  ---- -------- ------ ------ ---------------- -------- ----- ------------- -----
  Encl 00020000      2      2 100000a0b84d910b  t0   

From the output, identify the drivePfaReason.

• If the drivePfaReason is DRIVE, the drive is reporting the fault, and indicates it should be replaced.
• If the drivePfaReason is SYNTHESIZED or if "drivePfaReason" field is not found, go to Step 5.
• If the drivePfaReason is NONE or something else, contact Oracle for further investigation.

5.  Review the event log for event type 1016 (Unrecoverable Media Error) within the last 2 weeks.

Use the user interface to see if there are any event types 0x1016 for Unrecoverable Media Errors on the drive(s) listed in the impending failure fault, in the last two weeks.
See the appendix at the end of the document for details on how to get these events.

• If there are 1016 errors for the drive(s) in the fault(s), contact Oracle to have the drive(s) replaced.
• If there ae no 1016 errors drive(s) in the fault(s), go to Step 6.

6.  Review the event log for event type 100A (Check Condition) within the last 2 weeks

Use the user interface to see if there are any event types 0x100A for Check Conditions  on the drive(s) listed in the impending failure fault, in the last two weeks. 
See the appendix at the end of the document for details on how to get these events.

• If there are 100A events for the indicated drive, with event specific code 03/xx/xx, the drive should be replaced as this is a Drive Medium error
• If there are 100A events fro the indicated drive, with event specific code 04/xx/xx, the drive should be replaced as this is a Drive Hardware failure
• If there are 100A events for the indicated drive, with event specific code B/88/3, the drive should be replaced as this is a transmission failure between the drive interposer and the drive.
• If there are 100A events fro the indicated drive, with event specific code 1/nn/nn and there are 30 or more of these for the same drive during the two week period, the drive should be replaced due to recoverable errors.
• If there are no 100A events in the last two weeks, or they do not match the event specific codes, go to Step 7.

7. Based on the troubleshooting you have performed, you have identified that the drive has faulted with a Synthesized PFA, likely due to bug 15481652,15529046 or 15726317
<Bug 15481652> <Bug 15529046> <Bug 15726317>
The bug is due to what is called a Stagnant IO threshold. This is a threshold based on the number of times an IO to a drive timed out in the array controllers, within a set amount of seconds. This threshold is set too low for Virtual Disks that have more than one volume and the ownership of the volumes is split between controllers, as you indicated in Step 5, and there is no user interface to change the threshold. You are more likely to see this issue recur when the affected drives are under heavy load.

For 6140, 6540, Flexline 380 arrays, there is a firmware fix in array firmware 07.60.56.10 that permanently sets the threshold.  This is delivered via Sun StorageTek[TM] Common Array Manager version 6.8.1  or later.

For 6180, 6580, and 6780 arrays, there is a firmware fix in array firmware 07.77.13.11.  This is delivered in Sun StorageTek[TM] Common Array Manager version 6.8 or later.

For other arrays, there may be an NVSRAM fix that can be requested against this bug. It is not currently available for download, and is only available upon request. NOTE: The NVSRAM referenced to fix this issue is delivered via escalation to Tier 3 support for the product.

Neither the application of the NVSRAM nor the firmware upgrade to your array will clear the counters on the drive and clear the alarm. You will still need to perform the actions below.

In order to alleviate the bug, and clear the counters:

• Migrate all volumes in the affected VDisk that you identified in Step 3 to a single array controller.
  Reference <Document 1006464.1> Changing a volume's "owning controller" and "preferred controller" in Sun StorageTek[TM] Common Array Manager
  
  
• Use the Revive function in the User Interface to clear the Synthetic Counters

CAM GUI

1. Log into Common Array Manager
2. Click on your array name
3. Click on Virtual Disks in the left hand tree
4. In the large pane, click on your VDisk ID
5. In the Related Information, click on Disks
6. In the Disks Summary, click on your affected Disk.
7. Click the Revive button(it should be blue. If it's grey, try the CLI below)

CAM CLI

OR

Alternatively:
service -d <array-name> -c revive -t <tXdriveY>

SANtricity GUI

1. Launch SANtricity ES
2. Double Click on your Array Name to launch the Array Management Window
3. Click on the Physical Tab
4. Click on the affected drive slot in the array graphic
5. Open the Advanced Menu
6. Open the Recovery Sub Menu
7. Open the Revive sub Menu
8. Click Drive
9. Type "yes" when prompted and click continue.
   
   

• If the PFA critical fault is still present on the array after 5 minutes, contact Oracle.
• If the PFA critical fault is cleared, no further work is required.

Appendix:

Collecting Support Information

Reference <Document 1002514.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] Common Array Manager.
Reference <Document 1014074.1> Collecting Support Data for Arrays Using Sun StorageTek[TM] SANtricity Storage Manager.

How to get event lists from the user interface:

Sun StorageTek Common Array Manager:

Browser:

1. Expand Storage Arrays in the left menu pane.
2. Expand your storage array name in the left menu pane.
3. Expand Troubleshooting in the left menu pane.
4. Click on Events.
5. In the right pane, click on the -|-> icon.  If you mouse over it it will state Advanced Filter.
6. Set Event to Log Events.
7. Set Event Type to Component.
8. Set Read the last X Kbytes From Log File to 100.
9. Set String Filter to 0x100A  or 0x1016.  (you will have to run a search for both)
   
10. Click on the Details of any alarm that is shown.

SSCS

SANtricity Storage Manager:

GUI:

1. Launch SANtricity.
2. Double Click on your array name to open the Array Management Window.
3. Click on the Advanced Menu.
4. Click on the Troubleshooting Sub-Menu.
5. Click on View Event Log.
6. Un-Check View Only Critical Events.
7. Click on the Component Type field header to sort the events.
8. Look for Drive in the list of events.
9. For any Drive event, highlight it, and check the View Details box.
10. Get the value of the Event type and Event Specific Details field for each Drive event.

SMcli:

Get the list of events by saving off the event log:

Open a text viewing application to look at the individual events.
Get the value of the Event type and Event Specific Details field for each Drive event.

References

This solution has no attachment