Sun Storage J4400 Array
 
Sun Storage J4500 Array
 
Sun Storage J4200 Array
 

PLA-Support>Sun Systems>DISK>Arrays>SN-DK: J4xxx JBOD
 

In this Document

Goal

Solution

References

Applies to:

Goal

The J4000 family of arrays provides for the requirement of a large scale storage solution. The drawback to the J4000 is that it is only a JBOD (Just a Bunch of Disks). In order to introduce redundancy, a software RAID solution, such as ZFS, is usually applied to the disks. In this working example, a multi disk failure in a J4500 and its associated zpool will be repaired. 

Solution

 Starting with an optimal zpool. The 5Tb pool is made up of 12 disks and 2 spares. Redundancy is built in with ZFS, RAID Z2 (Also known as RAID 6).  The pool is online.

The J4500 incurs a hard fault. At this point in time, the cause is unknown. ZFS has failed 4 of the drives. RAID Z2 redundancy has maintained access to data. The pool has toggled to a DEGRADED state.

In order to triage, diagnose and repair this problem, two data collections must to be performed.

• An explorer from the server. See <Document 1312847.1> Oracle Explorer Data Collector Resource Center
• A supportdata from the J4500. See <Document 1002514.1> Collecting Sun Storage Common Array Manager Support Data for Arrays.

As with most J4000 disk faults, the failure is only seen at the Operating System layer. The supportdata typically reports the J4000 as healthy. Some of the more common Solaris utilities reporting the problem are...

• # zpool status will report disks as UNAVAIL. (as seen above)
• # echo | format will report disks as unknown ->   46.  c6t5000CCA214C48071d0 <drive type unknown>
• /var/adm/messages reports multiple transport failures. See <Document 1353887.1> Sun Storage J4000 JBOD Array: Troubleshooting Disk Failures
• Fault Management Services (fms) will warn the administrator about the zpool. 

All of these utilities need to be used together to arrive at a FRU strategy to repair the problem. In this particular instance, it has been decided to replace 3 of the UNAVAIL disks and repair the 4th. The first step in performing this work will be to identify the disk locations in the J4500. In order to do this, dataStore.txt file, collected in the supportdata, will be required. From this file, the UNAVAIL disk in the zpool can be correlated to a specific slot in the J4500. Here are the 3 candidates for replacement found in the dataStore.txt file.

To further identify the physical location of the disks in the array, disable and locate them with the service utility.  This should toggle the fault / location indicator LED of the drive.

 Using the ZFS zpool command, offline the faulted mpt devices.

<Bug 20365630> zfs sets vdev to FAULTED state without always closing, preventing replacement

This means means that a vdev that is faulted, or degraded, may still be kept open by zfs.
If this is the case, kernel structures for the failed disk cannot be cleared by the mpt driver, which can prevent a new replacement disk
from being correctly configured before it can be used ( particularly where mpxio is enabled )

The workaround is to issue the following command before physically
replacing the disk:

zpool offline <pool> <device>

which forces zfs to close the failed vdev

Using the Service Adviser utility in Common Array Manager, replace the disks. The device drivers may need to be reloaded to recognise the new drives.

A new supportdata is collected. The disk positions now reflect new device names for the new disk drives. This information is used to repair the zpool.

4 unique operations will need to be done on the zpool to return it to full redundancy. The first disk to repair is c6t5000CCA214C488EAd0. As previously mentioned, there appeared to be no hardware faults on this disk. A resilver attempt,  back to itself will be performed.

The next disk to repair is c6t5000CCA214C3897Fd0. From the supportdata, it is determined that this drive was replaced with c6t5000CCA214C3B635d0. Simply replace the UNAVAIL disk with the new drive inserted into slot 30.

Next, copy back all the data from the spare disk, c6t5000CCA214C49ECBd0 to the new disk, c6t5000CCA214C489E1d0 and get rid of the UNAVAIL disk c6t5000CCA214C48071d0. The spare will also be returned to the hot spare pool.

The last repair takes a different approach. No resilver will be done. Simply remove the UNAVAIL disk, c6t5000CCA214C48993d0 from the pool. As soon as this is done, spare disk c6t5000CCA214C458B3d0 will become a permanant member of datapool. Then add new disk c6t5000CCA214C482FBd0 in as a new spare. This approach avoids any resilver and does not impact redundancy or availability as all the disks reside within the same enclosure.

This solution has no attachment