Oracle ZFS Storage Appliance: All Read Cache SSD's Present in a ZFS Cluster Node are Missing from the BUI/CLI

Asset ID:	1-72-2296823.1
Update Date:	2017-08-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2296823.1 : Oracle ZFS Storage Appliance: All Read Cache SSD's Present in a ZFS Cluster Node are Missing from the BUI/CLI

Applies to:

Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS5-2 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

An Oracle ZS4-4 appliance has two online read cache SSDs in the cluster head. The SSDs are located in HDD2 and HDD3 respectively.

The SSDs appear fine and exhibit a green LED display. The problem is that neither of the disks show up in the AFS Appliance management software.

We can see the boot disks, but the SSDs are missing:

ZS4-4> maintenance hardware select chassis-000 select disk show
Disks:

LABEL STATE MANUFACTURER MODEL SERIAL RPM TYPE
disk-004 HDD 4 ok HITACHI H109090SESUN900G 001427A462TF KXH462TF 10000 system
disk-005 HDD 5 ok HITACHI H109090SESUN900G 001426A4BBZF KXH4BBZF 10000 system
disk-006 HDD 6 absent - - - - --
disk-007 HDD 7 absent - - - - --

When viewing the SSDs from the ILOM during boot, they appear to be fine.:

PCI ENCL LUN VENDOR PRODUCT PRODUCT SIZE \
SLOT SLOT NUM NAME IDENTIFIER REVISION NVDATA
----- ---- --- -------- ---------------- ----------- ---------
2 LSI SAS2008-IR 11.05.02.00 0A:03:00:1A
2 4 0 HITACHI H109090SESUN900G A720 900.1 GB
2 5 0 HITACHI H109090SESUN900G A720 900.1 GB
2 6 0 SANDISK LB1606R SUN1.6T S311 1.600 TB
2 7 0 SANDISK LB1606R SUN1.6T S311 1.600 TB

The disks also show up in the secure shell with the format utility. Here we see the boot disks and the two SSD disks.

0. c0t5000CCA0710F5B00d0 <HITACHI-H109090SESUN900G-A720 cyl 54717 alt 2 hd 255 sec 126>
/scsi_vhci/disk@g5000cca0710f5b00 /dev/chassis/SYS/HDD4/disk
1. c0t5000CCA07100DBBCd0 <HITACHI-H109090SESUN900G-A720 cyl 54717 alt 2 hd 255 sec 126>
/scsi_vhci/disk@g5000cca07100dbbc /dev/chassis/SYS/HDD5/disk
182. c0t5001E82002810A18d0 <SANDISK-LB1606R SUN1.6T-S311-1.46TB>
/scsi_vhci/disk@g5001e82002810a18
183. c0t5001E82002810E88d0 <SANDISK-LB1606R SUN1.6T-S311-1.46TB>
/scsi_vhci/disk@g5001e82002810e88

Changes

There may not have been any changes to the system. It may just be that the Administrator was not aware that the disks were missing.

In other cases, the problem may occur after the motherboard or any hardware on the cluster node has been replaced.

The problem may also occur after a failover resulting in no read cache disks assigned to the storage pool.

Cause

The problem is with the cabling between the Motherboard on the ZS4-4, and the Disk Drive backplane. There are two cables that connect these components. The problem is that the cables are crisscrossed.

You can also see the problem via the locations reported for boot disks in the management software. The locations are wrong.

The boot drives are physically located in HDD0 and HDD1 on the appliance. They are not in HDD4 and HDD5.

When correctly cabled, all disks must show up in the following locations:

ZS4-4> maintenance hardware select chassis-000 select disk show
Disks:

LABEL STATE MANUFACTURER MODEL SERIAL RPM TYPE
disk-000 HDD 0 ok HITACHI H109090SESUN900G 001427A462TF KXH462TF 10000 system
disk-001 HDD 1 ok HITACHI H109090SESUN900G 001426A4BBZF KXH4BBZF 10000 system
disk-002 HDD 2 ok SANDISK LB1606R--SUN1.6T 41500600 -- cache
disk-003 HDD 3 ok SANDISK LB1606R--SUN1.6T 41502244 -- cache

Solution

The fix is to re-cable the connections between the MB and the disk backplane per the diagram below:

Cable Connections between MB and BackPlane

Attachments

This solution has no attachment