ZFS: 'zpool import' Fails With "error: cannot import 'pool name': one or more devices is currently unavailable" After Sun Storedge 3510 Controller Replacement

Asset ID:	1-72-1540292.1
Update Date:	2016-07-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1540292.1 : ZFS: 'zpool import' Fails With "error: cannot import 'pool name': one or more devices is currently unavailable" After Sun Storedge 3510 Controller Replacement

Applies to:

Solaris Operating System - Version 10 3/05 to 10 1/13 U11 [Release 10.0]
Sun Storage 3510 FC Array - Version Not Applicable to Not Applicable [Release N/A]
Information in this document applies to any platform.

Symptoms

After planned maintenance to replace a Sun Storedge 3510 Controller, 'zpool import poolname' failed with the following error:

# zpool import storage
cannot import 'storage': one or more devices is currently unavailable

Cause

The 'zpool import' failure was due to EIO while reading the VDEV Labels 2 and 3. When device is confgured into a ZFS pool, 4 vdev (virtual device) labels are automatically placed on the disk. LABEL 0 & 1 are located at the front of the physical disk slice LABEL 2 & 3 are located at the end. The four labels are identical and any copy can be used to access and verify the contents of the pool. The vdev labels are placed at the fixed location on the device and updates does not follow ZFS copy-on-write (COW) symentics. That means contents of the label are overwritten on updates.

Running zdb -l on the device reports failure on reading LABEL 2 and 3 as shown in the following example.

# zdb -l /dev/rdsk/c5t600C0FF0000000000083F44CF643BF00d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version: 29
name: 'storage'
state: 0
txg: 56606456
pool_guid: 17582685044487868067
hostid: 745400582
hostname: 'sunxvm1'
top_guid: 8285032951382859016
guid: 8285032951382859016
vdev_children: 1
vdev_tree:
type: 'disk'
id: 0
guid: 8285032951382859016
path: '/dev/rdsk/c5t600C0FF0000000000083F45806C33000d0s0'
devid: 'id1,sd@n600c0ff0000000000083f45806c33000/a,raw'
phys_path: '/scsi_vhci/disk@g600c0ff0000000000083f45806c33000:a,raw'
whole_disk: 1
metaslab_array: 23
metaslab_shift: 35
ashift: 9
asize: 3598318829568 <<<<<<
is_log: 0
DTL: 148
--------------------------------------------
LABEL 1
--------------------------------------------
version: 29
name: 'storage'
state: 0
txg: 56606456
pool_guid: 17582685044487868067
hostid: 745400582
hostname: 'sunxvm1'
top_guid: 8285032951382859016
guid: 8285032951382859016
vdev_children: 1
vdev_tree:
type: 'disk'
id: 0
guid: 8285032951382859016
path: '/dev/rdsk/c5t600C0FF0000000000083F45806C33000d0s0'
devid: 'id1,sd@n600c0ff0000000000083f45806c33000/a,raw'
phys_path: '/scsi_vhci/disk@g600c0ff0000000000083f45806c33000:a,raw'
whole_disk: 1
metaslab_array: 23
metaslab_shift: 35
ashift: 9
asize: 3598318829568
is_log: 0
DTL: 148
--------------------------------------------
LABEL 2
--------------------------------------------
failed to read label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to read label 3

After controller replacement, the LUN size reported by the array was 1.2TB instead of 3.27TB hence why the labels at the end of the disk could not be read. The following shows what the array was reporting:

bash-3.00# ./sccli
sccli: selected device /dev/rdsk/c5t600C0FF0000000000083F44CF643BF00d0s2 [SUN StorEdge 3511 SN#0083F4]
sccli> show inq
Vendor: SUN
Product: StorEdge 3511
Revision: 327R
Peripheral Device Type: 0x0
NVRAM Defaults: 327R 3511 v2.65
Current NVRAM Version: 3.x
Bootrecord version: 1.31H
Serial Number: NNNNNN
Page 80 Serial Number: 0083F44CF643BF00
Page 83 Logical Unit Device ID: 600C0FF0000000000083F44CF643BF00
Page 83 Target Device ID: 206000C0FF0083F4
IP Address: 10.25.7.65
Page D0 Fibre Channel Address: 9F (id 44)
Page D0 Node Name: 206000C0FF0083F4
Page D0 Port Name: 256000C0FFC083F4
Ethernet Address: 00:C0:FF:00:83:F4
Device Type: Primary
sccli> show disks
Ch Id Size Speed LD Status IDs Rev
----------------------------------------------------------------------------
2 0 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGR9ZBC
2 1 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGT653C
2 2 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGRPPDC
2 3 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGT245C
2 4 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGT452C
2 5 372.61GB 200MB GLOBAL STAND-BY HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGSN2VC
2 6 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGT6W6C
2 7 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGSU12C
2 8 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGSWD6C
2 9 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGSRTGC
2 10 372.61GB 200MB ld0 ONLINE HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGT6LWC
2 11 372.61GB 200MB GLOBAL STAND-BY HITACHI HDS7240SBSUN400G A47A
S/N KRFS06RAGSWJDC
sccli> show logical-drive
LD LD-ID Size Assigned Type Disks Spare Failed Status
------------------------------------------------------------------------
ld0 4CF643BF 1.27TB Primary RAID5 10 2 0 Good

Solution

Array Firmware version 3.27R has a limitation of 2TB LUNS. The mis-match of firmware between the current controller and replacement controller appeared to cause the issue. After upgrading the array firmware and restoring the saved configuration via the upgrade process, the LUN reported the expected and correct size of 3.27TB:

sccli> show lun-maps
Ch Tgt LUN ld/lv ID-Partition Assigned Filter Map
---------------------------------------------------------------------
4 44 0 lv0 5806C330-00 Primary
sccli>
sccli> show logical-drives
LD LD-ID Size Assigned Type Disks Spare Failed Status
------------------------------------------------------------------------
ld0 4CF643BF 3.27TB Primary RAID5 9 1 1 Rebuilding
Write-Policy Default StripeSize 128KB

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join, or start a discussion in the My Oracle Support Community, Oracle ZFS Filesystem forum.

References

<NOTE:1516050.1> - ZFS Pool Recovery: "zpool status" reports "One or more devices could not be used because the label is missing or invalid."

Attachments

This solution has no attachment