Solaris Volume Manager (SVM) How to Replace Internal System FC-AL Disks in 280R, V480, V490, V880, V890, and E3500 Servers

Asset ID:	1-71-1010753.1
Update Date:	2018-04-16
Keywords:

Solution Type Technical Instruction Sure

Solution 1010753.1 : Solaris Volume Manager (SVM) How to Replace Internal System FC-AL Disks in 280R, V480, V490, V880, V890, and E3500 Servers

Applies to:

Sun Enterprise 3500 Server - Version All Versions to All Versions [Release All Releases]
Sun Solaris Volume Manager (SVM) - Version 11.9.0 to 11.10.0 [Release 11.0]
Sun Fire V480 Server - Version All Versions to All Versions [Release All Releases]
Sun Fire V490 Server - Version All Versions to All Versions [Release All Releases]
Sun Fire 280R Server - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on SPARC (32-bit)

Goal

This document provides a working example of how to replace a system's failed internal Fibre Channel Arbitrated Loop (FC-AL) disk when it is under Solaris Volume Manager (SVM) control. The most typical scenario involves a failed boot disk mirrored with SVM.

Solution

1) Let's start with a working mirrored pair of boot disks in a V490.
In this example, the root and swap partitions are mirrored.
Notice the optimal outputs from metastat, metadb and format.

# metastat

d1: Mirror
     Submirror 0: d11
       State: Okay
     Submirror 1: d21
       State: Okay
     Size: 1052163 blocks (513 MB)

     d11: Submirror of d1
         State: Okay
         Size: 4194828 blocks (2.0 GB)
             Device     Start Block Dbase        State Reloc Hot Spare
             c4t1d0s1          0     No            Okay   Yes
     d21: Submirror of d1
         State: Okay
         Size: 4194828 blocks (2.0 GB)
             Device     Start Block Dbase        State Reloc Hot Spare
             c4t0d0s1          0     No            Okay   Yes

d0: Mirror
     Submirror 0: d10
       State: Okay
     Submirror 1: d20
       State: Okay
     Size: 16629921 blocks (7.9 GB)

     d10: Submirror of d0
         State: Okay
         Size: 25166079 blocks (12 GB)
             Device     Start Block Dbase        State Reloc Hot Spare
             c4t1d0s0          0     No            Okay   Yes
     d20: Submirror of d0
         State: Okay
         Size: 25166079 blocks (12 GB)
             Device     Start Block Dbase        State Reloc Hot Spare
             c4t0d0s0          0     No            Okay   Yes

     Device Relocation Information:
     Device   Reloc Device ID
     c4t0d0   Yes    id1,ssd@n20000004cf7fe655
     c4t1d0   Yes    id1,ssd@n20000004cf8f57c1

# format
      28. c4t0d0 <SEAGATE-ST336605FSUN36G-0438 cyl 24620 alt 2 hd 27 sec 107>
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf7fe655,0
      29. c4t1d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8f57c1,0

# metadb
        flags           first blk       block count
     a m p luo        16              8192            /dev/dsk/c4t0d0s7
     a    p luo        8208            8192            /dev/dsk/c4t0d0s7
     a    p luo        16400           8192            /dev/dsk/c4t0d0s7
     a    p luo        16              8192            /dev/dsk/c4t1d0s7
     a    p luo        8208            8192            /dev/dsk/c4t1d0s7
     a    p luo        16400           8192            /dev/dsk/c4t1d0s7

2) Disk c4t1d0 fails. Notice how the outputs have changed. Submirrors are in "Needs Maintenance", state database replicas have write errors and format output shows "drive type unknown". Other logs such as /var/adm/messages should be reviewed as well for evidence of failure.

# metastat

d1: Mirror
     Submirror 0: d11
       State: Needs maintenance
     Submirror 1: d21
       State: Okay
     Size: 1052163 blocks (513 MB)

     d11: Submirror of d1
         State: Needs maintenance
         Size: 4194828 blocks (2.0 GB)
             Device     Start Block Dbase   State       Reloc Hot Spare
             c4t1d0s1          0     No     Maintenance   Yes
     d21: Submirror of d1
         State: Okay
         Size: 4194828 blocks (2.0 GB)
             Device     Start Block Dbase   State Reloc Hot Spare
             c4t0d0s1          0     No      Okay   Yes

d0: Mirror
     Submirror 0: d10
       State: Needs maintenance
     Submirror 1: d20
       State: Okay
     Size: 16629921 blocks (7.9 GB)

     d10: Submirror of d0
         State: Needs maintenance
         Size: 25166079 blocks (12 GB)
             Device     Start Block Dbase     State      Reloc Hot Spare
             c4t1d0s0          0     No      Maintenance   Yes
     d20: Submirror of d0
         State: Okay
         Size: 25166079 blocks (12 GB)
             Device     Start Block Dbase        State Reloc Hot Spare
             c4t0d0s0          0     No            Okay   Yes

     Device Relocation Information:
     Device   Reloc Device ID
     c4t0d0   Yes    id1,ssd@n20000004cf7fe655
     c4t1d0   Yes    id1,ssd@n20000004cf8f57c1

# format
      28. c4t0d0 <SEAGATE-ST336605FSUN36G-0438 cyl 24620 alt 2 hd 27 sec 107>
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf7fe655,0
      29. c4t1d0 <drive type unknown>
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8f57c1,0

# metadb
        flags           first blk       block count
     a m p luo        16              8192            /dev/dsk/c4t0d0s7
     a    p luo        8208            8192            /dev/dsk/c4t0d0s7
     a    p luo        16400           8192            /dev/dsk/c4t0d0s7
      W   p l          16              8192            /dev/dsk/c4t1d0s7
      W   p l          8208            8192            /dev/dsk/c4t1d0s7
      W   p l          16400           8192            /dev/dsk/c4t1d0s7

3) If any submirror in the failed disk is still reporting as "Okay", that submirror should be detached. For example if d10 was still reporting "Okay":

# metadetach d0 d10

4) We can now proceed with the hardware replacement of the disk.

# luxadm remove_device -F /dev/rdsk/c4t1d0s2

   WARNING!!! Please ensure that no filesystems are mounted on these device(s).
   All data on these devices should have been backed up.
   The list of devices which will be removed is:
     1: Device name: /dev/rdsk/c4t1d0s2
        Node WWN:    20000004cf8f57c1
        Device Type:Disk device
        Device Paths:
        /dev/rdsk/c4t1d0s2
   Please verify the above list of devices and
   then enter 'c' or <CR> to Continue or 'q' to Quit. [Default: c]:
   stopping: /dev/rdsk/c4t1d0s2....Done
   offlining: /dev/rdsk/c4t1d0s2....Done
   Hit <Return> after removing the device(s).
   ............
   Device: /dev/rdsk/c4t1d0s2 Removed.

If disk is Hot-Pluggable refer to:
<Document 1007367.1> Removing and Replacing 280R, V480, V490, V880, V880z or V890 Hot-Pluggable Internal Disk Drives

For the v880 only, you may remove the device by enclosure name and slot number. For example:

# luxadm remove_device FCloop,s1

if the luxadm remove fails, you may have to do the following additional step :

# init 0

Physically remove the disk cxtyd0

ok boot

# devfsadm -C

Verified this removed /dev/[r]dsk/cxtyd0*

# devfsadm -c disk

5) Physically replace the disk.

6) The next command will create the devices for the new drive.

# luxadm insert_device
     Please hit <RETURN> when you have finished adding Fibre Channel Enclosure(s)/Device(s):
     Waiting for Loop Initialization to complete...
     New Logical Nodes under /dev/dsk and /dev/rdsk :
        c4t1d0s0
        c4t1d0s1
        c4t1d0s2
        c4t1d0s3
        c4t1d0s4
        c4t1d0s5
        c4t1d0s6
        c4t1d0s7

For the v880 only, you may add the device by enclosure name and slot number. For example:

# luxadm insert_device FCloop,s1

Alternatively, you can use the command devfsadm -Cv to create the devices. Depending on your version of Solaris, the picld daemon may make them for you as well. If all these fail, a reconfiguration reboot usually solves the problem.

If device to replace is shown as as 'unusable' in cfgadm output then check:
<Document 1639070.1> Steps For Clearing Devices in Unusable or Failing State From cfgadm After LUNs Have Already Been Removed (Doc ID 1639070.1)

7) Once the disk is labeled and partitioned in Solaris, we can finish up the repairs in SVM.

Prepare the partition table on the disk to match that of the disk it is mirrored with:

# prtvtoc /dev/rdsk/c4t0d0s2 | fmthard -s - /dev/rdsk/c4t1d0s2
fmthard: New volume table of contents now in place.

See also:
<Document 1386408.1> Solaris Volume Manager (SVM): How To Copy The Partition Table From One Disk To Another

8) Restore the state database replicas.

# metadb -d c4t1d0s7
# metadb -a -c 3 c4t1d0s7

9) Remirror and monitor the resync using the metastat command. Note that any submirror which was manually detached needs to be reattached. For example, if metadetach was used on d10, reattach it:

# metattach d0 d10

9a) Otherwise, remirror with metareplace.

First Update Device Relocation Information (DRI) of replaced drive:

# metadevadm -u c2t0d0 (5.9 Only)

For Solaris 10 check if affected by
<Bug 22065674> - 5.10_u[10,11] - metareplace corrupting metadb's. metadevadm -u ineffective.
Typical affected systems are when one of the following patches is installed:
(SPARC) <SunPatch 145899-03> or higher - SunOS 5.10: SVM patch
(X86) <SunPatch 145900-03> or higher - SunOS 5.10_x86: SVM patch
both of which reached revision -15 before being rolled into patches
(SPARC) <SunPatch 147147-26> SunOS 5.10: kernel patch
(X86) <SunPatch 148076-09> SunOS 5.10_x86: md patch
Further details in:
<Document 2090016.1> Solaris Volume Manager (SVM) the 'metadevadm -u' Command Shows "New device reloc information" or "Invalid device relocation information detected" But DevID Is Not Updated

Then run metareplace.

# metareplace -e d0 c4t1d0s0
d0: device c4t1d0s0 is enabled
# metareplace -e d1 c4t1d0s1
d1: device c4t1d0s1 is enabled

10) If metattach/metareplace fails, check status of the meta mirror. And if both metadevices in 'Needs maintenance' then check 'Invoke' command in metastat output for good device.

d1: Mirror
     Submirror 0: d11
       State: Needs maintenance
     Submirror 1: d21
       State: Needs maintenance
     Size: 1052163 blocks (513 MB)

     d11: Submirror of d1
         State: Needs maintenance
         Size: 4194828 blocks (2.0 GB)
         Invoke: metareplace d1 /dev/dsk/c4t1d0s1 <new device>
             Device     Start Block Dbase   State       Reloc Hot Spare
             c4t1d0s1          0     No     Maintenance   Yes
     d21: Submirror of d1
         State: Needs maintenance
         Invoke: metasync d1
         Size: 4194828 blocks (2.0 GB)
             Device     Start Block Dbase   State Reloc Hot Spare
             c4t0d0s1          0     No      Okay   Yes

The submirror d21 can be returned to the "Okay" state by using the metasync command as shown in the output of metastat. When in okay state return to metattach/metareplace command.

Want to know more?

You can find further details on this topic in the Solaris Volume Manager Administration Guide. View the HTML version in your browser, download the PDF version for later reference

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the

My Oracle Support Community - Oracle Solaris Volume Manager

References

<NOTE:1386408.1> - Solaris: How to Copy the Partition Table From One Disk to Another With prtvtoc and fmthard Command also for Mirrored Disks and Solaris Volume Manager (SVM)
<NOTE:2090016.1> - Solaris Volume Manager (SVM) the 'metadevadm -u' Command Shows "New device reloc information" or "Invalid device relocation information detected" But DevID Is Not Updated
<NOTE:1007367.1> - Removing and Replacing the Sun Fire[TM] 280R , Sun Fire[TM] V480 ,Sun Fire[TM] V490 ,Sun Fire[TM] V880 ,Sun Fire[TM] V880z or Sun Fire[TM] V890 Hot-Pluggable Internal Disk Drives.
<NOTE:2332709.1> - Sparc Servers : Replacing a Smaller Older Disk with a Newer Larger Capacity Disk
<NOTE:2287066.1> - Using a 146GB FC-AL disk to replace a 72GB FC-AL disk which is part of a mirrored volume.

Attachments

This solution has no attachment