How to Setup and Verify the Multipath Configuration Between a Server and a Sun Storage J4000 Array

Asset ID:	1-71-1569646.1
Update Date:	2017-06-12
Keywords:

Solution Type Technical Instruction Sure

Solution 1569646.1 : How to Setup and Verify the Multipath Configuration Between a Server and a Sun Storage J4000 Array

Applies to:

Sun Storage J4200 Array - Version Not Applicable and later
Sun Storage J4500 Array - Version Not Applicable and later
Sun Storage J4400 Array - Version Not Applicable and later
Information in this document applies to any platform.

Goal

The J4000 family of arrays is designed for redundancy between the attached server and array . This is accomplished by cabling two data paths, along with the installation and configuration of multipath software. In this working example, we configure, verify, and test a multipath configuration between a T2000 server with Solaris 10, a J4500 array and MPxIO software (aka Solaris Traffic Manager). See the latest Array Release Notes for all supported operating systems, HBAs and patches required for multipathing support.

Note: NOT all SATA drives are supported with Solaris mpxio. See the J4200/J4400 Array Release Notes section "Supported Hard Disk Drives for Multipathing With Solaris" for supported DISKS. SATA drive model ST32500NSSUN250G is not supported. ST35000NSSUN500G and ST37500NSSUN750G require DRIVE f/w 3AZB which is included with CAM 6.6.x and later.

Solution

In our initial setup, two SAS/SATA HBA's (SG-XPCIE8SAS-E-Z) are installed in the server. Each HBA has a single connection to the Expander (or SIM) on the J4500. After the connection of the array and reconfiguration reboot of the server, the format utility can be used to see the disk drives.

# echo | format (For simplicity, only 3 of the 48 disks are shown)
........
5. c4t0d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk1
          /pci@780/pci@0/pci@8/LSILogic,sas@0/sd@0,0
6. c4t1d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk2
          /pci@780/pci@0/pci@8/LSILogic,sas@0/sd@1,0
7. c4t2d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk3
........
49. c5t0d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk1
          /pci@7c0/pci@0/pci@8/LSILogic,sas@0/sd@0,0
50. c5t1d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk2
          /pci@7c0/pci@0/pci@8/LSILogic,sas@0/sd@1,0
51. c5t2d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk3
          /pci@7c0/pci@0/pci@8/LSILogic,sas@0/sd@2,0
........

The volname feature of format has been used to write a unique name to the drive. The drives are named disk1, disk2 and disk3.
You can see that disk1, disk2 and disk3 show up twice in the format command.
Without multipath software configured, Solaris reports a disk drive on every path it is discovered on. It has no knowledge that this is indeed the same disk.
For example, disk1 was seen on path /pci@780/pci@0/pci@8/LSILogic,sas@0 as well as path /pci@7c0/pci@0/pci@8/LSILogic,sas@0.
You must see each J4000 disk twice in format, prior to enabling mpxio.

Our next step is to turn on mpxio. The complete set of instructions for enableing and disabling mpxio are provided under Oracle Solaris SAN Configuration and Multipathing Guide. In our example, we will tailor the command to explicitly turn on mpxio for just the external HBA ports.

This command will instruct solaris to enable mpxio on all mpt devices, including the internal disks.

# stmsboot -e -D mpt
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) y
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y) y/n

As we choose not to have the internal disks under mpxio control (there is only 1 path to these drives), a line is added to the /kernel/drv/mpt.conf file to tell mpxio not to enable multipathing on them.

#
name="mpt" parent="/pci@780/pci@0/pci@9" unit-address="0" mpxio-disable="yes";
#

A reconfiguration reboot of the server is performed. Upon a successful reboot, we can look at disk1, disk2, and disk3 in format.

# echo | format
      0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
      1. c0t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@2,0
      ............
      13. c6t5000CCA214C3B579d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk2
          /scsi_vhci/disk@g5000cca214c3b579
      ..........
      28. c6t5000CCA214C47BD4d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk1
          /scsi_vhci/disk@g5000cca214c47bd4
      29. c6t5000CCA214C47E97d0 <ATA-HITACHI HUA7250S-AC5A-465.76GB> disk3
          /scsi_vhci/disk@g5000cca214c47e97
      ...........

The volname written to the drives is persistent.
We can see that the two physical paths to disk1, disk2 and disk3 have been virtualized into one.
This virtual path is identified as a scsi_vhci device.
The internal drives were not virtualized. The physical paths to the drives are still there.

We can now look at this virtual device with the mpathadm command. This first command lets know two paths were created, and are operational.

# mpathadm list lu (parsed for disk1, disk2 and disk3)
        /dev/rdsk/c6t5000CCA214C3B579d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c6t5000CCA214C47BD4d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c6t5000CCA214C47E97d0s2
                Total Path Count: 2
                Operational Path Count: 2

To view a detailed breakdown of each virtual disk, the mpathadm show lu command is used.

# mpathadm show lu /dev/rdsk/c6t5000CCA214C47E97d0s2
Logical Unit: /dev/rdsk/c6t5000CCA214C47E97d0s2
        mpath-support: libmpscsi_vhci.so
        Vendor: ATA
        Product: HITACHI HUA7250S
        Revision: AC5A
        Name Type: unknown type
        Name: 5000cca214c47e97
        Asymmetric: no
        Current Load Balance: round-robin
        Logical Unit Group ID: NA
        Auto Failback: on
        Auto Probing: NA

        Paths:
                Initiator Port Name: 500605b000a12950
                Target Port Name: 5080020000493342
                Override Path: NA
                Path State: OK
                Disabled: no

                Initiator Port Name: 500605b000a132b0
                Target Port Name: 50800200004933c2
                Override Path: NA
                Path State: OK
                Disabled: no

        Target Ports:
                Name: 5080020000493342
                Relative ID: 0

                Name: 50800200004933c2
                Relative ID: 0

500605b000a12950 is the wwpn of the SAS/SATA HBA in PCI E Slot 0 (Pandora Card).
500605b000a132b0 is the wwpn of the SAS/SATA HBA in PCI E Slot 1 (Pandora Card).
Document 1566852.1 How To Determine the Firmware, Chipset and World Wide Port Number (wwpn) of the 8-Port 3Gbps SAS/SATA HBA explains how to determine the location of each HBA.
5080020000493342 and 50800200004933c2 are wwpn's on the disk. These are NOT the wwpn's of an expander or SIM board.
You will find these in the dataStore.txt file of the CAM supportdata bundle for the J4500.
There should always be two Paths:with an associated Port State: OK. Anything else, (For Example, Unavailable instead of OK) would indicate there is a problem.

In this last verification test, a raidz1 zpool is created from the 3 drives. Some I/O is generated and measured.

# zpool create testpool raidz1 c6t5000CCA214C3B579d0 c6t5000CCA214C47BD4d0 c6t5000CCA214C47E97d0

# zpool list
NAME       SIZE ALLOC   FREE    CAP HEALTH ALTROOT
testpool 1.36T 11.0G 1.35T     0% ONLINE -

# dd if=/dev/zero of=/testpool/file-of-zeros bs=8192k count=1000 &
# dd if=/testpool/file-of-zeros of=/dev/null bs=1024k count=1000 &

# iostat -Xxnt 5
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t %w %b device
137.9 430.2 15977.4 51835.6 0.0 8.7    0.0   15.3   0 96 c6t5000CCA214C47BD4d0
   69.1 214.6 7982.0 25923.0 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C47BD4d0.mpt0
   68.9 214.8 7995.5 25912.8 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C47BD4d0.mpt2

137.9 418.7 16111.9 50934.2 0.0 8.8    0.0   15.9   0 97 c6t5000CCA214C3B579d0
   68.9 209.1 8022.4 25463.9 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C3B579d0.mpt0
   69.1 208.7 8089.6 25470.6 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C3B579d0.mpt2

140.5 424.1 16461.5 50910.2 0.0 8.6    0.0   15.2   0 96 c6t5000CCA214C47E97d0
   70.3 211.4 8237.5 25454.3 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C47E97d0.mpt0
   70.1 211.9 8224.0 25456.0 0.0 0.0    0.0    0.0   0   0 c6t5000CCA214C47E97d0.mpt2

By using the correct options to iostat, we can see the device statistics for the disk, as well as each data path.
Notice how each data path assume 50% of the I/O to the disk.
This behavior occurs regardless of the drive type (SAS or SATA). SATA disks have an interposer up front, multiplexing the serial access from 1 port into 2.
The iostat data provides definitive proof that the J4000 is taking advantage of multipathing AND load balancing with all I/O to the array.

Note: The individual path statistics (mpt0 and mpt1) from iostat do not show up if the two paths to the J4000 are from one dual port SAS HBA.

Attachments

This solution has no attachment