Oracle ZFS Storage Appliance: Understanding the Relationship Between the ZFS Appliance and Solaris MPXIO

Asset ID:	1-79-2044778.1
Update Date:	2018-05-10
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2044778.1 : Oracle ZFS Storage Appliance: Understanding the Relationship Between the ZFS Appliance and Solaris MPXIO

Applies to:

Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Purpose

This document explains the relationship and behavior between the Oracle ZFS appliance and the Solaris mpxio multipath utility.

The appliance, although primarily designed for network storage, supports Fiber Channel Adapters in both initiator and target modes. When configured as targets, the ZFSSA takes on a behavior similar to that of an Asymmetric Array.

This document discusses the similarities and differences in Asymmetric behavior, as well as the best techniques to maximize availability during failover.

Scope

In the following diagram, we see a typical deployment of the ZFSSA in a Fiber Channel Storage Area Network.

ZFSSA in Fiber Channel SAN

World Wide Port Zoning is configured on the Brocade Switches. Two zones are created and added into the active configuration.

ZFS-NodeA target port 21:00:00:24:ff:3e:0b:69 is zoned with X4270 Initiator 21:00:00:24:ff:3e:85:1f
ZFS-NodeB target port 21:00:00:1b:32:ae:06:82 is zoned with X4270 Initiator 21:00:00:24:ff:3e:cf:37

Similarly, lun security is configured on the appliance to present a lun exclusively to the X4270 Server.

Initiator group X is created with both X4270 initiators 21:00:00:24:ff:3e:85:1f and 21:00:00:24:ff:3e:cf:37
Target group Y is created with both ZFSSA-CLuster targets 21:00:00:24:ff:3e:0b:69 and 21:00:00:1b:32:ae:06:82
A LUN is created and associated to Initiator Group X and Target Group Y.

Within the shell, we can use the stmfadm utility to view the FC configuration.

In this first set of commands, we look at the configuration from the node that owns the LUN.

ZFS-NodeA # stmfadm list-lu -v 600144F0E9434B4D000055B789470001
LU Name: 600144F0E9434B4D000055B789470001
    Operational Status     : Online
    Provider Name          : sbd
    Alias                  :
    View Entry Count       : 1
    Data File              : /dev/zvol/rdsk/pool/local/FC-Luns/dbase
    Meta File              : not set
    Size                   : 26843545600
    Block Size             : 512
    Management URL         : https://10.145.229.134:215/ak https://192.168.10.250:215/ak
    Software ID            : 000001
    Vendor ID              : SUN
    Product ID             : ZFS Storage 7320
    Serial Num             : not set
    Write Protect          : Disabled
    Write Cache Mode Select: Enabled
    Writeback Cache        : Disabled
    Access State           : Active

ZFS-NodeA # stmfadm list-hg -v 4270server
Host Group: 4270server
        Member: wwn.21000024FF3ECF37
        Member: wwn.21000024FF3E851F

ZFS-NodeA # stmfadm list-tg -v pci0port1
Target Group: pci0port1
        Member: wwn.21000024FF3E0B69
        Member: wwn.2100001B328E0682

ZFS-NodeA # stmfadm list-view -l 600144F0E9434B4D000055B789470001
   View Entry: 0
   Host group   : 4270server
   Target Group : pci0port1
   LUN          : 0

ZFS-NodeA # stmfadm list-target -v wwn.2100001B328E0682
    Target: wwn.2100001B328E0682
    Operational Status     : Online
    Provider Name          : pppt (physical proxy provider target)
    Alias                  : -
    Protocol               : Fibre Channel
    Sessions               : 0

ZFS-NodeA # stmfadm list-target -v wwn.21000024FF3E0B69
    Target: wwn.21000024FF3E0B69
    Operational Status     : Online
    Provider Name          : qlt
    Alias                  : qlt1,0
    Protocol               : Fibre Channel
    Sessions               : 1
        Initiator: wwn.21000024FF3E851F
            Alias: :qlc1
            Logged in since: Tue Jul 28 14:27:15 2015

In this second set of commands, we look at the configuration from the node that does not own the LUN.

ZFS-NodeB # stmfadm list-lu -v
LU Name: 600144F0E9434B4D000055B789470001
    Operational Status     : Online
    Provider Name          : sbd
    Alias                  : /dev/zvol/rdsk/pool/local/FC-Luns/dbase
    View Entry Count       : 0
    Data File              : prop unavailable in standby
    Meta File              : prop unavailable in standby
    Size                   : prop unavailable in standby
    Block Size             : prop unavailable in standby
    Management URL         : prop unavailable in standby
    Software ID            : prop unavailable in standby
    Vendor ID              : prop unavailable in standby
    Product ID             : prop unavailable in standby
    Serial Num             : prop unavailable in standby
    Write Protect          : prop unavailable in standby
    Write Cache Mode Select: prop unavailable in standby
    Writeback Cache        : prop unavailable in standby
    Access State           : Standby

ZFS-NodeB # stmfadm list-target -v wwn.2100001B328E0682
Target: wwn.2100001B328E0682
    Operational Status     : Online
    Provider Name          : qlt
    Alias                  : qlt0,0
    Protocol               : Fibre Channel
    Sessions               : 3
        Initiator: wwn.21000024FF3ECF37
            Alias: :qlc3
            Logged in since: Tue Jul 28 18:27:47 2015

ZFS-NodeB # stmfadm list-target -v wwn.21000024FF3E0B69
    Target: wwn.21000024FF3E0B69
    Operational Status     : Online
    Provider Name          : pppt (physical proxy provider target)
    Alias                  : -
    Protocol               : Fibre Channel
    Sessions               : 0

You can collect this configuration data in the CLI, by using the C_ISCSI module of the bundle.

ZFS-NodeA # akbundle -c C_ISCSI

Details

Three examples are provided. In the first example, you can see what the baseline configuration looks like with no failovers or errors.

EXAMPLE 1

The luxadm and format utilities are used on the client.

Sun_Fire_X4270 # luxadm -e port | grep -v NOT
/devices/pci@0,0/pci8086,3410@9/pci111d,806e@0/pci111d,806e@4/pci1077,171@0,1/fp@0,0:devctl CONNECTED
/devices/pci@0,0/pci8086,340e@7/pci111d,806e@0/pci111d,806e@4/pci1077,171@0,1/fp@0,0:devctl CONNECTED

Sun_Fire_X4270 # luxadm -e dump_map /devices/pci@0,0/pci8086,3410@9/pci111d,806e@0/pci111d,806e@4/pci1077,171@0,1/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN         Node WWN         Type
0    10000   0        2100001b328e0682 2000001b328e0682 0x0 (Disk device)
1    10900   0        21000024ff3ecf37 20000024ff3ecf37 0x1f (Unknown Type,Host Bus Adapter)
Sun_Fire_X4270 # luxadm -e dump_map /devices/pci@0,0/pci8086,340e@7/pci111d,806e@0/pci111d,806e@4/pci1077,171@0,1/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN         Node WWN         Type
0    20500   0        21000024ff3e0b69 20000024ff3e0b69 0x0 (Disk device)
1    20900   0        21000024ff3e851f 20000024ff3e851f 0x1f (Unknown Type,Host Bus Adapter)

# format
10. c0t600144F0E9434B4D000055B789470001d0 <SUN-ZFS Storage 7320-1.0 cyl 3260 alt 2 hd 255 sec 63> dbase
/scsi_vhci/disk@g600144f0e9434b4d000055b789470001

Sun_Fire_X4270 # luxadm display /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2

DEVICE PROPERTIES for disk: /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
Vendor:               SUN
Product ID:           ZFS Storage 7320
Revision:             1.0
Serial Num:           Unsupported
Unformatted capacity: 25600.000 MBytes
Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x0
Device Type:          Disk device
Path(s):

/dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
/devices/scsi_vhci/disk@g600144f0e9434b4d000055b789470001:c,raw
   Controller           /dev/cfg/c4
    Device Address              2100001b328e0682,0
    Host controller port WWN    21000024ff3ecf37
    Class                       secondary
    State                       STANDBY
   Controller           /dev/cfg/c2
    Device Address              21000024ff3e0b69,0
    Host controller port WWN    21000024ff3e851f
    Class                       primary
    State                       ONLINE

The mpathadm utility provides some critically important information. Notice how it reports the array as an Asymmetric array that does NOT support explicit failover.

Sun_Fire_X4270 # mpathadm show lu /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s0
Logical Unit: /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
        mpath-support: libmpscsi_vhci.so
        Vendor: SUN
        Product: ZFS Storage 7320
        Revision: 1.0
        Name Type: unknown type
        Name: 600144f0e9434b4d000055b789470001
        Asymmetric: yes
        Current Load Balance: round-robin
        Logical Unit Group ID: NA
        Auto Failback: on
        Auto Probing: NA

        Paths:
                Initiator Port Name: 21000024ff3e851f
                Target Port Name: 21000024ff3e0b69
                Override Path: NA
                Path State: OK
                Disabled: no

                Initiator Port Name: 21000024ff3ecf37
                Target Port Name: 2100001b328e0682
                Override Path: NA
                Path State: OK
                Disabled: no

        Target Port Groups:
                ID: 1
                Explicit Failover: no
                Access State: active optimized
                Target Ports:
                        Name: 21000024ff3e0b69
                        Relative ID: 256

                ID: 0
                Explicit Failover: no
                Access State: standby
                Target Ports:
                        Name: 2100001b328e0682
                        Relative ID: 1

EXAMPLE 2.

In this second example, we will introduce a fault on the appliance side. A panic is sent to ZFS-NodeA, which owns LUN 0 (ZFS-NodeA owns c0t600144F0E9434B4D000055B789470001d0s0). ZFS-NodeB will take ownership of all resources, including the LUN.

/var/adm/messages on Sun_Fire_X4270 logs the disappearance of the port and the failover of the LUN.

WARNING: fp(6)::GPN_ID for D_ID=20500 failed
WARNING: fp(6)::N_x Port with D_ID=20500, PWWN=21000024ff3e0b69 disappeared from fabric
/pci@0,0/pci8086,340e@7/pci111d,806e@0/pci111d,806e@4/pci1077,171@0,1/fp@0,0 (fcp6):
        offlining lun=0 (trace=0), target=20500 (trace=2800004)
        Initiating failover for device disk (GUID 600144f0e9434b4d000055b789470001)
        /scsi_vhci/disk@g600144f0e9434b4d000055b789470001 (sd191): path fp12/disk@w2100001b328e0682,0 is now ONLINE because a failover occured
        Failover operation completed successfully for device disk (GUID 600144f0e9434b4d000055b789470001): failed over from primary to secondary
/scsi_vhci/disk@g600144f0e9434b4d000055b789470001 (sd191) multipath status: degraded: path 13 fp6/disk@w21000024ff3e0b69,0 is offline

The luxadm command shows that the roles have reversed from out previous output. The original Secondary adapters are now Primary and the original Primary adapters are now Secondary.

EXAMPLE 3.

In this third example, we inject a fault to the initiator on the client side (the Sun_Fire_X4270). Recalling that the array does NOT support explicit failover, the LUN will not be moved from ZFS-NodeA to ZFS-NodeB

The primary initiator is faulted. I/O through the primary path stops. The mpxio driver issues an explicit failover command to the appliance.

Jul 29 12:09:51 x4270-ftlauder-b Initiating failover for device disk (GUID 600144f0e9434b4d000055b789470001)

The zfs appliance ignores the explicit failover command, and the LUN is not moved over to the other node.

Jul 29 12:09:54 x4270-ftlauder-b scsi_vhci: [ID 897919 kern.warning] WARNING: vhci_failover(7.2)(600144f0e9434b4d000055b789470001) failed on path 0xffffc1004d577ab0
Jul 29 12:09:54 x4270-ftlauder-b scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Jul 29 12:09:54 x4270-ftlauder-b Failover path currently unavailable for device disk (GUID 600144f0e9434b4d000055b789470001)

The luxadm command can no longer report the status of the LUN. All I/O to the LUN stops.

# luxadm display /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
Error: I/O failure communicating with /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2 OK

You cannot recover from this unless

The problem with the initiator is repaired.
You perform a failover on the appliance. This will move the LUN to the zfs node with a working path.

Because Explicit failover is not supported, it is difficult to achieve 100% redundancy.

However, you can maximize redundancy by increasing the number of paths between the Server and the Appliance. Ideally, you need to create as many primary paths as possible.

In this final diagram, zoning is expanded via the Inter Switch Link between the Brocade switches. Please note this will merge the two fabrics together.

Now, each adapter on the server has a primary path to the appliance. This insures redundancy in the event of a adapter failure.

mpxio_redundant

In this type of configuration, your luxadm output will look like this.

Sun_Fire_X4270 # luxadm display /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
Vendor:               SUN
Product ID:           ZFS Storage 7320
Revision:             1.0
Serial Num:
Unformatted capacity: 25600.000 MBytes
Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x0
Device Type:          Disk device
Path(s):

/dev/rdsk/c0t600144F0E9434B4D000055B789470001d0s2
/devices/scsi_vhci/disk@g600144f0e9434b4d000055b789470001:c,raw
   Controller           /dev/cfg/c4
    Device Address              2100001b328e0682,0
    Host controller port WWN    21000024ff3ecf37
    Class                       secondary
    State                       STANDBY
   Controller           /dev/cfg/c4
    Device Address              21000024ff3e0b69,0
    Host controller port WWN    21000024ff3ecf37
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c2
    Device Address              2100001b328e0682,0
    Host controller port WWN    21000024ff3e851f
    Class                       secondary
    State                       STANDBY
   Controller           /dev/cfg/c2
    Device Address              21000024ff3e0b69,0
    Host controller port WWN    21000024ff3e851f
    Class                       primary
    State                       ONLINE

Checked for relevancy - 10-May-2018

Attachments

This solution has no attachment