Oracle ZFS Storage: iSCSI Configuration Options for Windows, Linux, and Solaris

Asset ID:	1-79-1644939.1
Update Date:	2018-05-25
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1644939.1 : Oracle ZFS Storage: iSCSI Configuration Options for Windows, Linux, and Solaris

Applies to:

Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Purpose

Hosts connecting to the ZFS Storage Appliance over iSCSI have a number of different options to utilize depending the network layout. This document discusses the advantage and disadvantages of all available methods for Windows, Linux, and Solaris.

Details

Terminology

What is LACP?
Channel bonding, also known as LACP, combines two or more network interfaces into a single entity. It provides redundancy, link aggregation, and load balancing at Layer 2. It's available on Windows, Linux, and Solaris.

What is IPMP?
IP Multipathing (IPMP) groups multiple IP interfaces to provide redundancy at Layer 3. It's specific to Solaris only and available as part of the ZFS Storage Appliance.

What is multipath?
Multipathing uses path redundancy to aggregate and distribute data over multiple links. It sits above the SCSI layer and can provide service to Fibre Channel, iSCSI, iSER, and SRP. It's been implemented by the following OS's:

Windows - MPIO
Linux - DM-MPIO
Solaris - MPxIO

Windows

When should I choose MPIO?
Always. MPIO is the only recommended solution for Windows and the ZFS Storage Appliance.

How do I configure MPIO?
Refer to Using Sun ZFS Storage Appliance iSCSI LUNs in a Microsoft Windows Server 2008 R2 Environment

When should I choose LACP?
If your network links are leveraging CIFS/SMB traffic in addition to iSCSI, LACP can be used to aggregate throughput across all protocols.

When should I not choose LACP?
A dedicated SAN-like environment that only runs iSCSI does not require the functionality of LACP. MPIO includes link aggregation.

Linux

When should I choose multipath?
Multipath is for a more standardized SAN environment. It can load balance evenly across multiple links without any additional switch configuration. Because of this, it can easily span different VLANs and subnets. It can also load balance a single thread of IO across multiple active paths.

When should I not choose multipath?
A target and initiator with multiple paths connected to the same subnet may introduce ARP flux. Each outbound link needs to be associated with a different subnet. Additionally, if the iSCSI interfaces are shared amongst other protocols (such as NFS or CIFS), link aggregation is not possible for these protocols.

How do I use multipath on Oracle Linux?
Refer to Configuring Multipathing for Oracle Linux and the Oracle ZFS Storage Appliance.

When should I choose bonding?
Bonding should be used when your NAS infrastructure is simple and the initiator and target are on the same subnet.

Why should I not choose bonding?
The networking situation of bonding can become extremely complex when adding in VLAN tagging, different subnets, and LACP across multiple switches. Configuration changes need to be done at the initiator, target, and switch layer. This adds more administrative overhead and constrains the growth of your storage. Additionally, a single thread of IO will not be evenly spread out with any LACP algorithm.

How do I create a bond on Oracle Linux?
Refer to Configuring Network Interface Bonding.

Can I implement both bonding and multipathing?
No. This may cause instability with two load-balancing tools competing against one another.

Solaris

When should I use MPxIO?
Always. MPxIO is enabled by default in Solaris 10/11 x86, but disabled in SPARC. It is designed to interoperate with IPMP/LACP and should not be disabled.

Enabling on SPARC

# stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) y
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y) y
updating /platform/sun4v/boot_archive
rebooting...

When should I choose IPMP?
Multi-protocol networks (NFS,CIFS,ISCSI) where a network SPOF is unacceptable.

Why should I not choose IPMP?
If you have no LACP infrastructure, but desire throughput enhancements using link aggregation, rely on the underlying functionality of MPxIO rather than IPMP. Currently, Active/Active configurations of IPMP can cause packet flooding in certain network setups. See bug 15587188 for more information.

How do I create an IPMP link?

Example IPMP link using two interfaces

# ipadm create-ip net1
# ipadm create-ip net2
# ipadm create-ipmp ipmp0
# ipadm add-ipmp -i net1 -i net2 ipmp0
# ipadm create-addr -T static -a 192.168.1.1/24 ipmp0/v4
# ipadm create-addr -T static -a 192.168.1.2/24 net1/test
# ipadm create-addr -T static -a 192.168.1.3/24 net2/test
# ipadm set-ifprop -p standby=on -m ip net2

When should I choose LACP?
LACP provides the best method for increasing throughput via aggregation for multi-protocol networks. This does require that each connected device in your network supports the same trunking procedure.

When should I not choose LACP?
If you do not value link aggregation, avoid the use of LACP. It requires configuration changes at the host, switch, and storage layer and will introduce additional complexity with advanced network options such as VLAN tagging and multiple switches.

How do I create an LACP group?

Example aggregation using two interfaces

# dladm create-aggr -l net1 -l net2 aggr0
# ipadm create-ip aggr0
# ipadm create-addr -T static -a 192.168.1.1/24 aggr0/v4

Can I implement IPMP on top of LACP?
Yes. Use an Active/Standby IPMP layout that sits on top of a previously created LACP groups.

Can I do neither IPMP nor LACP?
Yes. This makes the most sense in a SAN-like environment that's running only iSCSI. It's still advisable to at least leverage multiple network links in any iSCSI configuration.

iSCSI clients should not have a too-short session replacement timeout value

NOTE: From the Appliance Firmware Release Notes under MOS Document ID 2021771.1 - Oracle ZFS Storage Appliance: Software Updates

2013.1.2.2 - Release Notes
2011.1.9.0 - Release Notes

    Some iSCSI clients configure a default session replacement timeout value or connection retry timeout
    value that is too short, and can lead to timeouts during cluster takeover and failback scenarios
    particularly when there are very large numbers of logical units and/or shares exported.
    It is recommended that the iSCSI session replacement timeout or connection timeout value be set to
    at least 240 seconds; this is a configuration option that should be applied on the client side -
    there is nothing on the appliance server that needs to be changed. Please consult the appropriate
    client software documentation to modify this configuration setting.

Examples - setting iSCSI timeout parameter to 240 seconds:

Solaris

# iscsiadm modify initiator-node -T conn-login-max=240

Windows - Modify either Disk class driver's TimeOutValue OR SrbTimeoutDelta

Linux - Modify /etc/iscsi/iscsid.conf to have node.session.timeo.replacement_timeout = 240

[ NOTE: For PCA environment, we have seen better results with node.session.timeo.replacement_timeout = 300 ]

***Checked for relevance on 25-MAY-2018***

Attachments

This solution has no attachment