| Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback |
| Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition | ||
|
|
||
 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Solution Type Technical Instruction Sure Solution 1628999.1 : Oracle ZFS Storage Appliance: How to set up Client Multipathing
In this Document
Applies to:Oracle ZFS Storage ZS3-2 - Version All Versions and laterSun ZFS Storage 7420 - Version All Versions and later Sun ZFS Storage 7320 - Version All Versions and later Oracle ZFS Storage ZS3-4 - Version All Versions and later 7000 Appliance OS (Fishworks) GoalThis document provides the reader with instructions for configuring client multipathing for the Oracle ZFS Storage Appliance environment. SolutionConfiguring FC Client MultipathingThe Oracle ZFS Storage appliance uses Asymmetric Logical Unit Access (ALUA) to provide FC target multipathing support. Please refer to SCSI Primary Commands (SPC) definition on t10 at http://www.t10.org if you need more information.
The following instructions provide a guide for setting up the Fibre Channel (FC) host clients that are connected to a FC target enabled clustered appliance. TasksConfiguring Solaris InitiatorsMPxIO is enabled on Solaris x86 platforms but disabled on SPARC by default. The mpathadm show LU command shows the path state changing from active to standby or standby to active. Alternately, you can also use luxadm display to show path state.
Configuring Windows InitiatorsALUA multipathing is supported by native Windows 2008/R2 and 2012 GA/R2 MPIO only. 2008 R2 is required to avoid issues during cluster failover and takeover. 2008 SP2 is not supported.
Windows Tunables - Microsoft DSM Details
Configuring Linux InitiatorsThe following instructions apply to these clients:
2. Stop the multipathd service. # service multipathd stop Stopping multipathd daemon: [ OK ] 3. Add the appropriate /etc/multipath.conf stanza shown below, replacing PRODUCT with the output of the following CLI script: run('cd /');
run('configuration version');
printf(get('product').replace(/^Sun /,"") + '\n');
For example: spadefish:> script
("." to run)> run('cd /');
("." to run)> run('configuration version');
("." to run)> printf(get('product').replace(/^Sun /,"") + '\n');
("." to run)> .
ZFS Storage 7420
or
ZFS Storage 7330
device
{
vendor "SUN"
product "ZFS Storage.*"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
hardware_handler "0"
path_grouping_policy group_by_prio
failback immediate
no_path_retry queue
rr_min_io 100
path_checker tur
rr_weight uniform
}
defaults {
find_multipaths yes
user_friendly_names yes
}
devices {
device {
vendor "SUN"
product "ZFS Storage.*"
getuid_callout "/sbin/scsi_id -g -u -p 0x83 -s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 600
rr_min_io 100
path_checker tur
rr_weight uniform
features "0"
}
}
defaults {
find_multipaths yes
user_friendly_names yes
}
devices {
device {
vendor "SUN"
product "ZFS Storage.*"
getuid_callout "/sbin/scsi_id -g -u -p 0x83 -s /block/%n"
prio alua
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 600
rr_min_io 100
path_checker tur
rr_weight uniform
features "0"
}
}
defaults {
find_multipaths yes
user_friendly_names yes
}
devices {
device {
vendor "SUN"
product "ZFS Storage.*"
getuid_callout "/lib/udev/scsi_id --page=0x83 --whitelisted --device=/dev/%n"
prio alua
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 600
rr_min_io_rq 100
path_checker tur
rr_weight uniform
features "0"
}
}
defaults {
find_multipaths yes
user_friendly_names yes
}
devices {
device {
vendor "SUN"
product "ZFS Storage.*"
prio alua
hardware_handler "1 alua"
path_grouping_policy group_by_prio
path_selector "round-robin 0"
failback immediate
no_path_retry 600
rr_min_io_rq 100
path_checker tur
rr_weight uniform
features "0"
}
}
defaults {
multipath_tool "/sbin/multipath -v0"
udev_dir /dev
polling_interval 100
default_selector "round-robin 0"
default_path_grouping_policy group_by_prio
default_getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"
default_prio_callout "/bin/true"
prio "alua"
default_features "0"
rr_min_io 100
failback immediate
user_friendly_names yes
path_checker tur
no_path_retry 1000
}
device {
vendor "SUN"
product "ZFS Storage.*"
}
defaults {
find_multipaths yes
user_friendly_names yes
}
devices {
device {
vendor "Sun"
product "ZFS Storage ####"
getuid_callout "/lib/udev/scsi_id -g -u --device=/dev/%n"
prio alua
features "1 queue_if_no_path"
hardware_handler "1 alua"
path_grouping_policy group_by_prio
failback immediate
no_path_retry 600
path_selector "round-robin 0"
rr_weight uniform
rr_min_io_rq 100
path_checker tur
}
}
4. Enable multipath and verify by starting the multipathd service. #service multipathd start Starting multipathd daemon: [ OK ] 5. Run the multipath command after the SCSI bus rescan is finished to verify multipath I/O is enabled. Note that standby paths will be shown as [failed][faulty] due to a known Linux bug. For this reason, it is recommended that users verify the paths are actually operational before putting the system into production. For more details, refer to the Troubleshooting section below. and/or the Appliance Online Help wiki (https://<NAS_IP_ADDRESS>:215/wiki/index.php/Configuration:SAN:FC#Troubleshooting). #multipath –ll Linux SCSI bus scanning
In current versions of Oracle Linux and Red Hat Enterprise Linux, getting the system to recognize additions, modifications, and removals of logical units attached over a SAN can be tricky. The tool most widely used and recommended for doing this is /usr/bin/rescan-scsi-bus.sh. When adding new Fibre Channel logical units or modifying existing logical units, (e.g., resizing), the following usage purports to apply the changes within the host's SCSI subsystem: # rescan-scsi-bus.sh -i And when removing Fibre Channel logical units, the following usage purports to eject the device from the host's SCSI subsystem: # rescan-scsi-bus.sh -ir However we have found these commands to be finicky at best, and at times they appear not to work at all. Red Hat's documentation states the following about the rescan-scsi-bus.sh utility at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/logical-unit-add-remove.html: When using the rescan-scsi-bus.sh script, take note of the following known issues:
In our testing we have found it necessary to run rescan-scsi-bus.sh twice as a matter of routine, not just when logical units are mapped for the first time. It's very important to wait until the utility has finished working before repeating the scan, or doing anything else with the devices. You can observe the utility's progress by tailing the kernel messages buffer. Thus, our procedure for rescanning looks like this: # rescan-scsi-bus.sh -i
[wait until scanning finishes]
# rescan-scsi-bus.sh -i
As noted above, the -r switch is supposed to make the utility remove the devices, but we've found that it doesn't work with either iSCSI or Fibre Channel. The solution for device removal is to remove them manually using the sysfs "delete" controls. For example, if want to remove the Fibre Channel device whose node is /dev/sdc, we'd use the following command: # echo '1' > /sys/block/sdc/device/delete You must do this for the device node associated with every path to the logical unit that is to be removed. Also note the importance of following the proper procedures for removing devices. For Red Hat based systems, these procedures can be found in Red Hat's online Storage Administration guide. The page for RH 7 is at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html. Configuring Mac InitiatorsSample Output: NOTE: MUST BE RUN AS ROOT!!! If you're not logged is as root you need to do that before the command will work!!! Configuring OVM
devices {
devices {
devices { Configuring VMware ESX Initiators VMware ESX HCL Refer to the VMware Compatibility Guide for supported Oracle products. http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=san&key=Oracle For vSphere 5: For ESX 4.1 servers: No addition rules need to be added because the default driver is ALUA aware. For ESX 4.0 servers: 1. Verify the current SATP plugin that is in use by issuing the esx nmp device list command # esxcli nmp device list
naa.600144f0ed81720500004bb3c1f60002
Device Display Name: SUN Fibre Channel Disk (naa.600144f0ed81720500004bb3c1f60002)
Storage Array Type: VMW_SATP_DEFAULT_AA
Storage Array Type Device Config:
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba0:C0:T1:L0;current=vmhba0:C0:T1:L0}
Working Paths: vmhba0:C0:T1:L0
VMW_SATP_DEFAULT_AA is the default plugin. This plugin is not ALUA-capable. 2. Add rules to enable the ALUA plugin for the appliances by using the esxcli nmp satp addrule command. # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7120" -c "tpgs_on" # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7320" -c "tpgs_on" # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7420" -c "tpgs_on" 3. Verify the rule was correctly added. # esxcli nmp satp listrules | grep SUN VMW_SATP_ALUA SUN ZFS Storage 7120 tpgs_on ZFS Storage 7000 VMW_SATP_ALUA SUN ZFS Storage 7320 tpgs_on ZFS Storage 7000 VMW_SATP_ALUA SUN ZFS Storage 7420 tpgs_on ZFS Storage 7000 4. Reboot the VMware ESX server. When server has rebooted, check to ensure the correct plugin is now in effect with the esxcli nmp device list command. # esxcli nmp device list
naa.600144f0ed81720500004bb3c1f60002
Device Display Name: SUN Fibre Channel Disk (naa.600144f0ed81720500004bb3c1f60002)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off;
explicit_allow=on;alua_followover=on;{TPG_id=0,TPG_state=STBY}{TPG_id=1,TPG_state=AO}}
Path Selection Policy: VMW_PSP_MRU
Path Selection Policy Device Config: Current Path=vmhba1:C0:T1:L0
Working Paths: vmhba1:C0:T1:L0
TroubleshootingThis section describes troubleshooting known issues. Multipath-tools version 0.4.7 bundled in OEL 5.4 is unable to recognize paths in ALUA standby access state In SCSI spec, a target port which is in standby state does not respond to Test Unit Ready command, so standby paths are shown as [failed] in multipath command output. The fix for this problem is committed into the multipath-tool source tree on 2009-04-21 (which is later than its 0.4.8 official release). Users have to obtain the latest version of the multipath-tool source code from: http://christophe.varoqui.free.fr/ Users should get the latest source code from its git repository. The multipath-tools-0.4.8.tar.bz2 tarball does not contain the fix. Finally, the status shown in multipath command output does not impact functionality like I/O and failover/failback, so updating the package is not mandatory. See Also
http://www.oracle.com/technetwork/documentation/oracle-unified-ss-193371.html
http://www.oracle.com/technetwork/documentation/solaris-11-192991.html
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/ Configuring iSCSI/iSER Client MultipathingSolaris iSCSI/iSER and MPxIO ConsiderationsMPxIO supports target port aggregation and availability in Solaris iSCSI configurations that configure multiple sessions per target (MS/T) on the iSCSI initiator.
Configuring SRP Client MultipathingVMWare 4.0The VMware Native MultiPath Plugin (nmp) has two components that can be changed on a device by device, path by path, or array basis. Path Selection Plugin (psp)This plugin controls which physical path is used for I/O: # esxcli nmp psp list
Name Description
VMW_PSP_MRU Most Recently Used Path Selection
VMW_PSP_RR Round Robin Path Selection
VMW_PSP_FIXED Fixed Path Selection
Storage Array Type Plugin (satp)This plugin controls how failover works: The SATP has to be configured to recognize the array vendor or model string in order to change the basic failover mode from a default Active/Active type array to ALUA. By default the appliance cluster was coming up as a Active/Active array only. Use the ESX CLI to add rules to have the ALUA plugin claim the 7000 luns. # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7120" -c "tpgs_on" # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7320" -c "tpgs_on" # esxcli nmp satp addrule -s VMW_SATP_ALUA -e "ZFS Storage 7000" -V "SUN" -M "ZFS Storage 7420" -c "tpgs_on"
options are: -s VMW_SATP_ALUA - for the ALUA SATP
-e description of the rule
-V Vendor
-M Model
-c claim option for Target Portal Group (7000 seems to support implicit)
If no luns have been scanned/discovered, you can simply rescan the adapter to find new luns. The luns will be claimed by the ALUA plugin. If luns are already present, reboot the ESX host. After the reboot, you will see the luns being listed under the VMW_SATP_ALUE array type. # esxcli nmp device list naa.600144f096bb823800004b707f2d0001
Device Display Name: Local SUN Disk (naa.600144f096bb823800004b707f2d0001)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config:
{implicit_support=on;explicit_support=off;explicit_allow=on;
alua_followover=on; {TPG_id=0,TPG_state=AO}{TPG_id=1,TPG_state=STBY}}
Path Selection Policy: VMW_PSP_MRU
Path Selection Policy Device Config: Current Path=vmhba_mlx4_1.1.1:C0:T1:L0
Working Paths: vmhba_mlx4_1.1.1:C0:T1:L0
Relevant lun path lists will show an Active and a Standby path # esxcli nmp path list gsan.80fe53553e0100282100-gsan.80fe8f583e0100282100
-naa.600144f096bb823800004b707f2d0001
Runtime Name: vmhba_mlx4_1.1.1:C0:T2:L0
Device: naa.600144f096bb823800004b707f2d0001
Device Display Name: Local SUN Disk (naa.600144f096bb823800004b707f2d0001)
Group State: standby
Storage Array Type Path Config:
{TPG_id=1,TPG_state=STBY,RTP_id=256,RTP_health=UP}
Path Selection Policy Path Config: {non-current path}
gsan.80fe53553e0100282100-gsan.80fe73583e0100282100
-naa.600144f096bb823800004b707f2d0001
Runtime Name: vmhba_mlx4_1.1.1:C0:T1:L0
Device: naa.600144f096bb823800004b707f2d0001
Device Display Name: Local SUN Disk (naa.600144f096bb823800004b707f2d0001)
Group State: active
Storage Array Type Path Config:
{TPG_id=0,TPG_state=AO,RTP_id=2,RTP_health=UP}
Path Selection Policy Path Config: {current path}
VMWare ESX 4.0 Issues
The esxcl nmp path list command will report an active and a standby path one each for the SRP targets in a cluster configuration. [root@ib-client-5 vmware]# esxcli nmp path list
gsan.80fe53553e0100282100-gsan.80fe8f583e0100282100-
naa.600144f096bb823800004b707f2d0001
Runtime Name: vmhba_mlx4_1.1.1:C0:T2:L0
Device: naa.600144f096bb823800004b707f2d0001
Device Display Name: Local SUN Disk
(naa.600144f096bb823800004b707f2d0001)
Group State: standby
Storage Array Type Path Config:
{TPG_id=1,TPG_state=STBY,RTP_id=256,RTP_health=UP}
Path Selection Policy Path Config: {non-current path}
gsan.80fe53553e0100282100-gsan.80fe73583e0100282100-
naa.600144f096bb823800004b707f2d0001
Runtime Name: vmhba_mlx4_1.1.1:C0:T1:L0
Device: naa.600144f096bb823800004b707f2d0001
Device Display Name: Local SUN Disk
(naa.600144f096bb823800004b707f2d0001)
Group State: active
Storage Array Type Path Config:
{TPG_id=0,TPG_state=AO,RTP_id=2,RTP_health=UP}
Path Selection Policy Path Config: {current path}
When this problem occurs, the active or standby path may not be shown in the output of esxcli nmp path list. Workaround: None
When this problem happens, the Linux guest system log will report in its /var/log/messages log: Feb 10 16:10:00 ib-client-5 vmkernel: 1:21:41:36.385 cpu3:4421)<3>ib_srp: Send tsk_mgmt target[vmhba_mlx4_1.1.1:2] out of TX_IU head 769313 tail 769313 lim 0 Workaround: Reboot guest VM
Attachments This solution has no attachment |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||