Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

The purpose of the document is to provide a general guideline on how to calculate the Queue Depth settings to be applied to SAN hosts that access the Flash Storage array.

In this Document

Goal

Solution

Applies to:

Goal

Verify that the Queue Depth is appropriately set on the hosts. Setting a Queue Depth too high or too low on the hosts can affect the SAN performance.

Solution

The Flash Storage array can have a total of 12 Fibre Channel/iSCSI HBA ports, 6 Fibre Channel/iSCSI ports per Controller. Each Controller can have up to 3 dual port HBAs (Slot1, Slot4 and Slot5).

The HBA ports are QLogic HBA's, the SCSI Command queue is also known as the Queue Depth for the HBA adapter. For Fibre Channel, each HBA port can handle 2048 outstanding I/O requests per port at any one time.  For iSCSI, each HBA port can handle 512 outstanding I/O requests per port at any one time.

In setting host side Queue Depth our default recommendation is a maximum of 64.

The recommended Queue Depth settings to be applied on the hosts are determined by 2 factors:
•    Total number of LUNs exposed to the target ports
•    Number of host ports accessing the target ports

Below is the formula that can be used to determine the Queue Depth settings on the hosts:

Details:

Queue Depth: the value to set on the host.

Target Port Queue Depth: the maximum number of outstanding I/O requests on an FS1 port (2048 for FC).

Number of host ports: these are the number of initiators accessing the LUNs on the FS1.

Example:
A single FS1 containing Fibre Channel HBA ports, has for each one of its target ports 4 initiators zoned in the switches (only one fabric). 16 LUNs are mapped to each one of these 4 initiators.
So in general, a target port provides access to 16 LUNs using 4 initiators (host ports).

Another example where a critical host (with 2 initiators) is zoned to the ports of the second HBA (Slot 4) of CONTROLLER-01 and CONTROLLER-02. The ports on Slot 4 are exclusively used with that host for performance purposes.
32 LUNs are mapped to the 2 initiators.
There are two SAN fabrics so one initiator is connected to CTRL-01 Slot 4 Port 0 and CTRL-02 Slot 4 Port 0 and the second initiator to CTRL-01 Slot 4 Port 1 and CTRL-02 Slot 4 Port 1.
Each target port is only zoned to one initiator as there are separate fabrics.

In clustered environments like IPStor, two hosts are zoned to the Flash Storage and all the LUNs should be mapped to the two hosts. However, the IOs on each LUN are performed by a single host only so the variable “number of host ports” can be divided by two (or by 4 if there are 2 fabrics).
It is different with VMware where multiple hosts access the same LUNs (a LUN/Datastore usually contains multiple VMs owned by separate hosts).

The same formula can be applied if your Flash Storage is using iSCSI.  In iSCSI each HBA port can handle 512 outstanding I/O requests per port at any one time.

A more realistic scenario is where there are two SAN fabrics and the initiators are not connected to the same amount of LUNs. For example, 3 initiators are connected to 8 LUNs and 4 initiators are connected to 10 LUNs.

If the first 3 initiators are for very busy hosts running critical applications, you can try to tweak these values (give more than a 32 Queue Depth for the first 3 hosts and less than 32 for the 4 initiators) as long as the total does not exceed 2048:

So the first 3 initiators can be set with a Queue Depth of 58 and the 4 initiators with a Queue Depth of 16.

Please make sure to follow the HBA vendor or iSCSI software initiator configuration to set the Queue Depth to the appropriate settings. Please also refer to the FSPM documentation for your hosts.

Some examples:

• Solaris 10 and lower:
  • For Sparc, edit /kernel/drv/ssd.conf and add the following line:         ssd-config-list = "Oracle  Oracle FS", "cache-nonvolatile:true, disksort:false, throttle-max:32";
  • For x86, edit /kernel/drv/sd.conf and add the following line:             sd-config-list = "Oracle  Oracle FS", "cache-nonvolatile:true, disksort:false, throttle-max:32";
  • The value of throttle-max (Queue Depth) can be adjusted.
  • Reboot the host to apply the change.

   

• Solaris 11:
  • Check if file /etc/driver/drv/ssd.conf exists, if not then copy ssd.conf from /kernel/drv/ to /etc/driver/drv/
  • For Sparc, edit /etc/driver/drv/ssd.conf and add the following line:         ssd-config-list = "Oracle  Oracle FS", "cache-nonvolatile:true, disksort:false, throttle-max:32";
  • For x86, edit /etc/driver/drv/sd.conf and add the following line:             sd-config-list = "Oracle  Oracle FS", "cache-nonvolatile:true, disksort:false, throttle-max:32";
  • The value of throttle-max (Queue Depth) can be adjusted.
  • Reboot the host to apply the change.

   

• Linux:
• For Emulex, edit /etc/modprobe.conf and set lpfc_lun_queue_depth=64 (or a lower value if needed) at the "options" line, rebuild the kernel and reboot the host (or use the utilities supplied by Emulex). Refer to the Emulex website or the FSPM manual for Linux for more details.
• For QLogic, edit /etc/modprobe.conf and set ql2xmaxqdepth=64 (or a lower value if needed) at the "options" line, rebuild the kernel and reboot the host (or use the utilities supplied by QLogic). Refer to the QLogic website or the FSPM manual for Linux for more details.

• Windows:
  • Install the HBA utilities from the manufacturer to set the following values:
    • QLogic: Execution Throttle = 64 (or lower if needed).
    • Emulex: Queue Depth = 64 (or lower if needed).
  • See the FSPM manual for Windows for more options.

   

• ESX:
  • Changing the Queue Depth for QLogic, Emulex and Brocade HBAs is explained on the following Knowledge Article: VMWare KB 1267

This solution has no attachment