Oracle FS1-2 Flash Storage System
 

PLA-Support>Sun Systems>DISK>Flash Storage>SN-EStor: FSx
 

In this Document

Symptoms

Changes

Cause

Solution

References

Created from <SR 3-11131042491>

Applies to:

Symptoms

This document highlights potential performance issues when the low performance tier (Capacity HDD) does not have enough Drive Groups.

The issue has been seen on a system with a single Capacity HDD Drive Enclosure (2 Drive Groups). Most of the LUNs are set to use Auto-Tiering.  The rest of the configuration has one Drive Group of Performance SSD and 6 Drive Groups of Performance Disk.

The slow response time is noticeable on the applications running on the hosts due to the latency on the Flash Storage.

On the FS1 GUI, under SAN -> Statistics and Trending -> LUNs, the IO Latency can be over 40 ms (milliseconds) for most of the LUNs.

The problem to a specific Drive Group can be isolated under System -> Reporting & Statistics -> Drive Group Statistics, double click on each Drive Group and measure the Total IO/s and Total Latency: the affected Drive Group(s) should report a latency of 100 ms or higher when the performance issue is observed.

Changes

 Performance is affected when multiple access to Capacity HDD Drive Groups is done simultaneously, for example, weekly backups.

Cause

Auto-Tiering uses the following Storage Classes as Tiers sorted by performance in descending order:

• Perf SSD (solid-state disks):   Flash Drives with single-level cell (SLC) technology, very fast to Write IOs
• Cap SSD: Flash Drives with multi-level cell (MLC) technology, less efficient than SLC but cheaper in price
• Perf HDD: SAS disks used for performance.
• Cap HDD: SAS disks with a lower speed but higher capacity compared to the SAS disks used for performance

Over time, LUN hot spots (heavy IO) are migrated to higher tiers whereas LUNs receiving fewer IO requests are migrated to lower tiers.  So hot data would go to the Performance SSD Drive Groups, some proportion of warm data would go to the 6 Perf Disk Drive Groups and a the remainder of cold data to the 2 Capacity HDD Drive Groups.

The latencies generated by the slow tier due to multiple access can affect the overall performance of the FS1-2.  This is especially true for the Auto-Tier LUNs that are waiting for the cold data to be processed before the Hosts can query other data.

Solution

The solution is to add more Capacity HDD Drive Enclosures to improve the response time of that Tier.

There are a few workarounds:

• Modify the most critical LUNs to transfer their cold data off the slowest tier: 
  1. In the FS1 GUI under SAN -> Storage -> LUNs, select the LUN to improve performance.
  2. Go to the Actions menu, select Modify, click on the Additional Options button and uncheck Capacity Disk. The change should be reflected a few minutes later by looking at the allocation graph on the Modify LUN page.
• Modify the Allowable Storage Capacity for Auto-Tiered LUNs:
  1. Under the System tab, go to the Storage Domain menu.
  2. Double click on the affected Storage Domain.
  3. On the right hand side of the pop-up window, reduce the Allowable Storage Capacity for the tier with latencies so some of the data will be dynamically moved to better tiers.

In addition, Auto-Tier LUNs should not have their Initial Storage Class set to Capacity Disk if that class has only 2 Drive Groups. The Initial Storage Class can be viewed or modified on the FS1 GUI under SAN -> Storage -> LUNs and using the View or Modify options in the Actions menu. The user might have to uncheck the Use Storage Profile option to modify the Initial Storage Class.
The Initial Storage Class is where the Auto-Tier LUN receives Write IOs (the new data will be automatically affected later to the appropriate tier based on the usage). The Progression Map virtual LUN (PM vLUN) is also stored on one of the Drive Groups in the Initial Storage Class. The PM vlun is a small metadata LUN that generates a lot of traffic to manage QoS+ statistics; there is one PM vlun for each Auto-Tier LUN.

It is best practice to have the Initial Storage Class of Auto-Tier LUNs to the best tier (Perf SSD) but also on the 2nd best tier (Cap SSD or Perf HDD) so the PM vluns are well spread across the Drive Groups (a PM vlun is assigned to one Drive Group; having all the PM vluns on the same Drive Group can create contention).
For example, if there is only one Drive Group in the Perf SSD but 6 Drive Groups of Perf HDD within the same Storage Domain, most critical Auto-Tier LUNs (database redo logs, indexes) with a High/Premium priority should have the Initial Storage Class set to Perf SSD and the rest set to the second tier (Perf HDD) as long as it has a sufficient number of Drive Groups (6 Drive Groups is a good number for Perf HDD).

To confirm the latencies, run /cores_data/local/tools/pillar/bin/getTraceDataR6.pl against the logs of both controllers (where the file output*.txt is located):

Example:

In this case, Drive Groups 0 and 5 are on the Capacity HDD DE and their average response time is above 400 milliseconds.

Finally, check with /cores_data/local/tools/pillar/FlashStorage/R6_RAID_PortAnalysis.pl on both controllers logs that all the drives in both Drive Groups have a high Average Ticks and Max Ticks compared to the rest of the drives.

Example:

The 24 drives in bold belong to the Capacity HDD DE.

Verify with ‘/cores_data/local/tools/pillar/FlashStorage/FSInfo.pl -u’ if there is any discrepancies between the tiers allocation.

Here is another tip to find out the allocation of the specific Auto-Tier LUN:

Search for the LUN in  A1*.chsh.xml and scroll down to the GeometryCapacities tag:

References

This solution has no attachment