Oracle ZFS Storage Appliance: Performance Issues - IO Reads with SSD cache devices

Asset ID:	1-72-2224745.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2224745.1 : Oracle ZFS Storage Appliance: Performance Issues - IO Reads with SSD cache devices

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS4-4 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Overall performance can be slower than expected on systems using SSD cache devices.
For example, nightly batch-jobs are taking longer after system was upgraded from 2013.06.05.4.0,1-1.7 to 2013.06.05.4.2,1-1.1.

Cause

Bug 23526362 - mptsas driver TRAN_BUSY on readzilla degraded shadow-migration performance

This causes the SSD cache device(s) to be needlessly busy affecting especially read operations, but in the end having an impact on performance as a whole.

Signature of this bug:

analytics
- the SSD cache devices can be more than 70% busy (Use cli> analytics datasets create io.disks[utilization=70][disk])
- the latency for SSD cache devices on IO reads is greater than 1ms (Use : cli> analytics datasets create io.ops[latency=1000][op=read][disk])
From inside the ZFFSA support bundle, status/iostat.out shows this for SSD cache devices :
- %b > 70 : percent of time the disk is busy (transactions in progress)
- wait > 1 : average number of transactions waiting for service (queue length) or host queue depth
- actv >= 1 : active queue is the number of pending I/O requests. This represents the number of threads doing IOs onto the same device
- asvc_t > 1ms : service time should be around 0.5ms max for SSD cache devices
- Examples :

   r/s     w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
1155.2   13.5   29.5   12.7 8.9  1.0 7.6    0.8 100  98 c0t5001E8200276417Cd0 << SSD cache device
1111.7   15.1   26.0   13.9   8.9  1.0 7.9 0.9 100  99 c0t5001E8200276417Cd0
1130.1   14.9   29.6   13.2   8.9  1.0 7.7 0.9 100  99 c0t5001E8200276417Cd0
1158.2   13.6   29.5   12.5 8.9  1.0 7.6 0.8 100  99 c0t5001E8200276417Cd0
1090.4   14.3   30.2   12.9   8.9  1.0 8.0 0.9 100  99 c0t5001E8200276417Cd0
1133.1   14.3   27.4   12.9   8.9  1.0 7.7 0.9 100  99 c0t5001E8200276417Cd0
  958.1   23.0   21.6   18.2 8.4  1.0 8.6 1.0 98  98 c0t5001E8200276417Cd0
  888.0   24.3   21.2   20.0   8.6  1.0 9.4 1.1 99  98 c0t5001E8200276417Cd0

r/s w/s    kr/s    kw/s wait actv wsvc_t asvc_t %w %b device
124.0     61.0 1423.0 38681.3 5.0   1.3   26.8 7.3 66 68 c0t5001E82002763F90d0 << SSD cache device
263.0     51.0 2233.4 36262.7 4.9   1.4   15.7 4.4 65 71 c0t5001E82002763F90d0
95.0     61.0 956.0 43606.2 6.8   1.8 43.6   11.8 89 95 c0t5001E82002763F90d0
288.0     56.0 4157.5 41983.7 5.5   1.5 16.0 4.4 73 76 c0t5001E82002763F90d0
172.0 64.0 1422.9 43494.1 5.4   1.4 22.8 6.1 69 74 c0t5001E82002763F90d0
55.0     58.0 517.1 40532.3 5.2   1.4 45.8   12.4 68 70 c0t5001E82002763F90d0
302.0     57.0 2640.2 44563.4 4.7   1.4   13.1 3.9 64 71 c0t5001E82002763F90d0
257.0     57.0 2928.9 40366.9 4.8   1.3   15.4 4.2 63 68 c0t5001E82002763F90d0

Solution

Bug 23526362 has been fixed since 8.6.11 (aka 2013.1.6.11)

References

<BUG:23526362> - MPTSAS DRIVER TRAN_BUSY ON READZILLA DEGRADED SHADOW-MIGRATION PERFORMANCE
<NOTE:1331769.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues

Attachments

This solution has no attachment