Oracle ZFS Storage Appliance: Performance Issue when Pool is almost Full

Asset ID:	1-72-1392492.1
Update Date:	2018-02-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1392492.1 : Oracle ZFS Storage Appliance: Performance Issue when Pool is almost Full

Applies to:

Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)
NAS head revision : [not dependent]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Older firmware versions suffered from performance problems when storage pools are beyond some usage (80%). This is no longer the case, with all supported current Appliance Firmware releases.

There is now no such threshold, such that we will be missing space on disk and having write failures in the first place when the pool becomes full.

We will no longer experience degraded I/O performance, when performing write operations, but get IO errors and conclude that some clean up is needed in order to empty on-disk space for new data.

Observe

- Pool usage

cli> status show
Storage:
   pool-0:
      Used     8.58T bytes
      Avail     496B bytes
      State          online
      Compression:   1x
      Dedup:         1.03x

- Share usage

Here, default is the project, fs01 is a filesystem inside the project.

cli> shares select default select fs01 show
     ...
               space_available = 495G
                   space_total = 49K
                    root_group = other

From OS shell, check:

# zpool list
NAME   SIZE   ALLOC FREE CAP DEDUP HEALTH
pool-0 9.06T 8.58T 495G 94% 1.03x ONLINE

# zfs list -o space
NAME                                              AVAIL   USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
mypool01                                          8.92T   612K         0     31K              0       582K
mypool01/local                                    8.92T   111K         0     31K              0        80K
mypool01/local/default                            8.92T    80K         0     31K              0        49K
mypool01/local/default/fs01 8.92T    49K       18K     31K              0          0

Cause

Storage pools with a capacity approaching 100% usage.

When a pool is close to full, the ZFS algorithm has to deal with any free (and small) blocks to retrieve some space.

Solution

Administrators running into the full pool problem are advised to immediately seek to reduce their zpool utilization by removing older unwanted data including snapshots or to increase the size of their pools by adding further disk trays.

There are bugs in older Appliance code versions in the way ZFS looks for space when the pool becomes highly used causing performance troubles (basically ak.2010 code).

Check the current level of Appliance firmware that the system is running and upgrade to the latest code.

For latest appliance software revisions see MOS Document ID 2021771.1 - Oracle ZFS Storage Appliance: Software Updates

Back to Document 1331769.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues.

Checked for Currency - 09-JAN-2016

Checked for Currency - 04-FEB-2018

References

<NOTE:1331769.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Performance Issues
<NOTE:2021771.1> - Oracle ZFS Storage Appliance: Software Updates
<BUG:15378956> - SUNBT6525233 VDEV FULLNESS CAN DEGRADE PERFORMANCE, SHOULD CAUSE ZPOOL TO BECOME
<BUG:15661408> - SUNBT6975500 CAN'T STOP ZPOOL SCRUB WHEN POOL IS FULL

Attachments

This solution has no attachment