Oracle ZFS Storage Appliance: Performance Issue During Backups

Asset ID:	1-72-2389560.1
Update Date:	2018-05-18
Keywords:

Solution Type Problem Resolution Sure

Solution 2389560.1 : Oracle ZFS Storage Appliance: Performance Issue During Backups

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-BA - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS4-4 - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS5-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

After network changes: Heads are having performance issues with very high response time for backups.

We see on one of the ZFSSA nodes that the logzilla disks remain over 95% in use during the two remaining backups.

In our case we found the disk were being heavily hit for writes and the logzillas were 100% busy.

Changes

Prior to the performance issue there were some network changes made. Switch spine-leaf activity was done, they are working fine now.

Cause

The maintenance that was done for the network switches caused a backlog of backups (There may be hung backups that may need to be cleaned up via the backup software).

The snowball of backups will effect write performance until the backups complete.

Discovered that the system backups are done every Saturday Night/Sunday Morning.

Looking at the analytics we found the array write performance peaks during the backups which run every week at that time.

Unfortunately they decided to do network maintenance when the system was at its highest workload, not the other way around.

Solution

Wait for the backlog of backups to complete.

The logzillas will free up and performance will come back to normal.

References

<NOTE:2359446.1> - How to Upload and Execute a Workflow on a ZFS Storage Appliance
<NOTE:1547088.2> - How to Upload Files to Oracle Support
<NOTE:1019887.1> - Sun Storage 7000 Unified Storage System: How to Collect a Support Bundle using the BUI or CLI
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices

Attachments

This solution has no attachment