Sun ZFS Storage 7420
 
Oracle ZFS Storage ZS3-2
 
Sun Storage 7110 Unified Storage System
 
Sun Storage 7210 Unified Storage System
 
Sun Storage 7410 Unified Storage System
 
Sun Storage 7310 Unified Storage System
 
Oracle ZFS Storage ZS3-4
 
Sun ZFS Storage 7120
 
Sun ZFS Storage 7320
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 
_Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
 

In this Document

Purpose

Troubleshooting Steps

Preliminary system checks

Step 1 - Framing the problem

Step 2 - Expected performance for various configurations.

Step 3 - Configuration and tuning

Step 4 - Known performance issues.

Step 5 - Collecting analytics data for performance issues.

Step 6 - other data collection.

References

Applies to:

Purpose

This document is provided to assist in troubleshooting performance issues.

It will help to frame the problem, determine what can be expected for various configurations, identifies some known issues and provide some guidelines to get the best possible performance for iSCSI, CIFS, NFS and FC protocols.

Finally, it provides some information to collect analytics in order to isolate the issue and provide required information to ORACLE support team, if needed.

This document has been written as a resolution path, each step giving links to other specific documents. It should be kept as a reference for any performance issue on ZFS storage appliance.

In some situations when analytics is not enough, some dtrace scripts can be used to narrow down the issue. Performance and Applications Engineering team (PAE) has written a tool to help. See Step 6 : Other data collection.

Troubleshooting Steps

Preliminary system checks

Before going through further steps, consider the following possible causes of performance issues. This may provide some easy clues.

• Check for (and investigate) any 'error/fault' system LEDs
• Check for (and resolve) any reported hardware faults/alerts
• Check for (and resolve) any reported system alerts
• Check for 'nearly' full filesystems
• For large pools, make sure that pool capacity has more than "several" gigabytes free is below about 95% full for best performance. Pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files, then keep pool in the 95-96% utilization range. Even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer.
  Please refer to Document 1392492.1 - Performance issue when pool is almost Full
  

Step 1 - Framing the problem

In order to provide a 'complete picture' of the perceived performance problem, we should collect information about the 'nature' of the issue - including a measure of the performance we are currently seeing, details of the system workload, how the issue is manifesting itself and when the performance is being seen.
Please refer to Document 1213705.1 - Sun Storage 7000 Unified Storage System: Performance issues - Framing the problem

This information will be used in subsequent steps in this troubleshooting document and will also be very useful to engineers should the issue need escalating into Oracle Support.

Possible outcomes :
If the system performance has never been good/acceptable, jump to step 2 : Expected performance for various configurations.
If system performance was good/acceptable and has now fallen, jump to step 4 : known performance issues.

Step 2 - Expected performance for various configurations.

To determine if there MAY be a performance issue on your system, it will be useful to compare the 'measured' performance metric for your current workload/system configuration with an 'expected performance' metric for an 'equivalent' reference workload/configuration.

Please refer to Document 1213714.1 - Sun Storage 7000 Unified Storage System: Performance clues and considerations

Step 3 - Configuration and tuning

The following documents provide details of the system configuration choices that affect performance. It provides guidance on how to configure the system (and possibly, clients) to maximise performance for either 'latency' or 'throughput'.

• NFS   : Please refer to Document 1213725.1 - Sun Storage 7000 Unified Storage System: Configuration and tuning for NFS performance
• CIFS  : Please refer to Document 1213723.1 - Sun Storage 7000 Unified Storage System: Configuration and tuning for CIFS performance
• iSCSI : Please refer to Document 1175573.1 - Sun Storage 7000 Unified Storage System: Configuration and tuning for iSCSI performance
• FC     : Please refer to Document 1928356.1 - Sun Storage 7000 Unified Storage System: Configuration and tuning for FC performance

Step 4 - Known performance issues.

The following documents provides details of any 'known' issues regarding performance.
These will include information on any product limitations (features not yet supported/implemented), references to any 'outstanding' bugs (bugs not yet fixed in the latest appliance firmware release), and known workarounds for some issues.

The following NAS wiki pages can be of interest :
memory issues : https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Possible+memory+issues
fragmentation issues : https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Metaslabs+-+space+maps+and+metaslab_unload_delay+tuning#Fragmentation
dedup usage : https://stbeehive.oracle.com/teamcollab/wiki/Best+Practise+Guide:Performance+issue+-+DDT+table+usage

• NFS     : Please refer to Document 1213739.1 - Sun Storage 7000 Unified Storage System: Known NFS performance issues
• CIFS    : Please refer to Document 1213735.1 - Sun Storage 7000 Unified Storage System: Known CIFS performance issues
• iSCSI   : Please refer to Document 1374509.1 - Sun Storage 7000 Unified Storage System: Known iSCSI performance issues

Step 5 - Collecting analytics data for performance issues.

The 7000 Series NAS appliance range provides a very powerful performance analysis tool called 'Analytics'.

Analytics is an advanced facility to graph a variety of statistics in real-time and record this data for later viewing. It has been designed for both long term monitoring and short term analysis. When needed, it makes use of DTrace to dynamically create custom statistics, which allows different layers of the operating system stack to be analyzed in detail.
The Analytics User Guide is available here : http://docs.oracle.com/cd/E26765_01/html/E26398/index.html
The embedded appliance wiki help pages are also a useful reference to get started : https://<yournasipaddress>:215/wiki/index.php/Analytics
Note that since 2011.1.3, a retention period can be set for analytics, avoiding too much space usage in system pool.

The following NAS wiki page can be of interest :  Assess performance using analytics : https://stbeehive.oracle.com/teamcollab/wiki/Best+Practise+Guide:Best+Practice+-+assess+performance+using+analytics

• NFS   : Please refer to Document 1230143.1 - Sun Storage 7000 Unified Storage System: Collecting analytics data for NFS performance issues
• CIFS  : Please refer to Document 1230145.1 - Sun Storage 7000 Unified Storage System: Collecting analytics data for CIFS performance issues
• iSCSI : Please refer to Document 1229193.1 - Sun Storage 7000 Unified Storage System: Collecting analytics data for iSCSI performance issues
• FC    : Please refer to Document 1375010.1 - Sun Storage 7000 Unified Storage System: Collecting analytics data for Fiber Channel performance issues

Step 6 - other data collection.

In some scenarios, some other data collection have to be done from inside or outside the NAS head.
Some network trace might be needed to narrow down the performance issue. A workflow can be used to collect snoop trace from inside the NAS head. A network trace may have to be collected from the client side as well. Some times, some specific dtrace scripts might be run. In such case, please contact the support services.

• Solaris hosts : Please refer to Document 1285485.1 - GUDS - A Script for Gathering Solaris Performance Data

• dtrace :
  
  • PAE Performance diagnostic dtrace scripts :  https://pae.us.oracle.com/twiki/bin/view/Public/ZFSPerformanceDiagnosis
  • AmberRoadSupport TSC page for clarifications and open comments on PAE dtrace scripts :  https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:PAE+-+ZFS+perf+diag

Back to Document 1416406.1 - ZFS Storage Appliance Product Information Center

References

This solution has no attachment