Sun Storage 7000 Unified Storage System: How to increase the TCP buffer size to improve NFS performance in fast 10Gb Ethernet or IPoIB networks

Asset ID:	1-71-1401076.1
Update Date:	2018-05-23
Keywords:

Solution Type Technical Instruction Sure

Solution 1401076.1 : Sun Storage 7000 Unified Storage System: How to increase the TCP buffer size to improve NFS performance in fast 10Gb Ethernet or IPoIB networks

Applies to:

Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)
NAS head revision : [not dependent]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]

Goal

This document will show that the TCP buffer size can be increased for the Sun Storage 7000 Unified Storage System. This is useful where packet drops and TCP transmission problems are contributing factors in performance degradation on fast networks such as 10Gb Ethernet or IPoIB.

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

If, when using a fast network technology such as 10 Gigabit Ethernet or IPoIB (IP over Infiniband), it is found that NFS performance suffers with dropped packets and TCP transmission problems, then this may be due to the TCP buffer size on the appliance being set at too small a default value to cope with these fast technologies.

Solution

The default value for tcp_xmit_hiwat and tcp_recv_hiwat is 1MB on the appliance. This should really be increased to improve NFS performance on fast networks.

NOTE: Since Appliance Firmware Release 2013.1.3.0, these values are automatically tuned - based on the installed RAM size.

Note, tuning can be achieved on the appliance by invasive support only, please call Oracle Support to raise a Service Request and reference this article.

The following CR was filed for this problem - Bug 15726207 closed as a duplicate: Base Bug 15736749

CR 15726207 Invasive Support: Workflow to change tcp_xmit_hiwat and tcp_recv_hiwat on the Appliance

Bug 15726207 has a workflow attached which will increase the tcp_xmit_hiwat and tcp_recv_hiwat buffers to a specified size.

Please notice that this workflow will change the parameters that will affect all TCP applications globally, not just for NFS.
The change made by the workflow WILL NOT survive over the boot of appliance.

Also note, Bug 17832061 - One dropped packet can permanently throttle TCP transfer

Under recent code, the tunable that was introduced from this fix from this under 2013.1.2.9 has helped under several cases where we are seeing performance issues under 10G / IB.

For tuning details please see -

https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Tuning+tcp_cwnd_normal+on+ZFSSA

PDIT also have a workflow to set this -

https://stbeehive.oracle.com/teamcollab/wiki/PDIT+Storage+Solutions+-+Architecture+and+Performance+Services:Oracle+PDIT+2013.1.4.4+ZFS+SA+Upgrade+Procedures#Increase+the+TCP+Congestion+Window

If you do not have access to this wiki you will need to raise a request to a member of the NAS support team.

Back to Document 1402579.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot NFS Problems.

***Checked for relevance on 23-Nov-2016***
***Checked for relevance on 23-MAY-2018***

References

<NOTE:1402579.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Problems with the NFS Service
<BUG:15726207> - SUNBT7062307 INVASIVE SUPPORT: WORKFLOW TO CHANGE TCP_XMIT_HIWAT AND TCP_RECV_HI
<BUG:15736749> - SUNBT7081230 NFS SOCKET SEND/RECV BUFFER SIZES NEED TO BE RE-EVALUATED

Attachments

This solution has no attachment