Oracle ZFS Storage: Performance of 'ls' command on NFSv3 Share From Linux client is 'slow' when invoked the first time

Asset ID:	1-72-1646387.1
Update Date:	2018-05-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1646387.1 : Oracle ZFS Storage: Performance of 'ls' command on NFSv3 Share From Linux client is 'slow' when invoked the first time

Applies to:

Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-BA - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

When you access to the share via NFSv3 from Linux environment, 'ls' command is very slow than the second 'ls'.

Cause

This is because Linux client changes its behavior for the first 'ls' and the second 'ls', even cache is disabled as below.

The first 'ls' sends NFSv3 READDIRPLUS commands and the second 'ls' does not. The flood of READDIRPLUS cause Linux client with slower response.

192.168.0.1:/export/test/backup /E0/backup nfs rsize=1048576,wsize=1048576,async,hard,intr,rw,bg,nfsvers=3,tcp,noac,actimeo=0 0 0

In order to identify the problem, use nfsstat command to see the growth of the READDIRPLUS command count or use packet capture to count up the requests during the time of executing the 'ls' command.

The below are samples:

### ls for the first time from Linux without nordirplus option set

60 NFS V3 ACCESS Call,
60 NFS V3 ACCESS Reply
60 NFS V3 FSSTAT Call,
60 NFS V3 FSSTAT Reply
354 NFS V3 GETATTR Call,
360 NFS V3 GETATTR Reply
60 NFS V3 READDIRPLUS Call,
60 NFS V3 READDIRPLUS Reply

You can use

tshark -tad -r <packet capturef file> > output file

Then use command like below to count up.

-bash-3.2$ cat test1.txt | awk '{print $6,$8,$9,$10}' | sort | grep 'NFS V3' | uniq -c
60 NFS V3 ACCESS Call,
60 NFS V3 ACCESS Reply
60 NFS V3 FSSTAT Call,
60 NFS V3 FSSTAT Reply
354 NFS V3 GETATTR Call,
360 NFS V3 GETATTR Reply
60 NFS V3 READDIRPLUS Call,
60 NFS V3 READDIRPLUS Reply

### ls for the second time for the same share without nordirplus
60 NFS V3 ACCESS Call,
60 NFS V3 ACCESS Reply
360 NFS V3 GETATTR Call,
360 NFS V3 GETATTR Reply

Solution

Add nordirplus option to your mount option such as below.

# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32868,tcp,actimeo=0,vers=3,timeo=600,nordirplus 192.168.100.1:/export/nfs /mnt

You need to unmount the file system to make the parameter effective. Other mount options should respect the I/O characteristics of your environment and the above parameters are shown as a sample only.

RedHat has the following document. https://access.redhat.com/site/solutions/209713

Oracle Linux bug below describes the symptom.

Bug 17291332 : LINUX INCORRECTLY USING READDIRPLUS ON HUGE DIRECTORY

*** Checked for currency - 01-JUN-2016 ***
*** Checked for currency - 25-MAY-2018 ***

References

<NOTE:1515808.1> - Sun Storage 7000 Unified Storage System: BUIまたはCLIを使用してサポート・バンドルを収集する方法
<NOTE:1567137.1> - Sun ZFS Storage Appliance: Oracle Database 11g R2 NFS Mount Point Recommendations
<NOTE:1514722.1> - Sun Storage 7000 Unified Storage System: NFSのパフォーマンスの問題に関する分析データの収集
<BUG:17291332> - LINUX INCORRECTLY USING READDIRPLUS ON HUGE DIRECTORY
<NOTE:2087231.1> - Guidelines When Using ZFS Storage in an Exadata Environment
<NOTE:1515740.1> - Sun Storage 7000 Unified Storage System: パフォーマンスの問題をトラブルシューティングする方法
<NOTE:359515.1> - Mount Options for Oracle files for RAC databases and Clusterware when used with NFS on NAS devices

Attachments

This solution has no attachment