Sun Storage 7000 Unified Storage System: System Is Not Responding (nfsd not responding)

Asset ID:	1-72-1545463.1
Update Date:	2018-01-08
Keywords:

Solution Type Problem Resolution Sure

Solution 1545463.1 : Sun Storage 7000 Unified Storage System: System Is Not Responding (nfsd not responding)

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Information in this document applies to any platform.

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

System was not responding well on 'status show'.

- nfsd not responding

- users cannot process backups

When checking the individual components of the 'status show' output by issueing 'status memory show', 'status activity show', and so on, it appeared that 'status storage show' was the one that responded poorly.

Customer even had no BUI access and had 'emergency shell' logins

The system is widely used for 700 clients backing up their data, but looking with 'iostat' we only see that it is reading data where we would expect lots of writes:

iostat -xnz 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
22.0 0.0 385.8 0.0 0.0 0.0 0.0 1.2 0 2 c2t5000CCA03E2C7AD0d0
42.0 0.0 815.7 0.0 0.0 0.1 0.0 2.8 0 9 c2t5000CCA03E2BE630d0
18.0 0.0 242.7 0.0 0.0 0.1 0.0 5.3 0 8 c2t5000CCA03E2C759Cd0
59.1 0.0 709.1 0.0 0.0 0.1 0.0 1.3 0 6 c2t5000CCA03E27C0A8d0
58.0 0.0 732.1 0.0 0.0 0.1 0.0 1.8 0 6 c2t5000CCA03E2D0910d0
57.0 0.0 732.6 0.0 0.0 0.1 0.0 1.0 0 3 c2t5000CCA03E2C5A84d0
40.0 0.0 855.7 0.0 0.0 0.1 0.0 2.5 0 5 c2t5000CCA03E27BEF4d0
58.0 0.0 721.6 0.0 0.0 0.1 0.0 1.5 0 6 c2t5000CCA03E2BE654d0
57.0 0.0 732.6 0.0 0.0 0.1 0.0 1.5 0 4 c2t5000CCA03E2D090Cd0
77.1 0.0 957.3 0.0 0.0 0.1 0.0 1.8 0 7 c2t5000CCA03E21FC88d0
16.0 0.0 258.7 0.0 0.0 0.1 0.0 5.4 0 5 c2t5000CCA03E27BD50d0
61.1 0.0 735.1 0.0 0.0 0.1 0.0 2.2 0 9 c2t5000CCA03E2D1084d0
59.1 0.0 708.6 0.0 0.0 0.1 0.0 2.1 0 8 c2t5000CCA03E27BCFCd0
56.0 0.0 719.6 0.0 0.0 0.1 0.0 0.9 0 3 c2t5000CCA03E2BE594d0
78.1 0.0 959.3 0.0 0.0 0.1 0.0 1.7 0 8 c2t5000CCA03E2D0900d0
75.1 0.0 957.8 0.0 0.0 0.1 0.0 0.8 0 4 c2t5000CCA03E2C5A74d0
78.1 0.0 959.8 0.0 0.0 0.2 0.0 2.9 0 13 c2t5000CCA03E2BE5D4d0
23.0 0.0 387.8 0.0 0.0 0.0 0.0 1.2 0 2 c2t5000CCA03E27BD78d0

Cause

NFSd has reached the threshold for maximum number of threads configured

# prstat -scpu

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
29108 daemon 11M 10M sleep 60 -20 19:41:54 1.4% nfsd/502

7420-01:configuration services nfs> show
Properties:
   = online
  version_min = 2
  version_max = 3
  nfsd_servers = 500
  grace_period = 90
  mapid_domain =
  enable_delegation = true
  krb5_realm =
  krb5_kdc =
  krb5_kdc2 =
  krb5_admin =

NFSd was configured for 500 Servers.

More then 500 Servers actively using NFSd at the same time.

Solution

Increased parameter nfsd_servers to 1000

7420-01:configuration services nfs> set nfsd_servers=1000
nfsd_servers = 1000 (uncommitted)
7420-01:configuration services nfs> commit

System is responding quickly now and clients are writing.

iostat -xnz 1

extended device statistics
  r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
  71.1 390.0 1249.1 18099.1 0.0 3.2 0.0 6.9 0 76 c2t5000CCA03E2C7AD0d0
  74.1 314.8 1444.1 18496.2 0.0 3.0 0.0 7.8 0 72 c2t5000CCA03E2BE630d0
  74.1 396.1 1250.1 18094.5 0.0 3.3 0.0 6.9 0 88 c2t5000CCA03E2C759Cd0
  19.3 333.1 445.3 17474.0 0.0 3.6 0.0 10.3 0 69 c2t5000CCA03E27C0A8d0
  21.3 371.7 471.2 17489.7 0.0 3.1 0.0 7.8 0 87 c2t5000CCA03E2D0910d0
  28.4 479.3 458.0 17487.1 0.0 3.1 0.0 6.1 0 75 c2t5000CCA03E2C5A84d0
  86.3 363.6 1590.8 18496.6 0.0 1.9 0.0 4.1 0 58 c2t5000CCA03E27BEF4d0
  18.3 403.2 557.5 17465.8 0.0 3.1 0.0 7.3 0 63 c2t5000CCA03E2BE654d0
  24.4 428.6 435.7 17514.7 0.0 3.5 0.0 7.7 0 88 c2t5000CCA03E2D090Cd0
  35.5 496.6 482.4 17491.9 0.0 2.3 0.0 4.3 0 62 c2t5000CCA03E21FC88d0
  76.2 420.4 1343.6 18091.6 0.0 3.1 0.0 6.3 0 79 c2t5000CCA03E27BD50d0

Attachments

This solution has no attachment