Sun Storage 7000 Unified Storage System: SMB operations per second is very high

Asset ID:	1-72-1401557.1
Update Date:	2015-09-11
Keywords:

Solution Type Problem Resolution Sure

Solution 1401557.1 : Sun Storage 7000 Unified Storage System: SMB operations per second is very high

Applies to:

Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)
NAS head revision : [before 2010.Q3.3.1]
BIOS revision : [not dependent]
ILOM revision : [not dependent]
JBODs Model : [not dependent]
CLUSTER related : [not dependent]

Symptoms

A very high amount of SMB operations per sec which are caused be a storm of smb_nt_transact_notify_change messages.

See 15663559 SUNBT6978261 Notify Change buffering mechanism can cause a torrent of notify change requests/responses

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Cause

Background info on the smb_nt_transact_notify_change messages

The smb_nt_transact_notify_change messages are generated to notify the client that something has changed in a directory. For example the windows explorer application will use it to display and update automatically the content of an smb share.
The Windows client will send a request and the storage will notify only when something in the directory has changed (maybe 5 min later).
The bug is that the storage will respond DIRECTLY to the client (something changed) even if NOTHING changed. Windows will send then an other request (each time Windows receives a notify change response, it needs to send an other request for the next change in the directory) ... and the storage will respond directly ... causing a torrent or storm until you close the Windows explorer.

Verification of the high amount of smb_nt_transact_notify_change messages

dknco-stor2> status activity show
Activity:
CPU 23 %util Partly Cloudy
FTP 0 bytes/sec Sunny
Disk 2009 ops/sec Cloudy
iSCSI 31 ops/sec Partly Cloudy
NFSv3 90 ops/sec Sunny
NFSv4 0 ops/sec Sunny
Network 12.4M bytes/sec Partly Cloudy
SMB 27697 ops/sec Rainy

This shows 27697 SMB ops/sec for only 2009 Disk ops/sec.

The verification that the system is hitting this bug should be done by Oracle TSC by creating a service request and referencing this document.

The below dtrace script counts the number of notify_change requests done in 10sec. Having > 300000 (so 30000/s) is crazy. How can something change so quickly in a directory !
Normal value is not easy to find but is more likely that notify_change should no be more than other regular SMB requests. If nothing has changed, the storage should not respond to a notify change.

dknco-stor2# dtrace -n 'smb_nt_transact_notify_change:entry{ @[stack(60)] = count() }' -n 'tick-10sec{ exit(0) }'
dtrace: description 'smb_nt_transact_notify_change:entry' matched 1 probe
dtrace: description 'tick-10sec' matched 1 probe
CPU ID FUNCTION:NAME
1 78654 :tick-10sec

smbsrv`smb_nt_trans_dispatch+0x13a
smbsrv`smb_com_nt_transact+0x172
smbsrv`smb_dispatch_request+0x472
smbsrv`smb_session_worker+0x4a
genunix`taskq_d_thread+0xb1
unix`thread_start+0x8
327531

There are 327531 messages per 10 sec generated, 32753 per/sec
This verifies we are hitting bug 15663559 SUNBT6978261 Notify Change buffering mechanism can cause a torrent of notify change requests/responses

Solution

Upgrade to at least AK-version 2010.08.17.3.1 (2010Q3.3.1).

See 15663559 SUNBT6978261 for further details.

See the ak-2010.08.17.3.1 Release Notes at : MOS Document ID 2021771.1 - Oracle ZFS Storage Appliance: Software Updates

Back to <Document 1408716.1> Sun Storage 7000 Unified Storage System: How to Troubleshoot SMB (CIFS) Problems.

References

<NOTE:1408716.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot SMB (CIFS) Problems

Attachments

This solution has no attachment