Oracle ZFS Storage Appliance: NFSv3 Unable to remove client locks with 'Clear locks' workflow

Asset ID:	1-72-1909839.1
Update Date:	2018-05-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1909839.1 : Oracle ZFS Storage Appliance: NFSv3 Unable to remove client locks with 'Clear locks' workflow

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

After NFSv3 client reboot applications detect remaining locks on files on NFSv3 Server.

Upon restart of client application such as Oracle database can report :

new-host $
alter database mount
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/DATA/UAT/oradata/cntrl02.dbf'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8

Locks held by a no longer existing NFS client still present in kernel:

exx-zfs-head1:> confirm shell echo ::nlm_lockson | mdb -k
host lock-addr sysid(x) vnode pid cmd state type
.
old-host.example.com fffff60a89f28d80 10(L) fffff619b671f000 18 <remote> 3 WR
old-host.example.com fffff602f1ff6640 10(L) fffff619a20cc000 18 <remote> 3 WR
old-host.example.com fffff604f7d06e80 10(L) fffff6197b6c0040 18 <remote> 3 WR
old-host.example.com fffff61a460e5e40 10(L) fffff603bbfcf040 18 <remote> 3 WR
old-host.example.com fffff604d008d040 10(L) fffff619acf86080 18 <remote> 3 WR
old-host.example.com fffff60aacac8240 10(L) fffff6197a9de080 18 <remote> 3 WR
old-host.example.com fffff619ed003d40 10(L) fffff6198836c080 18 <remote> 3 WR

Changes

NFS Client's reconfigured in recent past - in particular the hostname has been changed.

Cause

If NFS client host shutdown or lost contact with the NFS server and is then renamed while locks remained on files in NFS server, then upon client with new name restarting, the old locks will not be recognised.

This will prevent same client but with new name from acquiring the required locks.

The NFS server will remember the locks held by the old name of the NFS client for as long as the NFS server is running. The 'clearlocks' workflow will not be able to remove this.

Solution

Stop and restart the NFS service on the appliance.

The lockd and statd process will re-poll the NFS clients to determine what locks were held.

If an NFS client is not contactable upon NFS server restart, then its locks will be cleared.

References

<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices
<NOTE:1018699.1> - How to clear NFS application (hang) lock state.
<NOTE:1402579.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Problems with the NFS Service

Attachments

This solution has no attachment