Oracle ZFS Storage Appliance: Can't look up SM_NOTIFY sender: n2a: unknown error

Asset ID:	1-72-2278444.1
Update Date:	2017-08-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2278444.1 : Oracle ZFS Storage Appliance: Can't look up SM_NOTIFY sender: n2a: unknown error

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Message: "Can't look up SM_NOTIFY sender: n2a: unknown error #-5" being reported in debug log.

Cause

The message are shown in the supportbundle debug.sys log file :file.

May 24 00:04:01 h1-storadm statd[21363]: [ID 908464 daemon.error] Can't look up SM_NOTIFY sender: n2a: unknown error #-5
May 24 00:04:01 h1-storadm statd[21363]: [ID 652648 daemon.notice] Received SM_NOTIFY (amdsc5client02) from UnknownHost (192.157.28.27)

The message show the "n2a" - name to address lookup is failing from an "UnknownHost"

SM_NOTIFY is used by the NFS service to send reboot notifications to NFS peers. It is a helper program that notifies NFS peers after the local system reboots.
Lock recovery after a reboot is critical to maintaining data integrity and preventing unnecessary application hangs.

When the local system reboots, the sm-notify command reads the list of monitored peers from persistent storage and sends an SM_NOTIFY request to the NSM service on each listed remote peer.
It uses the mon_name string as the destination. To identify which host has rebooted, the sm-notify command normally sends my_name string recorded when that remote was monitored.
The remote rpc.statd matches incoming SM_NOTIFY requests using this string, or the caller's network address, to one or more peers on its own monitor list.

The message implies that the "unknown host" is on this IP address - 192.157.28.27

The netstat-pn.out output shows it connected to this Infiniband IP address:

ipmp1 192.157.28.27 255.255.255.255 80:00:00:6d:fe:80:00:00:00:00:00:00:00:21:28:00:01:ef:b7:a0

Solution

To help rpc.statd match SM_NOTIFY requests to NLM requests, a number of best practices should be observed, including:

a/ The nodename of your systems should match the DNS names that NFS peers use to contact them
b/ The nodenames of your systems should always be fully qualified domain names (FQDN)
c/ The forward and reverse DNS mapping of the nodenames should be consistent
d/ The hostname the client uses to mount the server should match the server's mon_name in SM_NOTIFY requests it sends

I have been able to confirm that there is an open bug on the sm_notify messages:

Bug id: 15758296 - SUNBT7116904 statd error message causing false alarms, should be removed

The bug mentions that these alerts emanate from the "private internal network addresses" for the Fibre Channel link, and so do not have access to allow hostname resolution.

The workaround is to add the infiniband IP addresses to your DNS server to clear up the messages.

References

<NOTE:2231880.1> - Can't Look Up SM_NOTIFY Sender
<BUG:15758296> - SUNBT7116904 STATD ERROR MESSAGE CAUSING FALSE ALARMS, SHOULD BE REMOVED
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices

Attachments

This solution has no attachment