Avahi Causes Network Disconnections and Re-assignment of IP Addresses to Network Interfaces

Asset ID:	1-72-1571485.1
Update Date:	2017-08-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1571485.1 : Avahi Causes Network Disconnections and Re-assignment of IP Addresses to Network Interfaces

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Exadata X3-2 Hardware - Version All Versions and later
Linux OS - Version Oracle Linux 5.1 and later
Information in this document applies to any platform.

Symptoms

Oracle Database servers encounter errors due to network address changes.

These can be see in the database instance alert log, for example:

alert_oradb.log
------------------------------------------------------------
Mon Jun 10 22:31:57 2013
Errors in file /u01/app/oracle/diag/rdbms/oradb/oradb/trace/oradb_ora_28394.trc (incident=128001):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 169.254.186.178 not found. Check output from ifconfig command
Incident details in: /u01/app/oracle/diag/rdbms/oradb/oradb/incident/incdir_128001/oradb_ora_28394_i128001.trc

Process trace files show additional information, for example:

oradb_ora_28379.trc
------------------------------------------------------------
SKGXP:[2b2ecf834ef8.0]{0}: skgxpcini: Invalid IP address [169.254.186.178] provided
struct ksxpp * ksxppg_ [0xb2d4f30, 0x2b2ecfbc67b0) = 0x2b2ecfbc67a8
..
*** 2013-06-10 22:31:57.769
Incident 128002 created, dump file: /u01/app/oracle/diag/rdbms/oradb/oradb/incident/incdir_128002/oradb_ora_28379_i128002.trc
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvad
kgefec: fatal error 0

skgxpcini: Invalid IP address

Cause

In the above example, the IP address of the server had been momentarily changed into 169.254.186.178 by the "avahi-daemon", which performs automated network configuration tasks that override manual network configuration.

The example below illustrates the behavior of avahi-daemon when the IP address of eth0 is being changed from 192.168.140.70 to 192.168.140.110. After all the IP addresses are removed and before new IP address (192.168.140.110) is set, avahi-daemon assigns the ipv4 link-local 169.254.186.178 for a moment, which was picked by oracle process to cause ORA-27303 error.

Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 192.168.140.73 on eth0.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 192.168.140.74 on eth0.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 192.168.140.70 on eth0.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.140.70.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: iface.c: interface_mdns_mcast_join() called but no local address available.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Interface eth0.IPv4 no longer relevant for mDNS.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 192.168.234.3 on eth1.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Leaving mDNS multicast group on interface eth1.IPv4 with address 192.168.234.3.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: iface.c: interface_mdns_mcast_join() called but no local address available.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Interface eth1.IPv4 no longer relevant for mDNS.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 192.168.1.99 on eth2.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Leaving mDNS multicast group on interface eth2.IPv4 with address 192.168.1.99.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Joining mDNS multicast group on interface eth2.IPv4 with address 169.254.186.178.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: IP_ADD_MEMBERSHIP failed: No such device
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Withdrawing address record for 169.254.186.178 on eth2.
Jun 10 22:31:44 oradb1 avahi-daemon[2753]: Interface eth2.IPv4 no longer relevant for mDNS.
Jun 10 22:31:47 oradb1 avahi-daemon[2753]: New relevant interface eth0.IPv4 for mDNS.
Jun 10 22:31:47 oradb1 avahi-daemon[2753]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.140.110.
Jun 10 22:31:47 oradb1 avahi-daemon[2753]: Registering new address record for 192.168.140.110 on eth0.

Solution

There are two solutions, either of which will resolve the problem:

stop avahi-daemon on the server.
configure your mDNS on the avahi-server appropriately (beyond the scope of this document)

To stop the avahi-daemon, for OL5/OL6:

# service avahi-dnsconfd stop # ignore any errors
# service avahi-daemon stop
# chkconfig avahi-dnsconfd off
# chkconfig avahi-daemon off

for OL7:

# systemctl stop avahi-daemon.socket avahi-daemon.service
# systemctl disable avahi-daemon.socket avahi-daemon.service

A similar issue is reported in Doc ID 731609.1. If the same issue occurs after stopping avahi-daemon, refer to this for details of stopping NetworkManager.

This issue happens due to avahi-daemon, therefore this is independent from Oracle DB, Oracle RAC, Exadata components or their versions.

References

<NOTE:731609.1> - Oracle Linux: Manual Changes Made To /etc/hosts Or /etc/sysconfig/network-scripts/ifcfg-* Are Lost
<NOTE:551051.1> - Linux OS Service 'avahi-daemon'

Attachments

This solution has no attachment