ILOM Interconnect Fault FMD-8000-D6 occurs after Solaris Upgrade

Asset ID:	1-72-2287852.1
Update Date:	2017-07-20
Keywords:

Solution Type Problem Resolution Sure

Solution 2287852.1 : ILOM Interconnect Fault FMD-8000-D6 occurs after Solaris Upgrade

Applies to:

SPARC T5-2 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

FMD-8000-D6 reoccurs after being cleared.

Jun 16 17:18:07 e9a61487-0b4b-43fa-a77a-dab6944088ca FMD-8000-D6 Major
Problem class : alert.oracle.solaris.fmd.ip-transport.interconnect-down
Description : FMD is unable to use the interconnect from Host to SP. The last error was [failed host MAC to IP addr lookup].

The interface status appears to be normal, but Solaris's portion of the network is misconfigured. An ILOM snapshot will indicate that the interconnect is enabled.

##### ilom/@usr@local@bin@collect_properties.out #####
/SP/network state = enabled OR ipv4-only iff ipv6 is disabled
/SP/network/interconnect state = enabled

The Explorer will indicate that the ILOM interconnect is enabled on Solaris:

##### sysconfig/svcs-av.out lists HMP related services #####
STATE NSTATE STIME CTID FMRI
online - Jun_14 - svc:/network/ilomconfig-interconnect:default
online - Jun_14 142 svc:/system/sp/management:default

The Explorer indicates that usbecm2 is configured on one net (9 in this case):

##### netinfo/dladm/dladm_show-phys_-Z.out #####
LINK ZONE MEDIA STATE SPEED DUPLEX DEVICE
net9 global Ethernet up 10 full usbecm2
net11 global Ethernet up 1000 full vsw3

The Explorer indicates that address 169.254.182.77 is configured on a different net (11 in this case):

##### sysconfig/ifconfig-a.out #####
net9: flags=100001000842"BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING" mtu 1500 index 11
inet 0.0.0.0 netmask ffffff00 broadcast 0.0.0.255 ether 2:21:28:57:47:17
net11: flags=100001000843"UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING" mtu 1500 index 5
inet 169.254.182.77 netmask ffffff00 broadcast 169.254.182.255 ether 0:14:4f:f8:8d:e7

Cause

The Solaris ip-filter was misconfigured which prevented communication on the interconnect. Performing a Solaris upgrade caused the Interconnect's IP address to be assigned to a different net so could not be enabled afterwards.

In this case, /etc/ipf/ipf.conf blocked the interconnect since it was not listed in the list of allowed IPs (169.254.182.76):

# ipf.conf
#
# IP Filter rules to be loaded during startup
#
# See ipf(4) manpage for more information on
# IP Filter rules syntax.

# allow loopback traffic
pass in quick on lo0 all
pass out quick on lo0 all

# allow all outgoing traffic
pass out quick all keep state

# allow all ICMP packets
pass in quick proto icmp all keep state

# ssh:
pass in quick proto tcp from 10.64.22.0/23 to any port = 22
pass in quick proto tcp from 10.80.22.0/23 to any port = 22
pass in quick proto tcp from 10.65.22.22 to any port = 22

# block everything else
block in quick all

Solution

First, the following line must be added to /etc/ipf/ipf.conf since all not listed are blocked:

pass in quick proto tcp from 169.254.182.76 to any port = 24

The next step is to remove the network plumbed to 169.254.182.77 (or change it's IP addr).

Then re-enable usbecm via the "ilomconfig enable" command. The solution portion of doc 2266235.1 provides an example of how the enable is performed & commands to list the status:

# ilomconfig enable interconnect --ipaddress=169.254.182.76 --hostipaddress=169.254.182.77 --netmask=255.255.255.0

This step will allow traffic to pass on this link, but an interconnect fault may exist after it is re-enabled. The fault should be cleared, & a power cycle will prevent future faults.

References

<BUG:26316052> - HOST-ILOM INTERCONNECT COMMUNICATION FAILS
<NOTE:1543136.2> - My Oracle Support How to: How to enable Service Request Email Update Notification with SR content
<NOTE:1588459.1> - Quick User Guide to Upload Files to My Oracle Support - MOS
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices

Attachments

This solution has no attachment