ASR did not create MOS SR when supported ASR ILOM event occurred on ILOM

Asset ID:	1-72-1961258.1
Update Date:	2016-01-19
Keywords:

Solution Type Problem Resolution Sure

Solution 1961258.1 : ASR did not create MOS SR when supported ASR ILOM event occurred on ILOM

Applies to:

Exadata X3-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

ASR did not automatically create SR when fault occurred, due to Customer Network issue.

Changes

Not known if or when changes were implimented

Cause

In this instance little traps were seen, but no big trap. currently ASR utilizes BIG MIB Hardware traps...

iso.org.dod.internet.private.enterprises.sun.products.ilom.sunHwTrapMIB.sunHwTraps.sunHwTrapPrefix.sunHwTrapFaultDiagnosed",".1.3.6.1.4.1.42.2.175.103.2.0.80"

All little traps are currently in X86 set to not create case. When we tested connection ping with -s setting at 2048, this failed. Through investigation it was discovered that a large packet ping to the ASR Manager was not returned. Smaller ping packets below MTU, 1048 size where responded too correctly. Through further examination, it was discovered that the Customers firewall on same network had larger MTU configured (9000), it is best to try to avoid situations where you have jumbo frame (9000 MTU) enabled host NIC's talking to non-jumbo frame enabled host NIC's.

5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 0.494/0.535/0.616/0.053 ms
rpri115@eXXXXXX-XXX:~> []
$ ping -s 1473 10.0.X.X
PING 10.0.X.X(10.0.X.X) 1473(1501) bytes of data.

--- 10.0.X.X ping statistics ---
16 packets transmitted, 0 received, 100% packet loss, time 14999ms
All network devices on a network should have the same MTU values, this was not the case.

If you require to validate ILOM this can be done via Restricted Mode in ILOM where PING is exposed ping -s 2048 <IP Address>

Solution

Then we captured tcpdump which showed the PING requests where not being responded too after passing to firewall. This lead us to the diagnosis, after Customer reporting large MTU value on firewall network port. Then after Customer made changes to there Network, so that a common MTU size was configured, Then re-testing the previous ping tests, a ping large packet now worked, which gave us confidence to test with error injection. This also worked and was the resolution to the issue.

Ping after changes

$ ping -s 2048 XXXXXX-XXXX
PING XXXXXX-XXXX (10.0.X.X) 2048(2076) bytes of data.
2056 bytes from ct-XXX-02 (10.0.X.X): icmp_seq=1 ttl=63 time=1.44 ms
2056 bytes from ct-XXX-02 (10.0.X.X): icmp_seq=2 ttl=63 time=1.08 ms
2056 bytes from ct-XXX-02 (10.0.X.X): icmp_seq=3 ttl=63 time=0.953 ms

References

<NOTE:1450112.1> - Engineered Systems ASR Configuration Check via ASREXACHK
<NOTE:1448069.1> - How to run an ILOM Snapshot on a Sun/Oracle X86 System
<BUG:20126933> - SNMP HARDWARE TRAP NOT SENT OUT FROM ILOM

Attachments

This solution has no attachment