IB Switch Messages Wrapping with "Possible SYN Flooding On Port 623"

Asset ID:	1-72-2023539.1
Update Date:	2016-03-10
Keywords:

Solution Type Problem Resolution Sure

Solution 2023539.1 : IB Switch Messages Wrapping with "Possible SYN Flooding On Port 623"

Applies to:

Exadata Database Machine X2-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

IB switch messages files are wrapping with the following.

/var/log/messages

Apr 22 22:38:45 ibs0 kernel: possible SYN flooding on port 623. Sending cookies.
Apr 22 22:40:37 ibs0 kernel: possible SYN flooding on port 623. Sending cookies.
Apr 22 22:42:54 ibs0 kernel: possible SYN flooding on port 623. Sending cookies.
Apr 22 22:44:10 ibs0 kernel: possible SYN flooding on port 623. Sending cookies.
Apr 22 22:46:11 ibs0 kernel: possible SYN flooding on port 623. Sending cookies.

/var/log/ipmi.log

Mon Apr 22 20:31:10 2015 [LANIfc: 3186] LANIfc.c::SendLANPkt::772: sendto returned EPIPE
Mon Apr 22 20:31:10 2015 [LANIfc: 3186] LANIfc.c::SendLANPkt::772: sendto returned EPIPE
Mon Apr 22 13:24:35 2015 [LANIfc: 3186] LANIfc.c::SendLANPkt::789: MsgPkt contained invalid socket
Mon Apr 22 13:24:35 2015 [LANIfc: 3186] LANIfc.c::SendLANPkt::789: MsgPkt contained invalid socket
Mon Apr 22 13:24:35 2015 [LANIfc: 3186] LANIfc.c::SendLANPkt::789: MsgPkt contained invalid socket

Cause

This is a defect, triggered by the local listener (i.e. /usr/local/bin/LAN), listening on both TCP port 623 and UDP port 623, having received bogus IPMI packets or ill-formed IPMI requests over UDP, such that it is becoming unresponsive, thus affecting its performance in TCP connection establishment against TCP port 623. This is the direct cause of the "possible SYN flooding on port 623" messages, where one or more TCP connections would stuck in SYN_RECV.

Those bogus IPMI packets or ill-formed IPMI requests over UDP against UDP port 623 are commonly characterized by having one or more NULL bytes in the data portion, so they are either illegitimate IPMI packets or they are legitimate probe packets by 3rd party security scan software.

This issue is documented in Bug# 18298852.

Solution

A permanent fix will be available in IB switch firmware release 2.2.1

In the mean time, the following workaround can be applied, on the affected IB switch, to block those bogus IPMI packets or ill-formed IPMI requests over UDP against UDP port 623.

1)
Reset the iLOM service state back to a clean state.

ibsw# service ilom stop && service ilom start

2)
Setup an IPTables rule.

ibsw# iptables -A INPUT -p udp --dport 623 -m length --length 29:31 -j DROP
ibsw# service iptables save

This is how IPv4 packet sizes of 29 and 31 are derived.

The IPv4 header size is 20 bytes.

The UDP header size is 8 bytes.

The IPv4 packet size is 20 + 8 + <data_size> bytes.

So, the above matching rule "-m length --length 29:31" will match UDP packets having between 1 ~ 3 bytes of data.

References

<NOTE:1594992.1> - How to Generate iLOM Snapshot on infiniband Switches
<NOTE:1448073.1> - Engineered Systems Upgrade/Supportability Reference Doc (INTERNAL ONLY)

Attachments

This solution has no attachment