Oracle Exadata Hardware
 
SPARC SuperCluster T4-4 Half Rack
 
Solaris Operating System
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 

Applies to:

Symptoms

The following entry is in the /var/log/messages file - what does this mean? 

or

Cause

 The message "hermon0: CQE transport retry counter exceeded" is related to the Infiniband HCA driver and simply means that the IB connection has gone down.

Solution

When an Infiniband stack client (ULP a.k.a. upper layer protocol) creates a queue pair it will specify a retry timeout and a retry count to the hardware which will dictate what the HCA hardware/firmware does once a message is sent via that QP onto the fabric.

The local HCA having sent the message once on the fabric receives no reply from the peer hardware in <retry timeout> time the local HCA hardware will retry the message.

This will happen up to <retry count> times at which time the HCA hardware/firmware will issue an error completion to the upper layer protocol (client) which owns the QP, one for each message in flight and unacked by the peer HCA HW.

Each error completion is also logged by the mcxnex driver (the message pair you see, to the system messages file. Each pair of message in the msgs file represents one in flight unacked message which has not been responded by the peer HW within (retry timeout * retry count).

Usually this is because the peer rebooted causing the HCA HW to be reset, alternatively it might be the result of a very bad cable or transceiver on the peer node (not this node).

In other words, if you are in the midst of any maintenance operations where the connection may be restarted due to a remote machine reboot, then this is expected. 

This message is purely informational as long as this doesn't occur frequently and unexpectedly.

This solution has no attachment