Sun SPARC Enterprise T5140 Server
 
Emulex FC HBA
 
Solaris Operating System
 
Solaris Operating System
 

PLA-Support>Sun Systems>DISK>HBA>SN-DK: FC HBA
 

In this Document

Symptoms

Changes

Cause

Solution

References

Created from <SR 3-8084961071>

Applies to:

Symptoms

Solaris 10 SPARC server shows one FC HBA LPe11000-S as NOT CONNECTED

The fcinfo command does not return any errors, just that the port c3 state is Offline, and firmware is below

The emlxs patch 149173-03 was installed, rebooted the server and upgraded firmware on that FC HBA.
After that the link come online, but it failed some minutes after that with this errors:

The link is up during booting :

But some minutes later it fails with this error:

Due to that fma reports these errors:

NOTE. Observation of the problem shows that if the server reboots, the link comes up online , but again after some minutes the link fails with the same errors.

Changes

 
On the FC switch port where this HBA is connected there are high values of "Invalid_word" and this is incremented while the link is online.

Cause

1) Failed Oracle Emulex FC HBA or Bug 24320491 - LPe12002-S ERROR: 420:Adapter hardware error.
  

This indicates that an interrupt has occurred and
the status register indicates a nonrecoverable hardware error.
This error usually indicates a hardware problem with the adapter.
Try running adapter diagnostics.
Report these errors to customer service.

From a recent Bug 24320491 - LPe12002-S ERROR: 420:Adapter hardware error.

From the error code in the emlxs driver messages we can state that the
following does indicate a parity error occurred,

(Host Error Attention: status=0x40000000 status1= 0x9ee1a4 status2=0x6000000e)

Specifically, "trap code of "0x6000000e" indicates that a parity error
was hit and detected by Saturn's Memory controller (a chip of the Emulex FC HBA)."

In the case of a parity error, Emulex does recommend that an adapter be reset
and put back into service, due to the possibility this was a one time
occurrence due to environmental factors, and if a parity error reoccurs then
the adapter should be RMA'd if under warranty, due to the possibility the
errors are occurring due to hardware failure. Firmware dumps can be reviewed
to note any hardware issues.

2) It has been found other cases with a similar failure, where the FC HBA has not been replaced and continues to work with no issues:

The fma fault associated:

Clear the error , see: How to repair FMA module errors seen in 'fmadm faulty' (Doc ID 1332409.1)

Then reboot the server and check it has been removed from retired list:
strings /etc/devices/retire_store

Instead of rebooting the server, before that the FC HBA can be reseted to see if this makes it work again:

a) use luxadm -e offline/online

  
OR

b) use One Command Manager tool utility from Emulex.

Select the failed port and "Reset FC port", a window box will appear with this:

After reboot, if the problem persist, replace the FC HBA.

Please refer emlxs_messages.h which has the messages, located here:
https://grok.cz.oracle.com/source/xref/nws10-patch-clone/src/sun_nws/emlxs/hdrs/emlxs_messages.h


204 /* MESSAGE defines */
205 #ifdef DEF_MSG_REPORT
206 emlxs_msg_t emlxs_message[] =
207 {
208 #endif /* DEF_MSG_REPORT */

...


650    DEFINE_MSG(420, \
651        emlxs_hardware_error_msg, \
652        "Adapter hardware error.", \
653        EMLXS_ERROR, \
654        MSG_SLI, \
655        "This indicates that an interrupt has occurred and the " \
656        "status register indicates a nonrecoverable hardware ", \
657        "error. This error usually indicates a hardware problem " \
658        "with the adapter. Try running adapter diagnostics. Report "\
659        "these errors to customer service.", \
660        NULL, \
661        0)



As for the status values these come directly from the HBA registers. I can't find a decode for them in the source code.
I suspect you will need to look at the CHIP/ fw documentation to decode them. Most likely not published by Oracle.
Probably need to ask emulex:


4609            status =
4610                READ_CSR_REG(hba, FC_HS_REG(hba));

4646        status1 =
4647            READ_SLIM_ADDR(hba,
4648            ((volatile uint8_t *)hba->sli.sli3.slim_addr + 0xa8));
4649        status2 =
4650            READ_SLIM_ADDR(hba,
4651            ((volatile uint8_t *)hba->sli.sli3.slim_addr + 0xac));
4652


4653        EMLXS_MSGF(EMLXS_CONTEXT, &emlxs_hardware_error_msg,
4654            "Host Error Attention: "
4655            "status=0x%x status1=0x%x status2=0x%x",
4656            status, status1, status2);
4657

Solution

If you are facing the scenario presented above, recommended action is to collect firmware dump (if possible) to troubleshoot this problem further (see below)
and depending on the error / number of times the problem occurred to replace Oracle Emulex FC HBA.

It has been found other cases with a similar failure, where the FC HBA has not been replaced and continues to work with no issues (see cause section)

For example, in the case of a parity error (a trap code of "0x6000000e" indicates that)
Emulex does recommend that an adapter be reset and put back into service,
due to the possibility this was a one time occurrence due to environmental factors, and
if a parity error reoccurs then the adapter should be replaced
due to the possibility the errors are occurring due to hardware failure.

Some other notes from Emulex:

If a hardware error is noted in the emlxs driver messages but a dump is not
present in the OS's filesystem (on /opt/ELXocm/Dump/ ), possibly due to OCM not being installed the
following workaround should be attempted.

OCM must be installed. Existing fw dumps can be captured by restarting OCM
daemon with the following commands

1. /opt/ELXocm/stop_ocmanager
2. /opt/ELXocm/start_ocmanager
3. dump file will be located in /opt/ELXocm/Dump/

Server reboots will also collect firmware dumps to /opt/ELXocm/Dump/

Firmware dumps initiated through OCM (with hbacmd)
collect the current state of the adapter and erase any dump that existed in the adapter.

There is a Bug closed to address this issue with hbacmd, see

Bug 24450164 - Firmware dump collected by emlxs driver in kernel needs to be usuable.

this bug was opened after the comments made by Emulex on this other bug:

Bug 24320491 - LPe12002-S ERROR: 420:Adapter hardware error.  --> this bug has been closed as not reproducible

References

This solution has no attachment