"Deferred Adapter Hardware Error Data:..." error on Oracle Emulex FC HBA card with old 1.00A12 firmware on Oracle Linux server

Asset ID:	1-72-1519636.1
Update Date:	2013-01-30
Keywords:

Solution Type Problem Resolution Sure

Solution 1519636.1 : "Deferred Adapter Hardware Error Data:..." error on Oracle Emulex FC HBA card with old 1.00A12 firmware on Oracle Linux server

Applies to:

Sun Fire X4270 M2 Server - Version Not Applicable to Not Applicable [Release N/A]
Emulex FC HBA - Version All Versions and later
Information in this document applies to any platform.

+++++++++++++++++++++++++++++++++++++++++
+++ Before Rebooting or Upgrading fc hba firmware/driver +++
+++++++++++++++++++++++++++++++++++++++++

Jan 3 08:39:17 pr1lnx08 kernel: lpfc 0000:29:00.1: 1:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:2f:00.1: 3:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:2f:00.0: 2:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:29:00.0: 0:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff

According to the Emulex Driver for Linux User Manual (for FC and FCoE version 8.2.0.66 and 8.2.0.63.3p) available at
http://www-dl.emulex.com/support/linux/uldk/1013/linux.pdf
the recommendation in response to this error is to perform a dump using hbacmd.

An sosreport should also be collected from the server.

Following docs can help with data collection:

How to Gather Information on Red Hat Enterprise Linux Systems [Document 1010058.1]

Microsoft Windows and Linux operating systems: How to obtain troubleshooting information [Document 1010936.1]

Collect Operating System Data to Troubleshoot x86 Platforms [Document 1431330.1]

Fibre Channel (FC) Multipathing Connectivity Issues - Basic information and data needed by Oracle [Document 1463600.1]

The adapter should create a firmware dump when it fails, if HBAnyware (One Command Software) is running on the machine. So check for any firmware dump file may have been automatically created.

Once dump is collected open an SR with Oracle FC HBA support group to have dump analyzed for root cause verification.

Symptoms

on Jan 3, 2013 server lost access to storage on all paths

Changes

None

Cause

Both Oracle Emulex FC HBA cards experiencing a disruptive firmware error at same time

Jan 3 08:39:17 pr1lnx08 kernel: lpfc 0000:29:00.1: 1:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:2f:00.1: 3:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:2f:00.0: 2:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff
Jan 3 08:39:18 pr1lnx08 kernel: lpfc 0000:29:00.0: 0:0479 Deferred Adapter Hardware Error Data: xffffffff xffffffff xffffffff

Installed firmware was very old.

[root@pr1lnx08 log]# cat /sys/class/scsi_host/host9/fwrev
1.00A12 (U3D1.00A12), sli-3

[root@pr1lnx08 log]# cat /sys/class/scsi_host/host9/modelname
LPe12002-S

[root@pr1lnx08 log]# modinfo lpfc
filename:       /lib/modules/2.6.18-238.9.1.el5/kernel/drivers/scsi/lpfc/lpfc.ko
version:        0:8.2.0.87.1p
author:         Emulex Corporation - tech.support@emulex.com
description:    Emulex LightPulse Fibre Channel SCSI driver 8.2.0.87.1p

Solution

Customer rebooted server to recover from issue.

Then upgraded fc hba driver/firmware to latest to prevent future issues.

Current FW version and model:

[frli@pr1lnx08 ~]$ cat /sys/class/scsi_host/host9/fwrev
1.11A5 (U3D1.11A5), sli-3

[frli@pr1lnx08 ~]$ date
Mon Jan 14 17:51:18 GMT 2013

Attachments

This solution has no attachment