Root Domain Hangs After Executing 'luxadm forcelip' to Upgrade Emulex FC HBA Firmware

Asset ID:	1-72-2193279.1
Update Date:	2018-03-07
Keywords:

Solution Type Problem Resolution Sure

Solution 2193279.1 : Root Domain Hangs After Executing 'luxadm forcelip' to Upgrade Emulex FC HBA Firmware

Applies to:

SPARC T3-2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

On a Solaris 11.3 SRU 11.3.5.6.0 LDoms guest I/O service Root Domain we attempted to complete the HBA firmware upgrade procedure as described in Doc ID 1356876.1

Mar 10 17:42:07 serve01 emlxs: [ID 349649 kern.info] [ 1.038C]emlxs0:WARNING:1540: Firmware update required. (To trigger an update, a manual HBA or link reset using luxadm, fcadm, or emlxadm is required.)

After running the forcelip command the Solaris I/O Root Domain hangs, this is not expected.

Some data about the FC HBA being upgraded:

c2 /devices/pci@500/pci@2/pci@0/pci@a/SUNW,emlxs@0/fp@0,0 Sun SG-XPCIE1FC-EM8-Z (LPe12000-S) 371-4295 8Gb

HBA Port WWN: 10000000c99XXXXX
Port Mode: Initiator
Port ID: 1b0120
OS Device Name: /dev/cfg/c2
Manufacturer: Emulex
Model: LPe12000-S
Firmware Version: LPe12000-S 1.11a5 <<<---very old firmware
FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
Serial Number: 0999BT0-10010001GA
Driver Name: emlxs
Driver Version: 3.0.05.0 (2015.09.16.13.00)
Type: N-port
State: online
Supported Speeds: 2Gb 4Gb 8Gb
Current Speed: 8Gb
Node WWN: 20000000c99XXXXX
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 10
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 32
Invalid CRC Count: 0
NPIV Not Supported

Cause

Root cause could not be found as

this behavior was not reproducible on the Oracle Lab

and

no valid Solaris core file was collected from the I/O Root Domain, as explained on

How to Collect a Forced Crash Dump of a Hanging Solaris Guest LDom (Doc ID 1020884.1)

Solution

If luxadm fails (very unlikely), try Emulex utilities to upgrade firmware:

Emulex One Command Manager (OCM) --> hbacmd
or
previous HBAnyware utility --> emlxadm

Solaris 11.2 (or greater) already comes with hbacmd utility, available on:
/usr/lib/ssm/fwupdate/emulex/hbacmd
so no need to download OCM from Emulex/Avago web site

Command syntax would be like this:
hbacmd Download <WWPN|MAC> <FileName>

See also : Solaris 11.2 FC HBA - Update Firmware, FCode/BIOS(ie. Boot Code) (Doc ID 1955822.1)

On this particular case, customer used "emlxadm download_fw" as a satisfactory workaround for upgrading the firmware.

References

<NOTE:1927613.1> - FC HBA Emlxs Firmware Update - ERROR:1522: Firmware Download Failed. (Unable to Erase Flash. Mailbox cmd=1c status=fe)
<NOTE:1602837.1> - FC HBA Emlxs ERROR: 420: Adapter Hardware Error. (Host Error Attention: Status=0x40000000
<NOTE:1955822.1> - Solaris 11.2 FC HBA - Update Firmware, FCode/BIOS(ie. Boot Code)
<NOTE:1356876.1> - Firmware Update Required. (A Manual Hba Reset Or Link Reset (Using Luxadm Or Fcadm) Is Required
<NOTE:1020884.1> - How to Collect a Forced Crash Dump of a Hanging Solaris Guest LDom

Attachments

This solution has no attachment