WWN Change After Emulex FC HBA Boot Code / Fcode Upgrade

Asset ID:	1-72-1670000.1
Update Date:	2016-07-13
Keywords:

Solution Type Problem Resolution Sure

Solution 1670000.1 : WWN Change After Emulex FC HBA Boot Code / Fcode Upgrade - Cannot See Storage LUNs

Applies to:

SPARC T4-1 - Version All Versions and later
Emulex FC HBA - Version All Versions and later
Information in this document applies to any platform.

Symptoms

This Solaris 10 T4-1 server has two Oracle FC HBAs , two ports connected to the SAN to access two EMC storage arrays:

C# INST# PORT WWN MODEL FCODE STATUS DEVICE PATH
-- ----- -------- ----- ----- ------ -----------
c6 emlxs0 10000090fa6235fe LPe12002-S 3.10a3 NOT CONNECTED /pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0
c7 emlxs1 10000090fa6235ff LPe12002-S 3.10a3 CONNECTED /pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0,1
c11 emlxs2 10000090fa62339a LPe12002-S 3.10a3 NOT CONNECTED /pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0
c12 emlxs3 10000090fa62339b LPe12002-S 3.10a3 CONNECTED /pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1

We have just installed Solaris 10 Update 11 on a new T4-1 server, and presented storage from an EMC VNX array.
We could see the storage ok until we added patch 143247-02 to upgrade the Fcode on LPe12002s HBAs (from 3.01a1 to 3.10a3)

We can see new Boot Code and Fcode after reboot on "Apr 23 16:51:09" :

Apr 23 16:43:25 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs1: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a0 Fcode:3.01a1)
Apr 23 16:43:35 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs2: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a0 Fcode:3.01a1)
Apr 23 16:43:53 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs0: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a0 Fcode:3.01a1)
Apr 23 16:43:54 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs3: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a0 Fcode:3.01a1)

Apr 23 16:51:09 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs1: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a8 Fcode:3.10a3)
Apr 23 16:51:19 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs2: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a8 Fcode:3.10a3)
Apr 23 16:51:36 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs0: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a8 Fcode:3.10a3)
Apr 23 16:51:37 server1 emlxs: [ID 349649 kern.info] [ B.1A94]emlxs3: NOTICE: 100: Driver attach. (Firmware:2.01a4 (U3D2.01A4) Boot:5.03a8 Fcode:3.10a3)

There is no difference on the WW Port Name (WWPN) after upgrade
but I can see the the WW Node Name (WWNN) has changed after Boot Code / Fcode upgrade,

Apr 23 16:43:25 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs1: NOTICE: 100: Driver attach. (WWPN:10000090FA6235FF WWNN:20000090FA6235FF)
Apr 23 16:43:53 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs0: NOTICE: 100: Driver attach. (WWPN:10000090FA6235FE WWNN:20000090FA6235FE)
Apr 23 16:43:35 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs2: NOTICE: 100: Driver attach. (WWPN:10000090FA62339A WWNN:20000090FA62339A)
Apr 23 16:43:54 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs3: NOTICE: 100: Driver attach. (WWPN:10000090FA62339B WWNN:20000090FA62339B)

Apr 23 16:51:09 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs1: NOTICE: 100: Driver attach. (WWPN:10000090FA6235FF WWNN:20000120FA6235FF)
Apr 23 16:51:36 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs0: NOTICE: 100: Driver attach. (WWPN:10000090FA6235FE WWNN:20000120FA6235FE)
Apr 23 16:51:19 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs2: NOTICE: 100: Driver attach. (WWPN:10000090FA62339A WWNN:20000120FA62339A)
Apr 23 16:51:37 server1 emlxs: [ID 349649 kern.info] [ B.1ACC]emlxs3: NOTICE: 100: Driver attach. (WWPN:10000090FA62339B WWNN:20000120FA62339B)

Changes

Used patch 143247-02 to upgrade the Fcode on LPe12002s HBAs (from 3.01a1 to 3.10a3)

Cause

The WWNN of the FC HBAs changed after boot code / Fcode upgrade with patch 143247-02.
As the customer was using WWNN on the FC switch and storage configuration, after the WWNN change they lost access to the storage.

Tthis is not expected behavior, there is an old Bug 15678670 - SUNBT6996631 LP11002-S WWN changed after Boot Code upgrade with a similar issue, closed due to lack of information.

Solution

Configure on the Storage and FC switches the WWPN , instead of the WWNN. There is no need to use WWNN, use the WWPN instead.

After that, access to the EMC storage LUNs was recovered.

1) About WW Node Name (WWNN) change after boot code / Fcode upgrade with patch 143247-02 , this is not expected behavior,

there is an old Bug 15678670 - SUNBT6996631 LP11002-S WWN changed after Boot Code upgrade with a similar issue, closed due to lack of information.

Once possible explanation is that Emulex HBAs default to their original WWN when boot code is upgraded (a known design decision).
Q. Did you manually change the WWN on these HBAs previously?

This is an example of how WWNN or WWPN can be changed , using the Emulex OneCommand Manager Core Version (CLI):

ChangeWWN
WWPN: World Wide Port Name of Object HBA
New WWPN: new World Wide Port Name of Object HBA
New WWNN: new World Wide Node Name of Object HBA
TYPE: {0, 1}
0: Volatile
1: Non-Volatile

In your case, if you want to change to the previous WWNN, you should run:

# /opt/ELXocm/hbacmd ChangeWWN 10:00:00:90:fa:62:35:fe 10:00:00:90:fa:62:35:fe 20:00:00:90:fa:62:35:fe 1
# /opt/ELXocm/hbacmd ChangeWWN 10:00:00:90:fa:62:35:ff 10:00:00:90:fa:62:35:ff 20:00:00:90:fa:62:35:ff 1
# /opt/ELXocm/hbacmd ChangeWWN 10:00:00:90:fa:62:33:9a 10:00:00:90:fa:62:33:9a 20:00:00:90:fa:62:33:9a 1
# /opt/ELXocm/hbacmd ChangeWWN 10:00:00:90:fa:62:33:9b 10:00:00:90:fa:62:33:9b 20:00:00:90:fa:62:33:9b 1

2) To troubleshoot this further and try to understand why the WWNN changed

collect a firmware dump, along with the Serial number of the adapter in question.

This document explains how to collect that:
How To Get a Firmware Dump From an Emulex HBA (Doc ID 1629921.1)
Please, be aware that getting a firmware dump will stop the IO flowing through the card while the data is being collected.

If customer provides this information, open a new Bug based on BUG 15678670, se we can escalate this to Emulex

Emulex will require to collect a firmware dump in case problem reappear:

"If this issue reappears again at any site: It is important to use One Command Manager, or the emlxadm tool ( from core utility ,kit) to gather a firmware dump.
Provide that to Emulex, along with the Serial number of the adapter in question.
With a current firmware dump, and the serial number we can investigate whether the adapter has had a Non-Volatile WWN change occur to a region of the firmware. "

3) If possible, see if we can replicate the behavior using the latest Boot Code and utilities from their web page.

As explained on latest doc "1389639.1 - FAQ Oracle FC HBA: FCode/BIOS(ie. Boot Code), Firmware, and Drivers"
Oracle is no longer providing fcode/bios patches (notice patch 143247-02 is from Feb 2010)
Any fcode/bios version beyond those listed above will need to be downloaded from the respective FC HBA OEM site.

On this case, these FC HBAs LPe12002-S are model SG-XPCIE2FC-EM8-Z :
http://www.avagotech.com/support/oem/oracle-fc/fibre-channel-8gb/sg-xpcie2fc-em8-z
"Contains: x86 BootBIOS version 2.11a4, OpenBoot (Fcode) 3.10a3 and EFIBoot 4.12a3"

References

<BUG:15678670> - SUNBT6996631 LP11002-S WWN CHANGED AFTER BOOT CODE UPGRADE
<NOTE:1351617.1> - FAQ on Emulex HBA Boot Code/Fcode Firmware and driver patching
<NOTE:1389639.1> - FAQ Oracle FC HBA: FCode/BIOS(ie. Boot Code), Firmware, and Drivers
<NOTE:1629921.1> - How To Get a Firmware Dump From an Emulex FC HBA

Attachments

This solution has no attachment