Sun SPARC Enterprise M5000 Server
 
Emulex FC HBA
 
Solaris Operating System
 
Solaris Operating System
 

PLA-Support>Sun Systems>DISK>HBA>SN-DK: FC HBA
 

In this Document

Symptoms

Cause

Solution

References

Created from <SR 3-9265034731>

Applies to:

Symptoms

Solaris 10 M5000 server "server01" with two Oracle FC HBAs LPe12002-S ,
two ports connected to the SAN to access EMC disk storage arrays:

Customer has installed emlxs patch 149173-04 on 24 June:
patch_date.out:drwxr-xr-x 2 root root 6 Jun 24 05:38 149173-04


On server reboot at "Jun 30 13:35:43" we see firmware update was required:

Customer initiated firmware update with "luxadm -e forcelip" as explained on doc:
Warning:1540: Firmware Update Required. (A Manual Hba Reset Or Link Reset (Using Luxadm Or Fcadm) Is Required (Doc ID 1356876.1)

Firmware update worked fine on first FC HBA (emlxs0 and emlxs1), these are the expected messages when firmware update is completed:

***BUT firmware update had a problem with FC HBA2,
here are the messages observed and the final error when updating emlxs3
(the instance associated to the FC HBA port not connected to any storage device)

Due to that an fma fault was created:

The same problem was found with emxls2 :

Cause

The cause was investigated on this BUG , worked by Emulex

Bug 19149937 - FC HBA becomes unusable due to emlxs Mailbox Timeout during firmware update

RCA could not be identified, a Driver and Firmware dump from the Emulex FC HBA was required

In this case the DOWNLOAD command is failing.
Description of Download command given in the SLI3 is below:
"It is timing out. Generally timeout happens when FW or hardware is not responding."

There was also a known issue of Driver Version: 2.90.15.0 (2014.01.22.14.50)  fails to update older firmware (Firmware Version: LPe12002-S 2.00a4)

"luxadm -e forcelip" command triggers the firmware update on emlxs driver.

Solution

Use the latest drivers and firmware, you should update OneCommand Manager CLI to the latest version
and then update the adapter to the latest firmware using that CLI.

In case you hit this issue, use the Emulex-supplied utilities, on the Broadcom site https://www.broadcom.com/support/oem/oracle-fc/ there are utilities for Solaris, Linux and Windows.

Notice. The drivers, and firmware, and utilities are released by Emulex as a kit. Best practice is to use the releases from the same kit.

Workaround to complete the firmware upgrade

1) Clear / repair the fma fault with command 

See also

How To Clear FMA faults from Solaris[TM] and SC (System Controller) on T1000/T2000 T5120/T5220/T5140/T5240/T5440, T3-1/T3-2/T3-4, T4-1/T4-2/T4-4 (Doc ID 1004229.1)

In a situation where "fmadm repaired [fmri|label]" or "fmadm acquit [fmri|label]" does not clear the device from the
persistent store (ie /etc/devices/retire_store file), the workaround is to delete the store and reboot solaris.

- After that try again the firmware upgrade with "luxadm -e forcelip"

# luxadm -e forcelip /devices/pci@12,600000/SUNW,emlxs@0,1/fp@0,0:devctl

2) If "luxadm -e forcelip " command does not trigger any new firmware update , then run

These commands should cause the adapter back to a good state that allows another update to be started.

About: /opt/EMLXemlxu/bin/emlxadm get_fw_rev
This command reads fw_version from the internally stored driver variable.

About: /opt/EMLXemlxu/bin/emlxadm diag emlxs_post
This command takes the adapter offline, resets adapter and takes the adapter again online.
emlxs_post reinitializes the adapter. Here all the Software data structure
is freed, adapter is reset and again reinitialized. In this case the adapter
must be coming out of the error, if any as we are reinitializing it.

- If you don't have older EMLXemlxu packaged installed, you can install the new Emulex One Command Manager ( OCM ).

 

This software can be downloaded here, for Solaris 10 u11:

https://www.broadcom.com/support/oem/oracle-fc/fibre-channel-8gb/sg-xpcie2fc-em8-z/

OneCommand Manager Enterprise Application Kit (GUI)
https://docs.broadcom.com/docs/12357358

OneCommand Manager Core Application Kit (CLI)
https://docs.broadcom.com/docs/12357359

You can install the CLI ( core ) package.

Once installed, the equivalents commands to run will be:

 

- After that, try another "luxadm-e forcelip" triggers second firmware update and it completes fine.

Example. See here how second luxadm -e forcelip (after running /opt/EMLXemlxu/bin/emlxadm diag emlxs_post) worked fine:

# luxadm -e forcelip /devices//pci@12,600000/SUNW,emlxs@0,1/fp@0,0:devctl

3) If problem persist, please contact with Oracle Support and open a new SR to address this problem.

In case problem is reproduced, it is required the following information:

Tested this emulex tool on the Oracle Lab server :
Test done on Solaris 10 server while writing on storage LUN, problem not reproduced,
but I'm able to run OneCapture_Solaris_ocmcore.sh or OneCapture_Solaris.sh while there is IO to the storage,
although during a period of 15 secs the link is down (while firmware dump is taken from FC HBA)
so applications could be affected

(not on my case , copying a file to a UFS filesystem on the storage LUN, only one scsi error "Command failed to complete" , no IO to the filesystem during these 15 secs)


/dev/dsk/c1t600A0B800035E82F0000352654190FA7d0s6
                    103008078 33463313 68514685    33%    /test


# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
      0. c0t0d0 <FUJITSU-MBB2147RC-3703 cyl 14087 alt 2 hd 24 sec 848>
         /pci@0,600000/pci@0/pci@0/scsi@0/sd@0,0
      1. c0t1d0 <FUJITSU-MBB2147RC-3703-136.73GB>
         /pci@0,600000/pci@0/pci@0/scsi@0/sd@1,0
      2. c1t600A0B800035E82F0000352654190FA7d0 <SUN-LCSM100_F-0735 cyl 51198 alt 2 hd 64 sec 64>  vol0
         /scsi_vhci/ssd@g600a0b800035e82f0000352654190fa7
Specify disk (enter its number): ^D
# luxadm display /dev/rdsk/c1t600A0B800035E82F0000352654190FA7d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c1t600A0B800035E82F0000352654190FA7d0s2
 Vendor:               SUN
 Product ID:           LCSM100_F
 Revision:             0735
 Serial Num:           SF71101701
 Unformatted capacity: 102400.000 MBytes
 Write Cache:          Enabled
 Read Cache:           Enabled
   Minimum prefetch:   0x3
   Maximum prefetch:   0x0
 Device Type:          Disk device
 Path(s):

 /dev/rdsk/c1t600A0B800035E82F0000352654190FA7d0s2
 /devices/scsi_vhci/ssd@g600a0b800035e82f0000352654190fa7:c,raw
  Controller           /devices/pci@1,700000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0
   Device Address              202400a0b835e82f,0
   Host controller port WWN    10000000c9c07ba0
   Class                       primary
   State                       ONLINE

#


bash-3.2# date
Mon Sep 29 14:28:32 CEST 2014
bash-3.2# ./OneCapture_Solaris_ocmcore.sh
Verifying archive integrity... All good.
Uncompressing Emulex OneCapture...
Running Emulex OneCapture Solaris, version 10.2.370.16
Emulex Corporation Report Utility
Started at Mon Sep 29 14:28:38 CEST 2014
Initializing report environment for host:m3000-tvp540-a
Collecting System Information...
Capture Completed
Generating Compressed Capture...
Done
OneCapture_Solaris_2014-09-29_142838.tgz is under current directory /var/tmp/onecap
bash-3.2#



Sep 29 13:28:56 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.03EF]emlxs0: NOTICE: 710: Link down.
Sep 29 13:28:56 m3000-tvp540-a scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Sep 29 13:28:56 m3000-tvp540-a  /scsi_vhci/ssd@g600a0b800035e82f0000352654190fa7 (ssd0): Command failed to complete (3) on path fp0/ssd@w202400a0b835e82f,0
Sep 29 13:29:08 m3000-tvp540-a emlxs: [ID 349649 kern.info] [13.0303]emlxs0: NOTICE: 200: Adapter initialization. (Firmware update not needed.)
Sep 29 13:29:10 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.062D]emlxs0: NOTICE: 720: Link up. (4Gb, fabric, initiator)




bash-3.2# ./OneCapture_Solaris.sh
Verifying archive integrity... All good.
Uncompressing Emulex OneCapture...
Running Emulex OneCapture Solaris, version 10.2.370.16
Emulex Corporation Report Utility
Started at Mon Sep 29 14:30:17 CEST 2014
Initializing report environment for host:m3000-tvp540-a
Collecting System Information...
Capture Completed
Generating Compressed Capture...
Done
OneCapture_Solaris_2014-09-29_143017.tgz is under current directory /var/tmp/onecap
bash-3.2#

Sep 29 13:30:34 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.03EF]emlxs0: NOTICE: 710: Link down.
Sep 29 13:30:34 m3000-tvp540-a scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Sep 29 13:30:34 m3000-tvp540-a  /scsi_vhci/ssd@g600a0b800035e82f0000352654190fa7 (ssd0): Command failed to complete (3) on path fp0/ssd@w202400a0b835e82f,0
Sep 29 13:30:46 m3000-tvp540-a emlxs: [ID 349649 kern.info] [13.0303]emlxs0: NOTICE: 200: Adapter initialization. (Firmware update not needed.)
Sep 29 13:30:48 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.062D]emlxs0: NOTICE: 720: Link up. (4Gb, fabric, initiator)






bash-3.2# ./OneCapture_Solaris_ocmcore.sh --deaddump
Verifying archive integrity... All good.
Uncompressing Emulex OneCapture...

You have chosen capture which include dead dump
Require to have a system reboot to revover your CNA/HBA
Please do not use this if system is boot from SAN thru HBA
use option --nodeaddump or --bootfromsan to skip dead dumps or all core dumps

Do you wish to continue this capture? (y/n) y
Running Emulex OneCapture Solaris, version 10.2.370.16
Emulex Corporation Report Utility
Started at Mon Sep 29 15:39:43 CEST 2014
Initializing report environment for host:m3000-tvp540-a
Collecting System Information...
Capture Completed
Generating Compressed Capture...
Done
OneCapture_Solaris_2014-09-29_153929.tgz is under current directory /var/tmp/onecap
bash-3.2#

Sep 29 14:40:01 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.03EF]emlxs0: NOTICE: 710: Link down.
Sep 29 14:40:01 m3000-tvp540-a scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Sep 29 14:40:01 m3000-tvp540-a  /scsi_vhci/ssd@g600a0b800035e82f0000352654190fa7 (ssd0): Command failed to complete (3) on path fp0/ssd@w202400a0b835e82f,0
Sep 29 14:40:13 m3000-tvp540-a emlxs: [ID 349649 kern.info] [13.0303]emlxs0: NOTICE: 200: Adapter initialization. (Firmware update not needed.)
Sep 29 14:40:15 m3000-tvp540-a emlxs: [ID 349649 kern.info] [ 5.062D]emlxs0: NOTICE: 720: Link up. (4Gb, fabric, initiator)

The output seems to be the same on all

-rw-r--r--   1 root     root      158540 Sep 29 14:29 OneCapture_Solaris_2014-09-29_142838.tgz
-rw-r--r--   1 root     root      159446 Sep 29 14:30 OneCapture_Solaris_2014-09-29_143017.tgz
-rw-r--r--   1 root     root      159812 Sep 29 15:40 OneCapture_Solaris_2014-09-29_153929.tgz

The three files have the same content:

bash-3.2# gzcat OneCapture_Solaris_2014-09-29_142838.tgz|tar tvf -
drwxr-xr-x   0/0        0 Sep 29 14:29 2014 dump/
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/System_Information/
-rw-r--r--   0/0    29524 Sep 29 14:28 2014 dump/System_Information/sys_info
-rw-r--r--   0/0      190 Sep 29 14:28 2014 dump/System_Information/BiosInfo
-rw-r--r--   0/0   154589 Sep 29 14:28 2014 dump/System_Information/pciconf_output
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/System_information/
-rw-r--r--   0/0      946 Sep 29 14:28 2014 dump/System_information/fcinfo_output
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/NIC_Information/
-rw-r--r--   0/0     5836 Sep 29 14:28 2014 dump/NIC_Information/OneConnect_NIC_information
-rw-r--r--   0/0   1116492 Sep 29 14:28 2014 dump/NIC_Information/sysdef_output
-rw-r--r--   0/0    22807 Sep 29 14:28 2014 dump/NIC_Information/netstat_output
-rw-r--r--   0/0     2260 Sep 29 14:28 2014 dump/NIC_Information/vmstat_output
-rw-r--r--   0/0      527 Sep 29 14:28 2014 dump/NIC_Information/interrupt_output
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/Roce_Information/
-rw-r--r--   0/0    49829 Sep 29 14:28 2014 dump/Roce_Information/Roce_information
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/Storage_Scsi_Info/
-rw-r--r--   0/0     3124 Sep 29 14:28 2014 dump/Storage_Scsi_Info/StorScsiInfo
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/MILI/
-rw-r--r--   0/0      171 Sep 29 14:28 2014 dump/MILI/mili2d.log
-rw-r--r--   0/0      153 Sep 29 14:28 2014 dump/MILI/mili_report
-rw-r--r--   0/0      279 Sep 29 14:28 2014 dump/MILI/MILIservice_info.txt
drwxr-xr-x   0/0        0 Sep 29 14:28 2014 dump/OCM/
-rw-r--r--   0/0     1030 Sep 29 14:28 2014 dump/OCM/ocm_info
-rw-r--r--   0/0     1394 Sep 29 14:28 2014 dump/OCM/rm.log
-rw-r--r--   0/0     1254 Sep 29 14:28 2014 dump/OCM/cnaboardmgmt.log
-rw-r--r--   0/0    12202 Sep 29 14:28 2014 dump/OCM/utils-install.log
-rw-r--r--   0/0     2846 Sep 29 14:28 2014 dump/OCM/hbacmd_hba_attr
-rw-r--r--   0/0     2324 Sep 29 14:28 2014 dump/OCM/hbacmd_port_attr
drwxr-xr-x   0/0        0 Sep 29 14:29 2014 dump/CoreDump/
-rw-r--r--   0/0   2266683 Sep 29 14:29 2014 dump/CoreDump/m3000-tvp540-a_10000000c9c07ba0_140929-142855.dmp
-rw-r--r--   0/0    12334 Sep 29 14:29 2014 dump/CoreDump/m3000-tvp540-a_10000000c9c07ba0_140929-142855.txt
-rw-r--r--   0/0      858 Sep 29 14:29 2014 dump/cmds_output
-rw-r--r--   0/0    33617 Sep 29 14:29 2014 dump/kernel_log
-rw-r--r--   0/0     2203 Sep 29 14:29 2014 dump/kernel_config
drwxr-xr-x   0/0        0 Sep 29 14:29 2014 dump/CrashDump/
-rw-r--r--   0/0     2638 Sep 29 14:29 2014 OneCapture-Solaris.html
bash-3.2#

References

This solution has no attachment