SPARC T3/T4 Servers - PCIe Host Bus Adapter (HBA) Disappears During Reboot

Asset ID:	1-72-1571110.1
Update Date:	2015-07-27
Keywords:

Solution Type Problem Resolution Sure

Solution 1571110.1 : SPARC T3/T4 Servers - PCIe Host Bus Adapter (HBA) Disappears During Reboot

Applies to:

SPARC T4-2 - Version All Versions and later
Netra SPARC T4-1 Server - Version All Versions and later
SPARC T4-4 - Version All Versions and later
Netra SPARC T4-2 Server - Version All Versions and later
Netra SPARC T4-1B - Version All Versions and later
Oracle Solaris on SPARC (64-bit)

Symptoms

PCIe Host Bus Adapter (HBA) is missing from the device tree after a reboot. This issue can effect any of the supported PCIe host bus adapter types, i.e. Fibre Channel, Network Interface, SCSI or SAS, etc.

For example, prior to a reboot /etc/path_to_inst and prtdiag -v output might contain the following entries:

"/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0" 0 "qlc"
"/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1" 1 "qlc"
"/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0" 2 "qlc"
"/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1" 3 "qlc"

================================ IO Devices ================================
Slot +            Bus   Name +                            Model
Status            Type Path
----------------------------------------------------------------------------
/SYS/MB/RISER0/PCIE0PCIE SUNW,qlc-pciex1077,2532 QLE2562
/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0
/SYS/MB/RISER0/PCIE0PCIE SUNW,qlc-pciex1077,2532 QLE2562
/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1
/SYS/MB/RISER1/PCIE1PCIE SUNW,qlc-pciex1077,2532           QLE2562
                        /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0
/SYS/MB/RISER1/PCIE1PCIE SUNW,qlc-pciex1077,2532           QLE2562

After a reboot, /etc/path_to_inst remains unchanged, but prtdiag -v output contains just the following entries:

================================ IO Devices ================================
Slot +            Bus   Name +                            Model
Status            Type Path
----------------------------------------------------------------------------
/SYS/MB/RISER1/PCIE1PCIE SUNW,qlc-pciex1077,2532           QLE2562
                        /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0
/SYS/MB/RISER1/PCIE1PCIE SUNW,qlc-pciex1077,2532           QLE2562
                        /pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1

Cause

During PCIe link retraining the system platform PCI Express Switch may lock up on individual ports resulting in that port not training and the PCIe Host Bus Adapter (HBA) not being mapped into the device tree.

Solution

This document has been created to help prevent the unnecessary replacement of HBAs when encountering this issue.

This issue has been addressed in SPARC T-Series System Firmware 8.4.0.a, POST Version 4.35.3, and Hypervisor 1.12.3.

<Patch 150676-01> 13-Sep-2013 FIRMWARE: SPARC T4-1 - Sun System Firmware 8.4.0.a
<Patch 150677-01> 13-Sep-2013 FIRMWARE: SPARC T4-2 - Sun System Firmware 8.4.0.a
<Patch 150678-01> 13-Sep-2013 FIRMWARE: SPARC T4-4 - Sun System Firmware 8.4.0.a
<Patch 150679-01> 13-Sep-2013 FIRMWARE: SPARC T4-1B - Sun System Firmware 8.4.0.a
<Patch 150680-01> 13-Sep-2013 FIRMWARE: Netra SPARC T4-1 - Sun System Firmware 8.4.0.a
<Patch 150681-01> 13-Sep-2013 FIRMWARE: Netra SPARC T4-2 - Sun System Firmware 8.4.0.a
<Patch 150682-01> 13-Sep-2013 FIRMWARE: Netra SPARC T4-1B - Sun System Firmware 8.4.0.a

Firmware component revisions included with this release:

Sun System Firmware 8.4.0.a 2013/08/20 16:29
--------------------------------------------
ILOM 3.2.1.5.c Aug 20 16:30:18 PDT 2013
Hostconfig 1.3.3.c 2013/08/20 15:36
Hypervisor 1.12.3.b 2013/08/20 10:18
OpenBoot 4.35.3 2013/08/05 11:35
POST 4.35.3 2013/08/05 11:39
GM 1.3.3.c 2013/08/20 15:47

The system firmware version can be verified with prtdiag -v:

============================ FW Version ============================
Version
------------------------------------------------------------
Sun System Firmware 8.4.0.a 2013/08/20 16:29

====================== System PROM revisions =======================
Version
------------------------------------------------------------
OBP 4.35.3 2013/08/05 11:35

Or from ILOM:

-> show /HOST sysfw_version

/HOST
Properties:
sysfw_version = Sun System Firmware 8.4.0.a 2013/08/20 16:29

This issue has also been addressed in SPARC T-Series System Firmware 8.3.1.b, POST Version 4.34.5, and Hypervisor 1.11.5.

<Patch 150407-05> 16-Dec-2013 FIRMWARE: SPARC T3-1 - Sun System Firmware 8.3.1.b

Firmware component revisions included with this release:

Sun System Firmware 8.3.1.b 2013/12/10 09:46
--------------------------------------------
ILOM 3.2.1.5.k Dec 10 09:46:31 PST 2013
Hostconfig 1.2.5 2013/10/28 10:52
Hypervisor 1.11.5 2013/10/28 10:16
OpenBoot 4.34.5 2013/10/28 09:25
POST 4.34.5 2013/10/28 09:35
GM 1.2.5

If for some reason the system firmware cannot be updated, some potential workarounds are to just reboot the server again, power cycle the server, or reseat the HBA.

References

<BUG:16725923> - PCI DEVICES DISAPPEAR DURING REBOOT (DURING PCI LINK RE-TRAIN) - POST
<BUG:16725733> - PCI DEVICES DISAPPEAR DURING REBOOT (DURING PCI LINK RE-TRAIN) - HYPERVISOR
<BUG:15849720> - PCI DEVICES PRESENT DURING POST PROBE BUT DISAPPEAR LATER
<BUG:16977142> - BACKPORT 16725923 TO 4.34.X: PCI DEVICES DISAPPEAR DURING REBOOT (DURING PCI LI
<NOTE:1439072.1> - Product Issues Shared By T3/T4/T5 Servers

Attachments

This solution has no attachment