SPARC T5-4
 
SPARC T5-8
 

PLA-Support>Sun Systems>SPARC>CMT>SN-SPARC: T5
 

In this Document

Symptoms

Changes

Cause

Solution

Created from <SR 3-7226337091>

Applies to:

Symptoms

SPARC T5-4 and SPARC T5-8 systems configured with SG-XPCIE2FC-QF8-Z Qlogic 'Pallene' fiber channel HBAs may panic on a fabric error, referencing one of the Pallene populated slots, or report ereports on Pallene populated slots with the HBA running at PCIe Gen2.

++ PCIe panic ++

The stack may differ, however the common symptom will be various correctable events reported against installed QLogic 'Pallene' HBAs, for example;

ereport.io.pci.fabric ena=2741168c7b21001 detector=[ version=0 scheme="dev"
 device-path="/pci@380/pci@1/pci@0/pci@a" ] bdf=250 device_id=80bf vendor_id=
 111d rev_id=3 dev_type=60 pcie_off=40 pcix_off=0 aer_off=100 ecc_ver=0
 pci_status=10 pci_command=147 pci_bdg_sec_status=0 pci_bdg_ctrl=3 pcie_status=
 1 pcie_command=12f pcie_dev_cap=8024 pcie_link_status=3042 pcie_dev_ctl2=0
 pcie_adv_ctl=1e5 pcie_ue_status=0 pcie_ue_mask=0 pcie_ue_sev=460031
 pcie_ue_hdr0=0 pcie_ue_hdr1=0 pcie_ue_hdr2=0 pcie_ue_hdr3=0 pcie_ce_status=
 1100 pcie_ce_mask=0 remainder=1 severity=3

++ Correctable PCIe events ++

This is the most common symptom and will report correctable errors against the HBA, for example from {snapshot}/fma/@usr@local@bin@fmdump_-ev.out;

ereport.chassis.post.io.pciex.correctable@/SYS/RCSA/PCIEx/CAR

++ PCIe link initialisation failures ++

Another common fault is a failure by HyperVisor to correctly train the PCI links during initialisation, for example from {snapshot}/fma/@usr@local@bin@fmdump_-ev.out;

ereport.io.pciex.link-init-fail@/SYS/RCSA/PCIEx

Changes

Cause

SPARC T5-4 and T5-8 systems support only the Gen1 version of the SG-XPCIE2FC-QF8-Z card, PN 7076907.

SG-XPCIE2FC-QF8-Z (371-4325) installed in SPARC T5-4 and SPARC T5-8 systems require a platform specific FW image to allow the cards to negotiate at PCIe Gen1 x8 speeds.

SG-XPCIE2FC-QF8-Z installed with stock FW will negotiate at PCIe Gen2 x4 speeds which can cause a variety of platform issues - documented in bug 15894188.

Solution

To confirm card presence via the SP;

-> show /System/PCI_Devices/Add-on/Device_9

  /System/PCI_Devices/Add-on/Device_9
    Properties:
        part_number = SG-XPCIE2FC-QF8-Z
        description = Sun StorageTek Dual 8 Gb Fibre Channel PCIe HBA
        location = PCIE9 (PCIE 9)
        pci_vendor_id = 0x1077
        pci_device_id = 0x2532
        pci_subvendor_id = 0x1077
        pci_subdevice_id = 0x0171

Use prtdiag to confirm whether the cards are operating at PCIe Gen1 or PCIe Gen2 from within Solaris.

Output showing cards operating at incorrect PCIe Gen2 speeds;

# prtdiag | grep QLE
/SYS/RCSA/PCIE3   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      5.0GTx4
/SYS/RCSA/PCIE3   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      5.0GTx4
/SYS/RCSA/PCIE9   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      5.0GTx4
/SYS/RCSA/PCIE9   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      5.0GTx4

Output showing cards running at the correct PCIe Gen1 speeds;

# prtdiag | grep QLE
/SYS/RCSA/PCIE3   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      2.5GTx8
/SYS/RCSA/PCIE3   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      2.5GTx8
/SYS/RCSA/PCIE9   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      2.5GTx8
/SYS/RCSA/PCIE9   PCIE  SUNW,qlc-pciex1077,2532           QLE2562      2.5GTx8

Cards found to be running at PCIe Gen2 should be replaced with the correct model.

A PCIe Gen1 QLogic Pallene DP HBA supported on SPARC T5 is available for customer order;

Physical Service Item FRU: 7076907
Packaged FRU: 7076908
Marketing FRU: 7106955

For any SPARC T5-4 and SPARC T5-8 systems configured with Manufacturing supplied SG-XPCIE2FC-QF8-Z cards that are found to be running at PCIe Gen2, please contact justin.hatch@oracle.com and peter.mo@oracle.com

Sun System Handbook T5-4 https://mosemp.us.oracle.com/handbook_internal/Systems/SPARC_T5_4/components.html#FibreChannel

Sun System Handbook T5-8 https://mosemp.us.oracle.com/handbook_internal/Systems/SPARC_T5_8/components.html#FibreChannel

NOTE : As per SSH - for SPARC T5-4 and T5-8 failed 371-4325 provided as part of the system build must be replaced with part number 7076907 which operate at PCIe Gen1x8 speed. Any impacted 371-4325 provided from a source other than Oracle Manufacturing should be replaced with 7076907 obtained from the same source.

In break/fix scenarios where there is insufficient local RSL stock to replace the customers HBAs, or where excessive lead times on replacements will negatively impact CSAT please contact the following regional engineers: paul.lim@oracle.com (APAC) justin.hatch@oracle.com (EMEA) peter.mo@oracle.com (AMER).

This solution has no attachment