SPARC M8 and M7 server PCIE faults may identify an incorrect FRU

Asset ID:	1-72-2391817.1
Update Date:	2018-04-26
Keywords:

Solution Type Problem Resolution Sure

Solution 2391817.1 : SPARC M8 and M7 server PCIE faults may identify an incorrect FRU

Applies to:

SPARC M7-8 - Version All Versions to All Versions [Release All Releases]
SPARC M7-16 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M7 Hardware - Version All Versions to All Versions [Release All Releases]
SPARC M8-8 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)

Symptoms

PCIE fault reports on SPARC M8 and M7 servers with Solaris 11.3 SRU24 and higher may indicate an incorrect FRU in 'fmadm faulty' output. Solaris FMA will list the IOS root port number rather than identifying the PCIE card slot. This may occur for all PCIE-8000- diagnosis codes.

Example,

# fmadm faulty

--------------- ------------------------------------ -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------ -------------- ---------
Feb 10 12:27:28 f8426610-1571-4fa9-9785-990644296c2c PCIEX-8000-0A Critical

Problem Status    : open
Diag Engine       : eft / 1.16
System
    Manufacturer : unknown
    Name          : unknown
    Part_Number   : unknown
    Serial_Number : unknown
    Host_ID       : 84fa7000

----------------------------------------
Suspect 1 of 1 :
   Problem class : fault.io.pciex.device-interr
   Certainty   : 100%
   Affects     : dev:////pci@303/pci@1/SUNW,emlxs@0,f
   Status      : faulted but still in service

   FRU
     Status           : faulty
     Location         : "/SYS/CMIOU0/IOH/IOS3/RP0"             <=======root port improperly indicated
     Manufacturer     : unknown
     Name             : unknown
     Part_Number      : unknown
     Revision         : unknown
     Serial_Number    : unknown
     Chassis
        Manufacturer : Oracle Corporation
        Name          : SPARC M7-8
        Part_Number   : 33972225+1+1
        Serial_Number : AK00361296

Description : A problem was detected for a PCIEX device.

Response    : One or more device instances may be disabled

Impact      : Loss of services provided by the device instances associated with
              this fault

Action      : Use 'fmadm faulty' to provide a more detailed view of this event.
              Please refer to the associated reference document at
              http://support.oracle.com/msg/PCIEX-8000-0A for the latest
              service procedures and policies regarding this diagnosis.

Changes

The Solaris 11.3 SRU24 fix that triggered the root port identification was the fix for bug 25965439

Cause

Solaris 11.3 SRU24 modified the NAC target name. This causes FMA to identify the IO switch root port as the faulty FRU rather than the suspected PCIE card.

Check the SRU level installed via the following command,

root@server_name:~# pkg info entire
             Name: entire
          Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.24.4.0).
      Description: This package constrains system package versions to the same
                   build. WARNING: Proper system update and correct package
                   selection depend on the presence of this incorporation.
                   Removing this package will result in an unsupported system.
                   For more information see:
                   https://support.oracle.com/rs?type=doc&id=2045311.1
         Category: Meta Packages/Incorporations
            State: Installed
        Publisher: solaris
          Version: 0.5.11 (Oracle Solaris 11.3.24.4.0)
    Build Release: 5.11
           Branch: 0.175.3.24.0.4.0
   Packaging Date: Fri Sep 08 18:22:56 2017
Last Install Time: Thu Nov 02 16:37:23 2017
             Size: 5.46 kB
             FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.24.0.4.0:20170908T182256Z

Solution

Fix: Install SysFW 9.8.5.a or higher

Important Note : For Oracle SuperCluster M7 and SuperCluster M8 Hardware (SuperCluster Patch Policy)

QFSDP release is the supported vehicle for SysFW deployment on SuperCluster. See Doc ID 1567979.1 for details. It may be necessary to seek exception approval for SysFW upgrade outside a QFSDP release.

Never tell a SuperCluster customer to patch an individual component in isolation. SysFW 9.8.5.a or higher is not yet in a QFSDP and must receive exception approval on a case-by-case basis for SuperCluster.

Reactive patching is only allowed for critical issues with no easy/viable workaround. For approval always check with SuperCluster Maintenance Group first - ssc_maintenance_grp@oracle.com.

Until the the server SysFW can be upgraded the following table can be used to identify the faulted PCIE card. Substitute the proper CMIOU number for each entry.

FMA Reported FRU Location	Actual Faulted FRU
/SYS/CMIOUn/IOH/IOS3/RP0	/SYS/CMIOUn/PCIE1
/SYS/CMIOUn/IOH/IOS0/RP0	/SYS/CMIOUn/PCIE2
/SYS/CMIOUn/IOH/IOS1/RP0	/SYS/CMIOUn/PCIE3

References

<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types
<NOTE:1967048.1> - SPARC M8 and SPARC M7 Series Servers : Firmware Image Software Version Matrix Information
<NOTE:1542610.1> - SPARC Mx-32, M7 and M8 Servers: Firmware FAQ

Attachments

This solution has no attachment