SPARC M8 CMIOU boards may be improperly marked faulty with FMA code SPSUN4V-8000-84

Asset ID:	1-72-2379255.1
Update Date:	2018-03-27
Keywords:

Solution Type Problem Resolution Sure

Solution 2379255.1 : SPARC M8 CMIOU boards may be improperly marked faulty with FMA code SPSUN4V-8000-84

Applies to:

SPARC M8-8 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M8 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

A SPARC M8 CMIOU board may be improperly marked faulty with FMA fault diagnosis code SPSUN4V-8000-84 when System Firmware earlier than 9.8.4.b is installed.

Check the SysFW version installed,

-> show /System/   system_fw_version

/System
    Properties:
        system_fw_version = Sun System Firmware 9.7.5.e 2017/06/02 13:19

The event log (-> show /SP/logs/event/list) will show a message similar to the example below,

Event
ID     Date/Time                 Class     Type      Severity
----- ------------------------ -------- -------- --------
209    Thu Feb 15 21:24:47 2018 Fault     Fault     critical
       Fault detected at time = Thu Feb 15 21:24:47 2018. The suspect
       component: /SYS/CMIOU1 has fault.cpu.generic-sparc.chip-uc with
       probability=100. Refer to http://support.oracle.com/msg/SPSUN4V-8000-84
       for details.

FMA will show the diagnosed faulty CMIOU board in 'fmadm faulty' output similar to the following,

------------------- ------------------------------------ -------------- --------
Time                UUID                                 msgid          Severity
------------------- ------------------------------------ -------------- --------
2018-02-15/21:24:47 ecec4814-7d20-64d1-bcec-aa0a7116bb2b SPSUN4V-8000-84 Critical

Problem Status           : open
Diag Engine              : fdd 1.0
System
   Manufacturer          : Oracle Corporation
   Name                  : SuperCluster M8
   Part_Number           : SuperCluster M8
   Serial_Number         : AK00415164

System Component
   Manufacturer          : Oracle Corporation
   Name                  : SPARC M8-8
   Part_Number           : 7347639
   Serial_Number         : AK00400000
   Firmware_Manufacturer : Oracle Corporation
   Firmware_Version      : (ILOM)4.0.1.1.c,(POST)5.7.0.a,(OBP)4.42.0,(HV)1.19.0.b
   Firmware_Release      : (ILOM)2017.09.06,(POST)2017.08.25,(OBP)2017.07.26,(HV)2017.08.25

----------------------------------------
Suspect 1 of 1
   Problem class : fault.cpu.generic-sparc.chip-uc
   Certainty      : 100%
   Affects        : /SYS/CMIOU1/CM/CMP
   Status         : faulted

   FRU
      Status            : faulty
      Location          : /SYS/CMIOU1
      Manufacturer      : Oracle Corporation
      Name              : CMIOU Module
      Part_Number       : 7346734
      Revision          : 01
      Serial_Number     : 465769T+1730GR0000
      Chassis
         Manufacturer   : Oracle Corporation
         Name           : SPARC M8-8
         Part_Number    : 7347639
         Serial_Number : AK00400000
   Resource
      Location          : /SYS/CMIOU1/CM/CMP

Description : This chip has encountered a chip-level uncorrectable error.

Response    : The system will attempt to retire affected resources.

Impact      : System performance may be affected.

Action      : Use 'fmadm faulty' to provide a more detailed view of this
              event. Please refer to the associated reference document at
              http://support.oracle.com/msg/SPSUN4V-8000-84 for the latest
              service procedures and policies regarding this diagnosis.

FMA diagnosis code SPSUN4V-8000-84 requires three criteria to match the misdiagnosed faulty condition.

The CMIOU must be a SPARC M8 CMIOU
SysFW must be earlier than 9.8.4.b
Error reports (ereports) must diagnose the uncorrectable CMIOU fault for error condition DVFS_ISENSE_ERR

This last condition must be checked in ereport telemetry just prior to the FMA fault diagnosis. Start the fault management shell (-> start -script /SP/faultmgmt/shell) and check the 'fmdump -eV' output for ereports similar to the one shown below,

2018-02-15/21:24:47 ereport.cpu.generic-sparc.gchip-uc@/SYS/CMIOU1/CM/CMP <<========
__tod-0 = 0x5a85ec8f
__tod-1 = 0x36f2f680
tstate = 0x19911001004
htstate = 0x4
ehdl = 0x201200000000001
tpc = 0xba6a3e4
tl = 0x2
tt = 0x154
stick = 0x947ecd15e5cf9
chip-seq-id = 0x1000000000001
cpuid = 0x201
diagnose = 0x1
error-condition = DVFS_ISENSE_ERR <<=======

Cause

Internal-only visibility bug 24468184.

Solution

Clear the improperly faulted CMIOU and install SysFW 9.8.4.b or higher. See References section for help upgrading System Firmware.

The faulted CMIOU can easily be cleared as follows,

-> set /SYS/CMIOU1 clear_fault_action=true
Are you sure you want to clear /SYS/CMIOU1 (y/n)? y
Set 'clear_fault_action' to 'true'

References

<NOTE:1542610.1> - SPARC Mx-32, M7 and M8 Servers: Firmware FAQ
<NOTE:1309092.1> - How to use the Oracle ILOM 3.x Fault Management Shell
<BUG:24468184> - FAULT.CPU.GENERIC-SPARC.CHIP-UC ON T8, EREPORT DVFS_ISENSE_ERR
<NOTE:1967048.1> - SPARC M8 and SPARC M7 Series Servers : Firmware Image Software Version Matrix Information

Attachments

This solution has no attachment