Sun SPARC Enterprise M8000 Server
 
Sun SPARC Enterprise M4000 Server
 
Sun SPARC Enterprise M3000 Server
 
Sun SPARC Enterprise M9000-32 Server
 
Sun SPARC Enterprise M5000 Server
 
Sun SPARC Enterprise M9000-64 Server
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: Mx000
 
_Old GCS Categories>Sun Microsystems>Servers>OPL Servers
 

Applies to:

Goal

The implementation of Fault Management Architecture (FMA) on Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers is complex. The goal of this document is not to describe how FMA behaves on the Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers but to help to identify and display the faults reported on the components of these platforms, and how and when these faults can be cleared and how and when the fault LEDs are turned ON or OFF.

Solution

Faults on Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers :

FMA (Fault Management Architecture) is available on Sun SPARC Enterprise platforms running Solaris[TM] 10 and beyond. For M-series systems FMA is also built into the Service Processor as part of the Service Processor (aka XSCF) software. Error reports and faults are passed between XSCF and the Solaris domain via the "Event Transport Module" (ETM) using "Domain to Service Processor Communications Protocol" (DSCP).

When a fault is diagnosed the system is usually able to identify one or more suspect components depending on the nature of the fault. The suspect or the list of the suspects can be displayed using the 'fmdump' command on XSCF.

Example where a list of suspects has been identified :

Example where only one suspect has been identified :

M-Series related Knowledge Article Documents (KA docs) available at https://support.oracle.com suggest that one collect the output from 'fmdump -m' to aid in diagnosisng these faults.  The "-m" option is available only on the XSCF (not on the Solaris domain) and displays the Fault Manager syslog message contents for the event(s).

Example :

XSCF user ID's with the 'platop', 'platadm', or 'fieldeng' privileges can run the 'fmdump' command.

Information about the faulty status of all components is available in the CMEM database on XSCF. Based on the 'level' of certainty of the diagnosis of any given fault, the following flags are set:

• CFF => Certainly Faulty Flag

• UFF => Uncertainly Faulty Flag

Every FRU that is marked as 'suspect' in the list will have the uncertain_secondary_status bit set; however, only the primary suspect will have either the CFF or UFF bit set as well.

As a result of having detected a fault and depending on which one of the CFF or UFF bits is set, the primary suspects in a suspect list are reported as "faulted" (completely broken/not working) or "degraded" (should be replaced, but is still working with some limitations). This status can be viewed in the output of the 'showhardconf' and 'showstatus' commands on the XSCF.

Secondary suspects show up as "degraded".  If such components are victims of a fault detected on another component they may be reported as "deconfigured".

The 'fmadm faulty' command on the XSCF is only available in Escalation mode, and examines the resource cache, which typically has less info about platform faults than the "CMEM" database. The CMEM database contains all of the information about faulty FRUs for OPL (M-series) platforms. For this reason the XSCF 'showstatus' command is the preferred method to be used by the customer and field engineering, as it provides the most accurate information pertaining to FMA faults.

The XSCF 'showstatus' and 'showhardconf' commands are available to users ID's with the following privileges :

useradm, platadm, platop, fieldeng

Example showstatus and showhardconf command output reporting a chip on a CPU Module as "faulted".

Example from a M9000 system where a CMU is reported as degraded due to some DIMMs deconfigured as a result of a a fault detected on a Memory Address Controller.

For components reported as the primary suspect and with certainly faulty (CFF) bit set, the Maintenance Action Required bit is set.  This information is available in the output of the 'fmdump -V' command.

Example :

Further OPL FMA information can be found at:

<Document 1386385.1> M3000/M4000/M5000/M8000/M9000 Server: How to Use FMA With OPL Servers

Steps to Follow

Clearing Faults on Sun SPARC [TM] Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers.

Note:  In XCP1115 and above the clearfault command can be performed from normal mode.

Clearing Faults (Sun SPARC Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers running XCP 1050 or later) :

1. Usual process :

When the system has identified faulty components on a platform, the correct action is normally to replace the primary suspect. There are however certain conditions detailed futher under 'Complex Cases' in this document where this might be deferred. Here we discuss standard procedure.

In order to repair a fault with single or multiple FRUs, the typical repair action will be:

• Replace the first FRU indicted in the suspect list; if the FRU is a CMU or sub-FRU of a CMU (on M8000/M9000), IOU (on M8000/M9000), FAN, PSU, DDC_A (on M8000), XSCFU (on M8000/M9000), or XSCFU_C (M9000 plus expansion cabinet) use the 'replacefru' command to do so; Otherwise, use cold replacement;
  The user must have the 'fieldeng' privilege in order to be able to run the 'replacefru' command.

• For all the other FRUs on the suspect list (secondary suspects), use the 'clearfault' command.
  The 'clearfault' command will return with an error message when trying to clear a FRU that is the first suspect on a suspect list :
  
  "clearfault: Fault cannot be cleared for this FRU"
  You must be in Service or Escalation mode to run the 'clearfault' command. 

• Sun Shared Shell
  The Service Engineer should offer the use of Shared Shell or some other remote access method to enter service (or escalatioin) mode and execute the clearfault command(s). Please be aware that, before *any* remote access session is terminated in which either service or escalation mode has been entered, it is essential the "disablemodes" command be executed to leave service or escalation mode.

2. Complex cases :

In some more complex cases, analysis of FMA events might result in the decision NOT to replace the primary FRU in the suspect list. This might be due to the diagnosis engine being affected by some SW issue, or perhaps analysis shows that the FRU in a single-FRU indictment seems to be wrong or the first FRU has already been replaced and this is a repeat fault with an identical list, etc. etc.

The decision to not replace the first FRU in the suspect list MUST be made by Service and the entire process of clearing a fault without replacing the suspect component must be done under the supervision of TSC, preferably via remote access shared session. The following section describe how to handle the complex cases in more detail.

2.1 - Power cycle via NFB (Non-Fused Breaker) Off/On :

The term 'NFB Off/On' means to power cycle the system by temporarily removing, and then restoring, AC input power. For systems running firmware version XCP1050 it is possible to clear fault status for primary suspects by power-cycling the platform with the keyswitch is in the 'service' position.  For systems running XCP1060 and later, faults are not cleared on NFB-on, no matter what the position of the keyswitch.

2.2 - Commands available to clear the faults :

2.2.1 - clearfru / clearstatus :

These two commands can be used to clear the fault information of all the FRUs (clearfru) or the fault information of FRUs that have been detected as faulty units (clearstatus).

You must be in Escalation mode to run the clearfru / clearstatus commands.

The domains must be down and an immediate platform power cycle is required. The component is reported as faulted as long as the power cycle hasn't occured.

The use of 'clearfru' and 'clearstatus' commands must be done *only* under direction from TSC and/or Engineering.

Example :

2.2.2 - clearfault :

The 'clearfault' command provides a way to manage faults for primary suspects and may be used for the following actions :

• Immediately clear fault status for FRUs other than the first suspect in a multi-suspect list,

• Clear faults for FRUs reported as the first suspect using 'clearfault [-l]'
  (no need for the keyswitch to be in the service position).

As a result of executing this command, faults will be cleared on the next power-cycle.

Notes :

• If the user invokes 'clearfault -l', there is no attempt to clear the fault status now. The user is prompted with the information that the  "FRU will be marked to clear fault on next circuit breaker off and on."

Example :

• If the user invokes 'clearfault' (without -l), the system will attempt to clear the status now.
  If the FRU is a primary suspect and clearfault cannot clear the fault status, it will prompt the user requesting whether or not the FRU should be marked to clear fault on NFB off/on; behaving the same manner as invoking 'clearfault -l' directly.

Example :

• If clearfault is invoked when another maintenance command is running, it will behave the same manner as invoking 'clearfault -l' directly.

Example :

2.2.2.2 - M3000/M4000/M5000 :

• Immediately clear the fault for the first suspect in a suspect list if the FRU is a PSU or a FAN,

Example :

• For the other FRUs, a power cycle is required to clear the fault.

Example :

2.2.2.3 - M8000/M9000 :

• Immediately clear the fault for the first suspect in a suspect list if the FRU is a PSU or FAN or DDC_A .

Example :

• Immediately clear fault if the FRU is a CMU or IOU, including sub-FRUs like CPUM, DIMM ... and non-FRU like DDC, SSM ..., without NFB off/on when :

  • the CMU, IOU (and sub-FRU and non-FRU) is not part of a running domain. See the Examples section below,

  • If the FRU is a CMU (and sub-FRU and non-FRU) or an IOU and is part of a running domain, Dynamic Reconfiguration (deleteboard) must be invoked prior to running the clearfault command in order to detach the FRU from the domain. After clearing the fault, the component can safely be added back to the domain. See the Examples section below.

  • If the FRU is a CMU (and sub-FRU and non-FRU) or an IOU and is part of a running domain but DR cannot be used then the domain must be powered off in order to immediately clear the fault status. Otherwise, the fault status will be cleared on next circuit breaker off and on. See the Examples section below.

    • Note : if the XSB containing the CMU/IOU is in quad-mode, all of the domains using the quads must be powered off.

• If the FRU is an XSCFU, a platform power cycle is required or else replacefru can be invoked from the other XSCFU (as long as it is MAIN) to dummy replace and power cycle the SPARE XSCFU.

2.3 - As a summary, to clear fault on a FRU :

M3000/M4000 / M5000 :

M8000 / M9000 :

For more detail on accessing escalation or service mode see <Document 1002928.1>

LED behaviour :

Each M3000/M4000/M5000/M8000/M9000 system has an Operator Panel (OPNL) with 3 LEDs :

• the Power LED,

• the XSCF Standby LED,

• the Check LED.

When turned ON, the Check LED, aka the System Check LED, indicates a fault on the system. See below.

Most of the FRUs on the SPARC Enterprise servers have a FRU check LED which reports that the unit contains an error. However, some FRUs like DIMMs or CPUMs do not have LEDs.  Refer to the SPARC Enterprise Mx000/Mx000 Servers Service Manuals for more information about LEDs.

For Sun SPARC Enterprise servers running a version of XCP later than 1050, the check LEDs will be set and reset as below :

• the FRU check LED is set if the FRU is the sole FRU in a suspect list; including sub-FRU (CPUM, DIMM ...) and non-FRU (DDC, SSM ...).

• the system check LED is set if there are any FRUs which is considered as the primary suspect (CFF / UFF) or secondary suspect; which means when 'showstatus' reports any FRUs as faulty or degraded.
  Including IO Box FRUs reported as suspect.

Check LEDs behaviour after clearfault, clearstatus, replacefru :

• replacefru :

  • the FRU's check LED is :

    • ON until the maintenance,

• blinking during the maintenance,

• OFF as soon as the replacefru as completed successfully.

  • the System check LED is OFF :

    • as soon as the replacefru as completed successfully,

• and there is no other suspect component in the system left,

• clearfault :

  • the FRU's check LED is turned off as soon as the clearfault command has succeeded successfully in clearing the fault for the FRU.

• the System's check LED is turned off as soon as the fault status for the latest suspect component is cleared.
  This implies that the LED will turn off after the subsequent power cycle in certain cases as applicable.

• clearstatus/ clearfru : the FRU and System check LEDs remain ON until the next power cycle,

Faults on IOBox :

Faults detected on IOBox are stored in the CMEM and in the FRUID of the IOBox (Status_CurrentR).
This information is reported in the showstatus output on the XSCF.
Example :

When a fault is reported on the IOBox or its components, the Service LED on the IOBox or PSU is lit.
When an iobox fru is discovered, dfrud reads the Status_CurrentR. If it contains fault info, the fault info is added to CMEM, and the Service led is turned on. This can be checked via the ioxadm command :

XSCF> ioxadm env -v Location Sensor Min Min Alarm Value Max Alarm Max Units [...] IOX@X156/IOB1 SERVICE - - On - - LED

Even if a fault is reported on IOBox and Service LED is lit, the OPNL System Ckeck LED is not lit .
The clearfault command can be used to clear the fault status for primary and secondary suspect on the IOBox and its components; similarly to any other components in the platform chassis (CMU, DIMM, IOU etc ...) for XCP > 1050.
Example :

Clearing the LINK to the IOBox:
Example:

Clearing a fault on an IOX

 
As soon as there is no more fault status reported in the showstatus output then all the Service LEDs are cleared. There is no condition requiring to power cycle the IOBox to clear a  fault status (similar to clearfault -l).

Hierarchical fault clearing :

In certain cases, the faulted resources appear to be hierarchical.
In the following example, after clearing the fault on CMU#0, we need to clear the fault on the subordinates.

CMU#0 remains in the output, although not marked faulted, until the subordinates are cleared:

1. M3000/M4000 / M5000 :

1.1 - clearing a fault on a PSU :

1.2 - clearing a fault on a DIMM :

1.3 - clearing a fault on a CPUM :

1.4 - clearing a degraded MBU :

2. M8000 / M9000 :

2.1 - clearing a fault on a PSU :

2.2 - clearing a fault on the OPNL :

2.3 - clearing a fault on an IOU not part of a running domain :

2.4 - clearing a fault on a CMU not part of a running domain

2.5 - clearing a fault on a CMU which is part of a running domain :

We can use DR to detach the XSB and clear the fault.

2.6 - clearing a fault on a CMU which is part of a running domain but DR cannot be used :

Since DR cannot be used for whatever reason, the domain must be powered off prior to using clearfault :

Keywords: opl, fault, clear, clearfault, clearstatus, fmdump, showstatus

References

This solution has no attachment