How To Clear FMA faults from Solaris[TM] and SC (System Controller) on T1000/T2000 T5120/T5220/T5140/T5240/T5440,T6320,T6340, T3-1/T3-2/T3-4, T4-1/T4-2/T4-4

Asset ID:	1-71-1004229.1
Update Date:	2018-03-14
Keywords:

Solution Type Technical Instruction Sure

Solution 1004229.1 : How To Clear FMA faults from Solaris[TM] and SC (System Controller) on T1000/T2000 T5120/T5220/T5140/T5240/T5440,T6320,T6340, T3-1/T3-2/T3-4, T4-1/T4-2/T4-4

Applies to:

Sun SPARC Enterprise T2000 Server - Version All Versions and later
SPARC T3-1 - Version All Versions and later
Sun SPARC Enterprise T1000 Server - Version All Versions and later
Sun Blade T6320 Server Module - Version All Versions and later
Sun SPARC Enterprise T5220 Server - Version All Versions and later
All Platforms

Goal

Description
This document describes how to clear the FMA faults and error logs from Solaris and SC/ALOM or SP/ILOM level on Sun Fire T1000/T2000, SPARC Enterprise T5x20/T5x40 and SPARC T3-x/T4-x servers

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the SPARC T-Servers Support Community

Solution

Steps to Follow

Clearing faults from SP (the commands shown below are ALOM-CMT commands that can be issued only from ALOM compatibility shell)(NOT USED FOR T3-X or T4-X):

a) Show the faults on the system controller

sc> showfaults -v

b) For each fault listed run clearfault followed by the UUID obtained from above.

sc> clearfault 6f6b149a-2206-c967-ebd4-a3c52be6bed0  (UUID in this example is 6f6b149a-2206-c967-ebd4-a3c52be6bed0)

c) If there are any disabled components run (NOTE: this step is not typically needed)

sc> clearasrdb

d) Clear ereports (NOTE: this step is not typically needed)

sc> setsc sc_servicemode true
sc> showfmerptlog1 -v
sc> showfmerptlog2 -v
sc> clearereports -y  (if you want to clear them and start with a clean slate)
sc> setsc sc_servicemode false
sc> resetsc

Clearing faults from SP/ILOM:

See section C.1 in doc 1155200.1 for Using the ILOM Command Line Interface (CLI) to Clear the Faults.

a) Refer to the T5120/T5220 Service Manual for detailed steps
b) Refer to the T5140/T5240 Service Manual for detailed steps

NOTE: There in no ALOM compatibility shell for the SPARC T3-x/T4-x platforms, and ILOM is the only available shell. See doc 1012360.1
The SPARC T3-x (& newer) servers also support the ILOM Fault Management Shell (FM shell), which can be accessed from SP through:

-> start /SP/faultmgmt/shell
Are you sure you want to start /SP/faultmgmt/shell (y/n) ? y
faultmgmtsp>

The FM shell allows the FMA logs to be analyzed with fma commands similar to what is used with FMA in Solaris. Refer to doc 1309092.1 for details. This will assist the Field in troubleshooting faults diagnosed by FMA. Note that in the case of faults proxied from the host OS and cleared by way of the Fault Management Shell, they will be cleared ONLY from the shell and ILOM but not Solaris. See FMA/PSH doc 1155200.1 for more details on using FM shell.

To clear the FMA faults and error logs from Solaris:

a) Show faults in FMA

# fmadm faulty

NOTE: Do not use 'fmadm faulty -a' in this step. When you specify the -a option
all resource information cached by the Fault Manager is listed, including 
faults, which have already been corrected or where no recovery action is needed
(see 'fmadm' man page). The listing also include information for resources,
that may no longer be present in the system

b) For each fault UUID/EVENT-ID listed in the 'fmadm faulty' run

# fmadm repair <uuid/event-id> example: #fmadm repair 6f6b149a-2206-c967-ebd4-a3c52be6bed0

Starting with Solaris 10 update 7 there is an equivalent syntax that can also be used

#fmadm repaired <fmri | label> example: #fmadm repaired /SYS/MB

# fmadm faulty     (to make sure the output is clean after repair)

 NOTE: For more details on FMA-based diagnosis and commands for clearing faults, 
refer to article 1173733.1 (section C.1.2.4 Clearing All Faults)

NOTE: There is an issue (CR 6983432), which causes previously diagnosed and repaired PSH faults from the host to reappear (to be replayed) in Oracle ILOM when the host reboots. It manifests itself as an incorrect report of a PSH-diagnosed fault represented through the Oracle ILOM CLI, BUI, and fault LED. You can identify this defect by checking to see if the same PSH fault was reported from the host as well.

The affected OS is Solaris 10 u10( 8/11) on all T3-x and T4-x platforms. The issue is fixed in patch 147790-01: SunOS 5.10: fmd patch.

See the workaround below:

# fmdump
TIME UUID SUNW-MSG-ID
Sep 16 08:38:19.5582 af875d87-433e-6bf7-cb53-c3d665e8cd09 SUN4V-8002-6E
Sep 16 08:40:47.8191 af875d87-433e-6bf7-cb53-c3d665e8cd09 FMD-8000-4M Repaired
Sep 16 08:40:47.8446 af875d87-433e-6bf7-cb53-c3d665e8cd09 FMD-8000-6U Resolved
#
# fmadm flush /SYS/MB
fmadm: flushed resource history for /SYS/MB
#
faultmgmtsp> fmadm repair /SYS/MB
faultmgmtsp> fmadm faulty
No faults found
faultmgmtsp>

Product
Sun Fire T1000 Server
Sun Fire T2000 Server
SPARC Enterprise T5120 Server
SPARC Enterprise T5220 Server
SPARC Enterprise T5140 Server
SPARC Enterprise T5140 Server

This document contains normalized content and is managed by the the Domain Lead
(s) of the respective domains. To notify content owners of a knowledge gap
contained in this document, and/or prior to updating this document, please
contact the domain engineers that are managing this document via the "Document
Feedback" alias(es) listed below:

Domain Lead: Dencho.Kojucharov@sun.com
VSP-SPARC-Normalization@sun.com

normalized, FMA, fault, error log, SC, fmadm, showfaults, T1000, T2000, T5x20, T5x40
Previously Published As
88049
Change History
Date: 2011-02-01
User Name: Dencho Kojucharov
Action: Updated
Comment: process comments, audited by Entry-Level SPARC Content lead
Date: 2009-11-18
User Name: Anthony Rulli
Action: Updated
Comment: currency check, audited by Anthony Rulli, Entry Level SPARC Content team

References

<NOTE:1483194.1> - Commands to run to fully clear ILOM/SP, faultmgmt shell, and FMA faults on the T3-x, T4-x , T5-x Servers
<NOTE:2216293.1> - Commands To Clear FMA faults on the T5-x, T7-x, S7-x Servers

Attachments

This solution has no attachment