SPARC T4-2
 
Oracle Virtual Compute Appliance X4-2 Hardware
 
Sun Blade 6000 System
 
Sun Fire X2200 M2 Server
 
SPARC M5-32
 
SPARC T3-2
 
SPARC T3-4
 
Sun Fire X4140 Server
 
SPARC M8-8
 
Sun SPARC Enterprise T5240 Server
 
SPARC T5-2
 
SPARC T3-1
 
SPARC M7-8
 
Exadata X3-2 Hardware
 
Sun Fire X4170 M2 Server
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: M7
 
_Old GCS Categories>Sun Microsystems>Servers>CMT Servers
 
_Old GCS Categories>Sun Microsystems>Servers>x64 Servers
 
_Old GCS Categories>Sun Microsystems>Servers>Blade Servers
 

In this Document

Goal

Solution

Applies to:

Goal

This document describes how to use the Fault Management Shell on ILOM 3.x and later platforms.

Caution - The purpose of the Oracle ILOM Fault Management Shell is to help Oracle Services personnel diagnose system problems. Customers should not launch this shell or run fault management commands in the shell unless requested to do so by Oracle Services.

The following fault management terms are defined :

Note : by using /X to navigate through the ILOM targets, the system will automatically drive you to /SP or to /CMM accordingly.  Example from a T3-4 system :

Solution

How to start/stop the Fault Management Shell.

The Fault Management shell is launched as a separate shell through the Oracle ILOM CLI. Only fault management commands can be run from this shell. To run standard Oracle ILOM commands, you must first exit the Fault Management shell.

To launch the shell, enter the following command when logged into the command line interface of the system's Oracle ILOM service processor:

The Fault Management shell includes the following commands.

   
- To exit the shell, enter the following command from the prompt:

How to track usage of the Fault Management Shell.

An audit log will is saved to the SP event log at: /X/logs/event/

Example :

Fault Management Shell Command Reference

fmadm - Fault Management Administration Tool

The fmadm utility can be used by administrators and service personnel to view and modify system fault management configuration parameters maintained by ILOM.
Use fmadm to:

• View the list of system components that have been diagnosed as faulty.
• Perform administrative tasks related to these entities.

Syntax
fmadm [subcommand [arguments]]

Subcommands
The fmadm utility accepts the following subcommands. Some of the subcommands accept or require additional options and operands

Note : when fault exists, 'fmadm faulty' equivalent information is available from spsh.

Ex :

fmdump - Fault Management Log Viewer

The fmdump utility can be used to display the contents of any of the log files associated with Oracle ILOM. Oracle ILOM receives telemetry information relating to problems detected by the system software, diagnoses these problems, and initiates proactive self-healing activities such as disabling faulty components. Oracle ILOM maintains two sets of log files for use by administrators and service personnel:

• error log  :   A log which records error telemetry; the symptoms of problems detected by the system.
• fault log   :  A log which records fault diagnosis information; the problems possibly related to the symptoms.

By default, fmdump displays the contents of the fault log, which records the result of each diagnosis made by the fault manager or one of its component modules.
Here is an example of a default fmdump display:

Each problem recorded in the fault log is identified by:

• The time of its diagnosis.
• A Universal Unique Identifier (UUID) that can be used to uniquely identify a particular problem across any set of systems.
• A message identifier that can be used to access a corresponding knowledge article located on Oracle's support web site.
• If a problem requires action by a human administrator or service technician or affects system behavior, ILOM also issues a human-readable message to its Event Log. This message provides a summary of the problem and a reference to the knowledge article on the Oracle web site.

You can use the -v and -V options to expand the display from a single-line summary to increased levels of detail for each event recorded in the log. The -u option can be used to filter the output by selecting only those events that match the specified uuid.

Syntax
fmdump [options [argument]]

Options
The following options are supported:

Example
This example dumps the fault log for the designated FRU UUID.

fmstat - Statistical Module Report Generator

Syntax
fmstat

The fmstat utility can be used by administrators and service personnel to report statistics associated with the Oracle ILOM Fault Manager and its associated set of modules. The Fault Manager runs in the background on each Oracle ILOM system. It receives telemetry information relating to problems detected by the system software, diagnoses these problems, and initiates proactive self-healing activities such as disabling faulty components.
You can use fmstat to view statistics for diagnosis engines that are currently participating in fault management.

The fmstat utility reports the following statistics for each of the diagnosis engines:

• engine    The name of the diagnosis engine. The engines execute rules for the fault diagnosis daemon based on ereport input. Oracle ILOM Fault Management engines include:
• repair - Rule that indicates a fault should be considered repaired if a specified ereport is logged. For example, the fault "fault.chassis.power.inadequate@/sys" would be considered repaired if "ereport.chassis.boot.power-off-requested@/sys" was logged.
• hysteresis - Rule to diagnose a fault if ereport A (initiation) is logged and ereport B (cancelation) is not logged within some specified time afterwards. For example, ereport A is "ereport.fan.speed-low-asserted" and ereport B is "ereport.fan.speed-low-deasserted". The time limit between the initiation/cancelation can be no greater than 10 seconds.
• SERD - Soft Error Rate Discrimination (SERD) is used in tracking multiple occurences of an ereport. If more than N ereports show up within time period T, the fault is diagnosed. For example, if too many correctable memory error ereports are logged within a specific time frame, a DIMM fault will be diagnosed.
• simple - Rule to allow one ereport to result in the diagnosis of multiple faults. For example, an ereport for an uncorrectable memory error can be diatnosed to the faults for two DIMMs in a DIMM pair.
• status    The status of the engine, either uninit, empty, enqueued, busy, or exiting.
• evts_in    The number of events received by the engine as relevant to a diagnosis.
• evts_out    The number of events sent by the engine.
• errors    The number of errors detected by the engine.

Example

etcd - Fault Event Injection Utility

The etcd (error telemetry collection daemon) utility can be used by administrators and service personnel to inject events in order to test the ILOM Fault Manager. The utility allows either ereports or sensor events with an offset to be entered.

Note : Starting from ILOM 3.2.1, etcd is only available from the Service mode. No longer from the Fault Management Shell.

Syntax
etcd [options [argument]]

Options
The following options are supported:

Example
This example injects an ereport into the Fault Management error log.

For ILOM > 3.2.1

How to identify an error injected using etcd

Usage of etcd is tracked via an entry in the SP event log.

Starting with 3.2.1, since etcd is available from Service mode, there is no more entry in the event logs but the 'fmdump -eV' would not report any detail about the detector. For example (fmdump -eV) :

This solution has no attachment