Sun Fire 15K Server
 
Sun Fire E20K Server
 
Sun Fire E2900 Server
 
Sun Fire 3800 Server
 
Sun Fire E25K Server
 
Sun Fire 6800 Server
 
Sun Fire 4810 Server
 
Sun Fire V1280 Server
 
Sun Fire 12K Server
 
Sun Fire 4800 Server
 
Sun Fire E6900 Server
 
Sun Fire V445 Server
 
Sun Fire E4900 Server
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: Exx00
 
_Old GCS Categories>Sun Microsystems>Servers>Midrange V and Netra Servers
 
_Old GCS Categories>Sun Microsystems>Servers>High-End Servers
 
_Old GCS Categories>Sun Microsystems>Servers>Midrange Servers
 

Applies to:

Goal

Certain hardware failures can result in multiple components being identified as at fault. In addition FMA bugs exist which can cause innocent components to be disabled in error.

Solution

1. Confirm what FMA believes to be currently at fault from the output of fmadm faulty

If the fmadm faulty output looks like the following example the system is patched at a very old level where multiple bugs exist.

At this level memory fault diagnosis is still very reliable but CPU faults are very likely to be false.

In this example two CPU cores from the same socket were identified as a result of a single fault diagnosis UUID 6edb6157-d195-e1ba-eed2-a021b432b99f.  The single bank of memory including four DIMMs was identified as a result of a second diagnosis UUID 709b49ae-c2c9-ea93-93a2-ed44337adf5d.

Exceptions: The output of fmadm faulty will only contain faults on components currently presented to Solaris. There are a number of reasons why components might not be presented to Solaris.

• The component is CHS disabled and no longer presented to Solaris
• The component failed POST and is no longer presented to Solaris 
• An admin manually removed components from the domain/system config.

Example fmadm faulty output from Solaris 10 u5 or above:

In certain cases the FMA system might no longer be running, or components might have been removed from the config. To see recent faults it can be useful to look at the raw fltlog. When looking at the raw fault and error logs be aware that the logs will contain information for resources that might no longer be present in the system. The fmdump command can be used to view the raw fltlog in this example we see both the two faults and also can confirm see the DIMMs were identified first.

Step 2. Confirm exactly which error/errors resulted in each fault diagnosis

For each active fault now look for the detail of what errors led to the fault diagnosis. To do this there are two options, either run commands against the raw fltlog/errlog or use the files collected by explorer itself.

In this example we have explorer version 5.4+ so all the outputs we need have been collected.

From the above output we can see that both diagnosis's took place in the same second. With the CPU identified 165 nanoseconds after the DIMMs were identified as at fault. The chances of two distinct failures occurring on a single system in a single day is very low, to have them happen within the same second is almost impossible.

If the fmdump-eu_ outputs have not been collected to manually run the same commands do the following:

Step 3. Confirm if those errors were actually the first significant errors or are as a result of earlier significant errors.

Looking at the raw errors around the time of the failure we see a CE followed by two DUEs then the UCU which resulted in the CPU being disabled.

Looking in more detail at the errors we see the CE and DUE come from the same bank of DIMMs and all the AFAR/Syndromes indicate a clear DIMM fault. In this example we can be 100% confident that the subsequent CPU fault is false ( Bug ID: 6606049 () was hit in this case ).

Step 4. What to do if a misdiagnosis is suspected

Patching Solaris is the first thing to be done.

Open an SR and have the data reviewed

TSC Engineer reccomendation:

Recommend patching if a known bug has been identified as the cause of the misdiagnosis

re-enable the innocent components if required. Or, collaborate to the next level of support,  so that your customer can be added to an existing open CR, or a new CR can be logged if required.

Additional info in case of panic


From the core files FMA gets the ereports which occurred during the panic. These get re-played and standard faults get generated.

• The ereport time stamps are correct
• The fault time stamps reflect when the diagnosis was carried out

Known bugs:

• CPUs disabled due to memory UE errors
• <BUG 15424025>  USIV+ CPUs disabled due to a memory UE fault
  
  • Fix in Solaris 10 137111-02
• <BUG 15323597>  On Panther, on an L2 or L3 xxU, cpumem DE needs to check two UE caches
  
  • Fix in Solaris 10 123839-06
• <BUG 15328432>  UE and DUE and WDU cause processor to be offlined
  
  • Fix in Solaris 10 125369-11
• Sun Fire Systems Equipped With UltraSPARC IV+ Processor Modules Running Solaris 9 or Solaris 10 may Exhibit Unnecessary CPU Offlining and Solaris Panics <Document 1000495.1>
• <BUG 15296134>  sparc cpumem-diagnosis needs to implement rules 4A and 4B
  • Fix in Solaris 10 125369-02

Tips Tricks and Tools

Use fmdump to look at errors before or after a certain date

• Display errors after Dec 1st 2007 fmdump -t 12/01/07 -e errlog
• Display errors before Dec 1st 2007 fmdump -T 12/01/07 -e errlog

CPU and Memory errors can generate thousands of errors, when looking into these faults a summary of the errors is far more useful than trying to look at individual errors.

• Findfma is a command line summary script for fmdump -eV outputs.

The fmdump -V output captures the serial numbers of DIMMs identified. This is useful for confirming that the correct DIMMs have actually been replaced if a fault returns.

References

This solution has no attachment