Oracle ZFS Storage Appliance: 7420 and ZS3-4 DIMM mis-identification on fault (when running ILOM/BIOS 3.0.16 and higher)

Asset ID:	1-72-2286134.1
Update Date:	2017-11-15
Keywords:

Solution Type Problem Resolution Sure

Solution 2286134.1 : Oracle ZFS Storage Appliance: 7420 and ZS3-4 DIMM mis-identification on fault (when running ILOM/BIOS 3.0.16 and higher)

Applies to:

Oracle ZFS Storage ZS3-4 - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

All faulty DIMM reports on X4470(-M2)-based ZFSSA with ILOM 3.0.16 and later indicate an incorrect serial number.

The identification LEDs are lit for the wrong DIMMs.

The truly faulty DIMMs are on the same memory riser.

Cause

Bug 24343557 (DIMM mapping wrong on X4470-M2-based appliances leading to replacing wrong DIMMs)

For NAS TSC Engineers, see - https://stbeehive.oracle.com/content/dav/st/AmberRoadSupport/Documents/7420%20ZS3-4%20DIMM%20Mapping%20June2017.pdf

Issue is actually tied to the version of ILOM/BIOS and can occur on both X4470 and X4470 M2 based appliances (7420, 7420+, 7420M2 and ZS3-4)

- Occurs when running ILOM/BIOS 3.0.16 and higher
- Order of the SMBIOS DIMM records changed with this version as per bug 15704775
- DIMM record order was 0,1,2,3,4,5,6,7 and now is 2,3,0,1,6,7,4,5
- Regardless of ILOM version, Service Processor diagnosed DIMM Faults point to correct DIMM
- ILOM record points to correct DIMM location and the correct Serial Number
- SP turns on the correct DIMM LED fault indicator
- SMBIOS information is also correct, however with newer ILOM, FMTOPO maps it incorrectly due to new ordering
- ASR record includes the correct location, but as it gets the Serial Number from FMTOPO it points to the wrong Serial Number
        - The FMA fault will persist until the incorrect Serial Number DIMM is replaced.
        - The DIMM will remain in use, but chassis fault will be set and needs to be manually corrected (acquitted)
        - Replacing DIMMs based on FMA Serial Number information will result in replacement of the wrong DIMM

Solution

DO NOT use fru-serial (serial number) to determine DIMM to replace - USE the LOCATION.

The location information correctly reflects the faulted DIMM.

A workaround is available to acquit the residual fault (on the 'incorrect' DIMM) after replacing the correct faulted DIMM.

Attachments

This solution has no attachment