"PCIE fault in downstream devices" or fault.io.intel.iio.pcie-fatal error encountered on Exadata

Asset ID:	1-72-1610302.1
Update Date:	2016-08-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1610302.1 : "PCIE fault in downstream devices" or fault.io.intel.iio.pcie-fatal error encountered on Exadata

Applies to:

Exalogic Elastic Cloud X4-2 Hardware - Version X4 and later
Exadata X3-2 Full Rack - Version All Versions and later
Exalogic Elastic Cloud X3-2 Quarter Rack - Version X3 and later
Exadata X3-2 Quarter Rack - Version All Versions and later
Exadata X3-2 Eighth Rack - Version All Versions and later
Linux x86-64

Symptoms

ILOM snapshot shows

Fatal:FIRST:PCIe6:port3c:PCIE:84::Completion Time-out Status

Fault class : fault.io.intel.iio.pcie-downstream-devices

Description : An Integrated I/O (II0) non-fatal error in downstream PCIE device has occurred.

Response : The service-required LED on the chassis will be illuminated.

Impact : System continues to run with degraded resources.

Cause

This is caused due to known BIOS unpublished bugs in 11.2.3.2.0 and 11.2.3.2.1 on Exadata X3-2 Compute Nodes.

Unpublished BUG 16885907 - ADVISORY NON-FATAL ERRORS ARE NOT PROPERLY MASKED ON X3-2 AND X3-2L

Solution

This issue is fixed at ILOM 3.1.2.12.c / BIOS 18.04.03.00 versions and later. (for compute node).

The solution is to upgrade Exadata Storage Software version to 11.2.3.3.0 or later.

As possible workaround, you can clear the fault on ILOM following below steps.

1. Connect to the ILOM cli

# ipmitool sunoem cli

2. Execute below commands.

-> start /SP/faultmgmt/shell
faultmgmtsp> prompt appears for Sun SP managed devices.
faultmgmtsp> fmadm faulty (to display active faults)
faultmgmtsp> fmadm acquit /SYS/MB/PCIE2
faultmgmtsp> exit

3. After the fault is cleared reboot the node and verify the issue did not return.

References

<BUG:16885907> - ADVISORY NON-FATAL ERRORS ARE NOT PROPERLY MASKED ON X3-2 AND X3-2L
<NOTE:1439904.1> - SPX86-8003-QH - IIO PCIE Non-Fatal Error in Downstream Devices.

Attachments

This solution has no attachment