Oracle ZFS Storage Appliance: System Panics Continuously with a 'PCIe error' after Rollback from 2013.1.x to 2011.1.x versions

Asset ID:	1-72-2082751.1
Update Date:	2017-06-30
Keywords:

Solution Type Problem Resolution Sure

Solution 2082751.1 : Oracle ZFS Storage Appliance: System Panics Continuously with a 'PCIe error' after Rollback from 2013.1.x to 2011.1.x versions

Applies to:

Oracle SuperCluster T5-8 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 to X4 [Release X2 to X4]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

A system was rolled back from 2013.1.4.9 to 2011.1.6.0 and upon rollback reboot the system went into a panic loop displaying :

NOTICE: unsupported 64-bit prefetch memory on pci-pci bridge [0/1/0]
NOTICE: unsupported 64-bit prefetch memory on pci-pci bridge [0/2/0]
NOTICE: unsupported 64-bit prefetch memory on pci-pci bridge [0/3/0]
NOTICE: hermon0: error during attach: UAR BAR mapping
NOTICE: hermon0: driver failed to attach: UAR BAR mapping

Followed by a panic with panic string: panic[cpu6]/thread=ffffff00b8ba5c40: pcieb-2: PCI(-X) Express Fatal Error. (0x40)

NOTE: This issue is likely to occur if a system running 2011.1.x is upgraded to 3.1.2.20 or newer ILOM code.

Please refer to KM doc: "Oracle ZFS Storage Appliance: Critical Security bug fixes for ZFSSA BIOS/ILOM (October 2015)" (Doc ID 2054991.1)

Changes

System was rolled back from a 2013.1.x release to a 2011.1.x release.

Cause

While the system was running on 2013.1.x the ILOM/BIOS had been upgraded to version 3.1.2.20.

The BIOS included in this version has the possibility to enable 64 bit PCI resource allocation, which seems to be enabled by default.

Older Solaris versions do not support 64 bit PCI resource allocation, and this is an issue with 2011.1.x versions as it is based on a Pre-solaris 11 OS.

Bug 15623214 - "pci_autoconfig: NOTICE: unsupported 64-bit prefetch memory on pci-p" was used to deal with the issue for 'base' Solaris systems, and this was incorporated in the Solaris development later than the base of the last 2011.1.x release.

2013.1.x releases are based on Solaris 11.1 and will support the 64 bit PCI resource allocation.

Solution

Enter the BIOS setup screen during a reboot (Control-E if you are viewing the console via serial-port or SSH to the ILOM),

Under the PCI menu, disable the "PCI 64-bit resource allocation" property

Go to the Exit menu and Save and Exit.

Now the system will boot up properly - unless there is an actual hardware issue on the PCI slot/card pointed out by the panic.

NOTE: The "PCI 64-bit resource allocation" property should be re-enabled when running 2013.x AK versions.

References

<NOTE:2054991.1> - Oracle ZFS Storage Appliance: Critical Security bug fixes for ZFSSA BIOS/ILOM (October 2015)
<NOTE:2011256.1> - Infiniband (IB) Interfaces Missing and Message "unsupported 64-bit prefetch memory on pci-pci bridge" in Solaris
<BUG:15623214> - SUNBT6927040 PCI_AUTOCONFIG: NOTICE: UNSUPPORTED 64-BIT PREFETCH MEMORY ON PCI-P
<BUG:17658302> - SOLARIS10U11 REPORT UNSUPPORTED 64BIT PREFETCH MEMORY AND DRIVER ISSUE ON CX2

Attachments

This solution has no attachment