M4000 Domain Panics Reporting Fatal Error Has Occured in: PCIe fabric.(0x1)(0x45) and subsequent reboots Fails with Same Panic

Asset ID:	1-72-1993928.1
Update Date:	2018-05-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1993928.1 : M4000 Domain Panics Reporting Fatal Error Has Occured in: PCIe fabric.(0x1)(0x45) and subsequent reboots Fails with Same Panic

Applies to:

Sun SPARC Enterprise M9000-64 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M9000-32 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M3000 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M4000 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M5000 Server - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

1.System panic with following panic string

Fatal error has occured in: PCIe fabric.(0x1)(0x45)

2. System panics with the same panic string while booting failsafe also.

3. XSCFU "showstatus" command does not shows any error

4. XSCFU "showlogs error"command shows only the panic

Date: Feb 04 05:24:43 ICT 2015 Code: 60000000-ffffffff-0109001500000000
Status: Warning Occurred: Feb 04 05:20:23.316 ICT 2015
FRU: /UNSPECIFIED
Msg: XSCF command: System status change (OS panic) (DID#00, path: 00)
Diagnostic Code:
00000000 00000000 00000000
00002140 01000000 00000000 00000000
00000000 00000000 00000000 00000000
UUID: 10bc0258-70ae-4ce5-994e-59602402cd8a MSG-ID: SCF-8005-PX

5. Solaris FMA is not available since the domain will not boot, XSCFU fma shows no errors.

Troubleshooting Steps

Following are the complete panic messages

Feb 04 16:36:43 ICT 2015 panic[cpu5]/thread=3000a1c8c40: Fatal error has occured in: PCIe fabric.(0x1)(0x45)
Feb 04 16:36:43 ICT 2015
Feb 04 16:36:43 ICT 2015 000002a1008e3bc0 px:px_err_panic+1ac (1994000, 1380800, 45, 2a1008e3c70, 1, 0)
Feb 04 16:36:43 ICT 2015 %l0-3: 0000000000000001 0000000001994000 0000000000000000 0000000000000001
Feb 04 16:36:43 ICT 2015 %l4-7: 0000000000000000 00000000018bb000 0000000000000001 0000000000000000
Feb 04 16:36:43 ICT 2015 000002a1008e3cd0 px:px_err_fabric_intr+1c0 (3000289c940, 1, 1996000, 1, 45, 400)
Feb 04 16:36:43 ICT 2015 %l0-3: 0000000000000000 0000000000000054 0000000001996398 0000000001996000
Feb 04 16:36:43 ICT 2015 %l4-7: 0000000001996390 0000000001996000 0000000000000001 000006004207ba40
Feb 04 16:36:43 ICT 2015 000002a1008e3e40 px:px_msiq_intr+1e8 (600420815a0, 300052a4250, 13709c0, 0, 1, 300052b2de8)
Feb 04 16:36:43 ICT 2015 %l0-3: 000006004210bda0 00000300052b1210 00000300052a4250 0000000000000000
Feb 04 16:36:43 ICT 2015 %l4-7: 0000000000000000 0000060045604000 000002a1008e3f40 0000000000000033
Feb 04 16:36:43 ICT 2015 000002a1008e3f50 unix:current_thread+164 (6004739a940, 29ebfcd2408, 1400, 1, 60047532af0, 0)
Feb 04 16:36:44 ICT 2015 %l0-3: 00000000010076c8 000002a104998471 000000000000000e 0000000070024180
Feb 04 16:36:44 ICT 2015 %l4-7: 0200000000000000 0000000000000000 0000000000000000 000002a104998d20
Feb 04 16:36:44 ICT 2015 000002a104998dc0 bge:bge_chip_start+1f98 (29ebfcd2000, 29ebfcd2000, 60047396000, 2, 60047396100, 200)
Feb 04 16:36:44 ICT 2015 %l0-3: 0000000000001648 00000000f0000000 0000029ebfcd2000 0000000000001400
Feb 04 16:36:44 ICT 2015 %l4-7: 0000000000000000 ffffffffffffffff 0000029ebfcd2000 0000000000003c00
Feb 04 16:36:44 ICT 2015 000002a104998ea0 bge:bge_m_start+180 (60047396000, 2ed0, 3000, 2c00, 600476eddd8, b1f7)
Feb 04 16:36:44 ICT 2015 %l0-3: 00000300052a0488 0000000000000000 0000000000000000 000000007b30c5e8
Feb 04 16:36:44 ICT 2015 %l4-7: 0000000000002ed8 0000000000002ed8 0000000000002ed0 000000007b30c400
Feb 04 16:36:44 ICT 2015 000002a104998f50 mac:mac_start+34 (600476eddd8, 7b2f87e8, 7017c3f0, 0, 0, 1)
Feb 04 16:36:44 ICT 2015 %l0-3: 00000300052a0488 0000000000000000 0000000000000000 00000600476eddd8
Feb 04 16:36:44 ICT 2015 %l4-7: 000000007008bfb0 00000000700015f8 000002a104998ef0 0000000070001400
Feb 04 16:36:44 ICT 2015 000002a104999000 dls:dls_vlan_hold+244 (2a104999350, 2a104999288, 1, 0, 600476efd68, 2a1049990b8)
Feb 04 16:36:45 ICT 2015 %l0-3: 00000000019d9bc8 0000000000000000 0000000000000000 0000000000000000
Feb 04 16:36:45 ICT 2015 %l4-7: 0000000000000000 000002a1049990d0 0000000000000000 00000000019d9bd8
Feb 04 16:36:45 ICT 2015 000002a1049991d0 dls:dls_open+c (2a104999350, 2a104999348, 300052a0488, 0, 600476bad00, 600476f3dc0)
Feb 04 16:36:45 ICT 2015 %l0-3: 0000000000000001 0000000000002006 0000000000000000 0000000000002000
Feb 04 16:36:47 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:47 ICT 2015 device-path="/pci@0,600000" ] primary=1 pcie_adv_rp_status=54
Feb 04 16:36:47 ICT 2015 pcie_adv_rp_command=0 pcie_adv_rp_ce_src_id=0 pcie_adv_rp_ue_src_id=400
Feb 04 16:36:47 ICT 2015
Feb 04 16:36:47 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:47 ICT 2015 device-path="/pci@0,600000" ] scan_bdf=400 scan_addr=0 intr_src=1 remainder=5
Feb 04 16:36:47 ICT 2015 severity=1
Feb 04 16:36:47 ICT 2015
Feb 04 16:36:47 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:48 ICT 2015 device-path="/pci@0,600000/pci@0" ] bdf=200 device_id=8532 vendor_id=10b5
Feb 04 16:36:48 ICT 2015 rev_id=bc dev_type=50 pcie_off=68 pcix_off=0 aer_off=fb4 ecc_ver=0 pci_status=
Feb 04 16:36:48 ICT 2015 10 pci_command=147 pci_bdg_sec_status=4000 pci_bdg_ctrl=3 pcie_status=0
Feb 04 16:36:48 ICT 2015 pcie_command=2f pcie_dev_cap=640001 pcie_adv_ctl=1ff pcie_ue_status=0
Feb 04 16:36:48 ICT 2015 pcie_ue_mask=0 pcie_ue_sev=62011 pcie_ue_hdr0=45008001 pcie_ue_hdr1=1001103
Feb 04 16:36:48 ICT 2015 pcie_ue_hdr2=3000004 pcie_ue_hdr3=0 pcie_ce_status=0 pcie_ce_mask=1 remainder=
Feb 04 16:36:48 ICT 2015 4 severity=1
Feb 04 16:36:48 ICT 2015
Feb 04 16:36:48 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:48 ICT 2015 device-path="/pci@0,600000/pci@0/pci@8" ] bdf=340 device_id=8532 vendor_id=
Feb 04 16:36:48 ICT 2015 10b5 rev_id=bc dev_type=60 pcie_off=68 pcix_off=0 aer_off=fb4 ecc_ver=0
Feb 04 16:36:48 ICT 2015 pci_status=10 pci_command=547 pci_bdg_sec_status=4000 pci_bdg_ctrl=3
Feb 04 16:36:48 ICT 2015 pcie_status=0 pcie_command=2f pcie_dev_cap=1 pcie_adv_ctl=1ff pcie_ue_status=0
Feb 04 16:36:48 ICT 2015 pcie_ue_mask=0 pcie_ue_sev=62011 pcie_ue_hdr0=45008001 pcie_ue_hdr1=1001403
Feb 04 16:36:49 ICT 2015 pcie_ue_hdr2=4080004 pcie_ue_hdr3=0 pcie_ce_status=0 pcie_ce_mask=1 remainder=
Feb 04 16:36:49 ICT 2015 3 severity=1
Feb 04 16:36:49 ICT 2015
Feb 04 16:36:49 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:49 ICT 2015 device-path="/pci@0,600000/pci@0/pci@8/pci@0" ] bdf=400 device_id=125
Feb 04 16:36:49 ICT 2015 vendor_id=1033 rev_id=8 dev_type=70 pcie_off=40 pcix_off=54 aer_off=100
Feb 04 16:36:49 ICT 2015 ecc_ver=0 pci_status=4010 pci_command=147 pci_bdg_sec_status=c420
Feb 04 16:36:49 ICT 2015 pci_bdg_ctrl=23 pcix_bdg_status=400 pcix_bdg_sec_status=83 pcie_status=4
Feb 04 16:36:49 ICT 2015 pcie_command=202f pcie_dev_cap=640001 pcie_adv_ctl=1e0 pcie_ue_status=0
Feb 04 16:36:49 ICT 2015 pcie_ue_mask=0 pcie_ue_sev=62010 pcie_ue_hdr0=4008001 pcie_ue_hdr1=1000703
Feb 04 16:36:49 ICT 2015 pcie_ue_hdr2=4020000 pcie_ue_hdr3=0 pcie_ce_status=0 pcie_ce_mask=0
Feb 04 16:36:49 ICT 2015 pcie_sue_adv_ctl=c pcie_sue_status=1200 pcie_sue_mask=0 pcie_sue_sev=1340
Feb 04 16:36:49 ICT 2015 pcie_sue_hdr0=510b0 pcie_sue_hdr1=e1 pcie_sue_hdr2=c0990800 pcie_sue_hdr3=0
Feb 04 16:36:49 ICT 2015 pcie_sue_tgt_trans=0 pcie_sue_tgt_addr=0 pcie_sue_tgt_bdf=ffff remainder=2
Feb 04 16:36:49 ICT 2015 severity=45
Feb 04 16:36:49 ICT 2015
Feb 04 16:36:50 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:50 ICT 2015 device-path="/pci@0,600000/pci@0/pci@8/pci@0/scsi@1" ] bdf=508 device_id=50 >>>>>>>>> IOU lSB0 scsi controller for disks, tape and dvd.
Feb 04 16:36:50 ICT 2015 vendor_id=1000 rev_id=2 dev_type=101 pcie_off=0 pcix_off=68 aer_off=0 ecc_ver=
Feb 04 16:36:50 ICT 2015 1000 pci_status=c230 pci_command=157 pcix_status=13430508 pcix_command=1060
Feb 04 16:36:50 ICT 2015 pcix_ecc_control_0=0 pcix_ecc_status_0=0 pcix_ecc_fst_addr_0=0
Feb 04 16:36:50 ICT 2015 pcix_ecc_sec_addr_0=0 pcix_ecc_attr_0=0 remainder=1 severity=40
Feb 04 16:36:50 ICT 2015
Feb 04 16:36:50 ICT 2015 ereport.io.pci.fabric ena=26b0ccb15001401 detector=[ version=0 scheme="dev"
Feb 04 16:36:50 ICT 2015 device-path="/pci@0,600000/pci@0/pci@8/pci@0/network@2" ] bdf=510 device_id=>>>>>>>>>>>>>>>>>>>IOU LSB0 bge0
Feb 04 16:36:50 ICT 2015 1648 vendor_id=14e4 rev_id=10 dev_type=101 pcie_off=0 pcix_off=40 aer_off=0
Feb 04 16:36:50 ICT 2015 ecc_ver=0 pci_status=22b0 pci_command=146 pcix_status=4430510 pcix_command=0
Feb 04 16:36:50 ICT 2015 remainder=0 severity=40
Feb 04 16:36:50 ICT 2015
Feb 04 16:36:50 ICT 2015 skipping system dump - no dump device configured
Feb 04 16:36:50 ICT 2015 rebooting...
Feb 04 16:36:50 ICT 2015 Resetting...

Cause

Looking at the device paths called out in the panic and comparing them to the device path documents should yield a reasonable action plan:

Sun SPARC[TM] Enterprise M4000 and M5000 Server Device Paths (Doc ID 1002807.1)

Sun SPARC(R) Enterprise M8000 and M9000 Device Paths (Doc ID 1004116.1)

https://mos-cores.us.oracle.com/cgi-bin/opltools/oplTools.cgi

Extras >> OPL IO Layout is also very usefule for path decoding.

In this case an M4000 with the following devices in the panic string indicate a faulty IOU,

/pci@0,600000/pci@0/pci@8/pci@0/network@2 = IOU LSB0 bge0
/pci@0,600000/pci@0/pci@8/pci@0/scsi@1 = IOU lSB0 scsi controller for disks, tape and dvd.

If one or more of the mentioned devices is a PCI Card, having a Field Engineer remove cards can be used to narrow the fault down to a single FRU.

So in this case the IOU#0 is the suspected FRU for replacement since the panic points to only on borad devices.

Solution

Gather a snapshot and open a Service Request for resolution.

References

<NOTE:1004116.1> - Sun SPARC(R) Enterprise M8000 and M9000 Device Paths
<NOTE:1002807.1> - Sun SPARC[TM] Enterprise M4000 and M5000 Server Device Paths

Attachments

This solution has no attachment