Failed to save core dump due to "panic: entering debugger (continue to save dump)"

Asset ID:	1-72-1945858.1
Update Date:	2018-03-01
Keywords:

Solution Type Problem Resolution Sure

Solution 1945858.1 : Failed to save core dump due to "panic: entering debugger (continue to save dump)"

Applies to:

Sun SPARC Enterprise M9000-64 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M9000-32 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M4000 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M5000 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise M3000 Server - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

If the kernel variables "halt_on_panic=1" and "obpdebug=1" are set in the /etc/system and no action for 30min after panic, Solaris would fail to save core dump as follows.

Oct 28 11:58:52 JST 2014      m4000-syd04-c console login:
Oct 28 11:59:09 JST 2014
panic[cpu17]/thread=2a1007efca0: System Panel Driver: Emergency panic request detected!
Oct 28 11:59:09 JST 2014
Oct 28 11:59:09 JST 2014      000002a100857df0 oplpanel:panel_intr+a0 (6001056c918, 10, 7bf95c00, 16, 0, 1895000)
Oct 28 11:59:09 JST 2014        %l0-3: 000002a100857da8 000002a100857dd0 0000000000000037 00000000018e9000
Oct 28 11:59:09 JST 2014        %l4-7: 0000000000000000 0000000000000001 0000000070069c00 0000000000000011
Oct 28 11:59:09 JST 2014      000002a100857ea0 pcicmu:pcmu_intr_wrapper+54 (300028b3428, 0, 30003921d68, 3000759e000, 8000, 1)
Oct 28 11:59:09 JST 2014        %l0-3: 0000000000000000 0000000000000001 0000060010755d50 0000000000000000
Oct 28 11:59:09 JST 2014        %l4-7: 0000000000000001 00000000018e928c 000006001056c918 000000007bf95cec
Oct 28 11:59:09 JST 2014      000002a100857f50 unix:current_thread+164 (1, 1, 1, 1000, 101000001000101, 100)
Oct 28 11:59:09 JST 2014        %l0-3: 00000000010076c8 000002a1007eefe1 000000000000000f 0000000070024540
Oct 28 11:59:10 JST 2014        %l4-7: 0000000000000003 0000000000000018 0000000000000000 000002a1007ef890
Oct 28 11:59:10 JST 2014      000002a1007ef930 unix:cpu_halt+13c (16, 3000759e000, 0, 18c1698, 16, 1)
Oct 28 11:59:10 JST 2014        %l0-3: 0000000000000009 0000000000000014 0000000000000000 000006001075efd8
Oct 28 11:59:10 JST 2014        %l4-7: 0000000001000000 0000000000000002 000000000128e400 0000000000000000
Oct 28 11:59:10 JST 2014      000002a1007ef9e0 unix:idle+128 (1832400, 0, 3000759e000, ffffffffffffffff, a, 1831000)
Oct 28 11:59:10 JST 2014        %l0-3: 000006001075efd8 000000000000001b 0000000000000000 ffffffffffffffff
Oct 28 11:59:10 JST 2014        %l4-7: 00000000000008fa 0000000001942800 000003000759e178 0000000001040334
Oct 28 11:59:10 JST 2014
Oct 28 11:59:10 JST 2014      panic: entering debugger (continue to save dump)
Oct 28 11:59:10 JST 2014      Type 'go' to resume
Oct 28 11:59:12 JST 2014      {11} ok
Oct 28 11:59:12 JST 2014      {11} ok
Oct 28 12:29:16 JST 2014      {11} ok
Oct 28 12:29:16 JST 2014      ERROR: Externally Initiated Reset has occurred. <<-- XSCF executes the system reset (XIR) due to timeout
Oct 28 12:29:18 JST 2014      Resetting...
Oct 28 12:29:51 JST 2014      POST Sequence 01 CPU Check
Oct 28 12:29:52 JST 2014      POST Sequence 02 Banner
Oct 28 12:29:52 JST 2014      LSB#00 (XSB#00-0): POST 2.17.0 (2011/11/17 10:29)
Oct 28 12:29:53 JST 2014      POST Sequence 03 Fatal Check
Oct 28 12:29:53 JST 2014      POST Sequence 04 CPU Register
Oct 28 12:29:53 JST 2014      POST Sequence 05 STICK
Oct 28 12:29:54 JST 2014      POST Sequence 06 MMU
Oct 28 12:29:55 JST 2014      POST Sequence 07 Memory Initialize
Oct 28 12:29:55 JST 2014      POST Sequence 08 Memory
Oct 28 12:30:06 JST 2014      POST Sequence 09 Raw UE In Cache
Oct 28 12:30:08 JST 2014      POST Sequence 0A Floating Point Unit
Oct 28 12:30:09 JST 2014      POST Sequence 0B SC
Oct 28 12:30:09 JST 2014      POST Sequence 0C Cacheable Instruction
Oct 28 12:30:10 JST 2014      POST Sequence 0D Softint
Oct 28 12:30:12 JST 2014      POST Sequence 0E CPU Cross Call
Oct 28 12:30:12 JST 2014      POST Sequence 0F CMU-CH
Oct 28 12:30:13 JST 2014      POST Sequence 10 PCI-CH
Oct 28 12:30:19 JST 2014      POST Sequence 11 Master Device
Oct 28 12:30:20 JST 2014      POST Sequence 12 DSCP
Oct 28 12:30:21 JST 2014      POST Sequence 13 SC Check Before STICK Diag
Oct 28 12:30:21 JST 2014      POST Sequence 14 STICK Stop
Oct 28 12:30:22 JST 2014      POST Sequence 15 STICK Start
Oct 28 12:30:22 JST 2014      POST Sequence 16 Error CPU Check
Oct 28 12:30:22 JST 2014      POST Sequence 17 System Configuration
Oct 28 12:30:23 JST 2014      POST Sequence 18 System Status Check
Oct 28 12:30:24 JST 2014      POST Sequence 19 System Status Check After Sync
Oct 28 12:30:24 JST 2014      POST Sequence 1A OpenBoot Start...
Oct 28 12:30:25 JST 2014      POST Sequence Complete.
Oct 28 12:30:28 JST 2014
Oct 28 12:30:28 JST 2014      SPARC Enterprise M4000 Server, using Domain console
Oct 28 12:30:28 JST 2014      Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights reserved.
Oct 28 12:30:28 JST 2014      Copyright (c) 2012, Oracle and/or its affiliates and Fujitsu Limited. All rights reserved.
Oct 28 12:30:28 JST 2014      OpenBoot 4.33.5.d, 8192 MB memory installed, Serial #70948389.
Oct 28 12:30:28 JST 2014      Ethernet address 0:14:4f:3a:96:25, Host ID: 843a9625.

Changes

Cause

If the kernel variables "halt_on_panic=1" and "obpdebug=1" are set in the /etc/system, the core data will not be dumped until type "go".

Oct 28 11:45:20 JST 2014        %l4-7: 0000000001000000 0000000000000002 000000000128e400 0000000000000000
Oct 28 11:45:20 JST 2014      000002a1007ef9e0 unix:idle+128 (1832400, 0, 3000755e000, ffffffffffffffff, a, 1831000)
Oct 28 11:45:20 JST 2014        %l0-3: 0000060010746fd8 000000000000001b 0000000000000000 ffffffffffffffff
Oct 28 11:45:20 JST 2014        %l4-7: 0000060010746fd8 ffffffffffffffff 00000000018c1698 0000000001040334
Oct 28 11:45:20 JST 2014
Oct 28 11:45:20 JST 2014      panic: entering debugger (continue to save dump)
Oct 28 11:45:20 JST 2014      Type 'go' to resume
Oct 28 11:45:23 JST 2014      {11} ok
{11} ok
{11} ok
{11} ok go
syncing file systems... 2 done
dumping to /dev/dsk/c0t0d0s1, offset 108396544, content: kernel
0:03 100% done
100% done: 74535 pages dumped, dump succeeded
Program terminated
{11} ok

If no action was taken for 30min after panic, the XSCF would execute the system reset (XIR) due to timeout.

Please refer to the document for more details about the XSCF settings.

>> Sun SPARC(R) Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers: How to deal with a hung or unresponsive domain ? (Doc ID 1020078.1)

Solution

If the "halt_on_panic=1" is set in the /etc/system without "obpdebug=1", the core data will be dumped as follows.

Oct 28 15:49:57 JST 2014      000002a1007ef930 unix:cpu_halt+13c (16, 3000759e000, 0, 18c1698, 16, 1)
Oct 28 15:49:57 JST 2014        %l0-3: 0000000000000009 0000000000000014 0000000000000000 0000060010750fd8
Oct 28 15:49:57 JST 2014        %l4-7: 0000000001000000 0000000000000002 000000000128e400 0000000000000000
Oct 28 15:49:57 JST 2014      000002a1007ef9e0 unix:idle+128 (1832400, 0, 3000759e000, ffffffffffffffff, a, 1831000)
Oct 28 15:49:57 JST 2014        %l0-3: 0000060010750fd8 000000000000001b 0000000000000000 ffffffffffffffff
Oct 28 15:49:57 JST 2014        %l4-7: 0000060010750fd8 ffffffffffffffff 00000000018c1698 0000000001040334
Oct 28 15:49:57 JST 2014
Oct 28 15:50:00 JST 2014      syncing file systems... 2 2 done
Oct 28 15:50:01 JST 2014      dumping to /dev/dsk/c0t0d0s1, offset 108396544, content: kernel
0:03 62% done JST 2014
0:04 100% done JST 2014      63% done
100% done: 96730 pages dumped, dump succeeded
Oct 28 15:50:14 JST 2014      Program terminated
Oct 28 15:50:15 JST 2014      {11} ok

Once you boot the system, the core files will be saved to /var/crash.

Oct 28 17:47:32 m4000-syd04-c savecore: Decompress the crash dump with
Oct 28 17:47:32 m4000-syd04-c 'savecore -vf /var/crash/m4000-syd04-c/vmdump.2'
Oct 28 17:47:32 m4000-syd04-c savecore: System dump time: Tue Oct 28 17:43:56 2014
Oct 28 17:47:32 m4000-syd04-c savecore: saving system crash dump in /var/crash/m4000-syd04-c/{unix,vmcore}.2

References

<NOTE:1003268.1> - How to Prevent Automatic System Reboot Following System Panic
<NOTE:1020078.1> - Sun SPARC(R) Enterprise M3000/M4000/M5000/M8000/M9000 (OPL) Servers: How to deal with a hung or unresponsive domain ?

Attachments

This solution has no attachment