Sun Fire[TM] 12K/15K/E20K/E25K: Reboot of one domain, 2 others dstop with “Timeout on head of CI queue”

Asset ID:	1-72-1010705.1
Update Date:	2016-07-21
Keywords:

Solution Type Problem Resolution Sure

Solution 1010705.1 : Sun Fire[TM] 12K/15K/E20K/E25K: Reboot of one domain, 2 others dstop with “Timeout on head of CI queue”

Applies to:

Sun Fire E25K Server - Version Not Applicable to Not Applicable [Release N/A]
Sun Fire 15K Server - Version Not Applicable to Not Applicable [Release N/A]
Sun Fire 12K Server - Version Not Applicable to Not Applicable [Release N/A]
Sun Fire E20K Server - Version Not Applicable to Not Applicable [Release N/A]
Sun SPARC Sun OS
All Platforms

Symptoms

A platform has been upgraded from sms 1.3 to sms 1.4.1. After the sms 1.3 upgrade to sms 1.4.1, the domains needed to be rebooted to take the new firmware in effect.

One domain has been shutdown with setkeyswitch and within a short time frame 2 other domains crashed with a dstop. The dstops look like this:

redxl> dumpf load dsmd.dstop.040916.1721.32
Created Thu Sep 16 17:21:34 2004

By hpost v. 1.4.1 Generic 117371-04 Jul 21 2004 12:53:56 executing as pid=6465
On ssc name: ssc0001.
Domain = 11=L = su01234 Platform = ssc0100 ChHostID = 5014936601194
Boards in dump: master SC CPs/CSBs[1:0]: 3
EXB[17:0]: 20000
Slot0[17:0]: 20000
Slot1[17:0]: 20000
-D option, -d
"DSMD DomainStop Dump"
0 errors occurred while creating this dump.

redxl> wfail
All master SDIs in this dump indicating valid error info [20000]
indicate the first error was Dstop for all of EXB EX17.
SDI EX17/S0 Master_Stop_Status0[31:0] = D00400CF
MStop0[3:0]: All SDI logic is DStopped + Recordstopped.
SDI EX17/S0 Dstop0[31:0] = 3001A000
Dstop0[16]: D DARB texp requests all Dstop (M)
Dstop0[28]: D Slot0 asserted Error, enabled to cause Dstop (M)
Dstop0[29]: D 1E Slot1 asserted Error, enabled to cause Dstop (M)
EPLD IO17 Err1_Dom1: Mask= B0 Err= 40 1stErr= 40
Err1[6]: 1E+ Error reported by BBC0
BBC IO17/BB0 Device_Err_Stat[31:0] = 80008100
DevErr[ 8]: 1E Port 0 Safari device asserted error
PCI IOC IO17/P0 Safari_Err_Log[63:0] = 80000000 00000200
ErrLog[ 9]: ErrOut Timeout on head of CI queue
ErrLog[63]: Error Out asserted (S_ERROR_L pin)
FAIL Port IO17/P0: Dstop detected by BBC IO17/BB0.
Primary service FRU is Slot IO17.
DARB C0: enabled ports (expanders) [17:0]: 28F57
DARB C0: other darb req Dstop+Rstop for exps[17:0]: 20000
DARB C1: enabled ports (expanders) [17:0]: 28F57
DARB C1: other darb req Dstop+Rstop for exps[17:0]: 20000

Cause

It turns out, an out-of-date, sms-backup file was restored during the sms upgrade. Causing conflicting information in the .pcd files and in the platforms registers. So a part of the PCD, the Platform Configuration Database, files in $SMSVAR/SMS/.pcd/ contain information inconsistent with the current domain layout. This will cause problems during setkeyswitch operations.

Solution

Setkeyswitch off and on the domains, this will rewrite the PCD information.
This is preferably done when the applications have been shutdown on the domains.

Internal Comments
Reboot of one domain, 2 others dstop with “Timeout on head of CI queue”

This is related to CASM bug 15416261 submitted Aug 13 2007.
If this timeout is received check the casm allocations to make sure they are ok.

Each slot that has a system board should also have a casm assigned Please escalate any cases that fit this bug

reboot, domain, dstop, keyswitch, pcd, corruption, inconsistency, smsbackup, smsrestore, setkeyswitch, crash, panic
Previously Published As 78300

Attachments

This solution has no attachment