SPARC Sun Fire[TM] 12K/15K/E20K/E25K (Starcat): Domains won't setkeyswitch on

Asset ID:	1-72-1002172.1
Update Date:	2017-07-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1002172.1 : SPARC Sun Fire[TM] 12K/15K/E20K/E25K (Starcat): Domains won't setkeyswitch on - PICLD and POST

Applies to:

Sun Fire E20K Server - Version Not Applicable and later
Sun Fire E25K Server - Version Not Applicable and later
Sun Fire 15K Server - Version Not Applicable and later
Sun Fire 12K Server - Version Not Applicable and later
All Platforms

Symptoms

Setkeyswitch on for a domain may fail if the picld daemon running on the Main System Controller (SC) has died and restarted in degraded mode.

Cause

The picld daemon is not running or not working properly:

# prtpicl -v.err
prtpicl: picl_initialize failed: Daemon not responding

In this case not running:

# ps -ef |grep picld

Solution

Scenario 1 (OLD SMS versions)

Setkeyswitch on SMS 1.4 fails when picld is running in degraded mode. Setkeyswitch on operation will go in loop with error " CHS error 2: I/O error"

sms-svc:40> setkeyswitch -d adl0700 on
Powering on: CSB at CS1
Already powered on: CSB at CS1
Powering on: CSB at CS0
Already powered on: CSB at CS0
Waiting on exclusive access to EXB(s): 00001.
Powering on: EXB at EX0
Powering on: CPU at SB0
Powering on: EXB at EX1
Powering on: CPU at SB1
Powering on: EXB at EX4
Powering on: HPCI at IO4
Powering on: EXB at EX5
Powering on: HPCI at IO5
Powering on: CPU at SB5

Significant contents of .postrc (platform)
/etc/opt/SUNWSMS/SMS1.4/config/platform/.postrc:

# ident "@(#)postrc 1.1 01/04/02 SMI"

libresolve_list(): dlsym() fail for bug_4974982_fix
Reading domain blacklist file /etc/opt/SUNWSMS/config/B/blacklist ...
# ident "@(#)blacklist 1.1 01/04/02 SMI"
Reading platform blacklist file /etc/opt/SUNWSMS/config/platform/blacklist ...
# ident "@(#)blacklist 1.1 01/04/02 SMI"
SEEPROM probe took 0 seconds.
Reading Component Health Status (CHS) information ...
getComponentStatus(CP C0,,) returns error 2. Ignoring...
CHS error 2: I/O error
getComponentStatus(CSB C0,,) returns error 2. Ignoring...
CHS error 2: I/O error
getComponentStatus(CP C1,,) returns error 2. Ignoring...
CHS error 2: I/O error
getComponentStatus(CSB C1,,) returns error 2. Ignoring...
CHS error 2: I/O error

Note: On SMS 1.4, it is seen that "setchs" and "showchs" gives the same error "CHS error 2: I/O error" when picld is running in degraded mode.

Scenario 2 (SMS 1.4.1 and ABOVE)

Setkeyswitch on SMS 1.4.1, 1.5 and 1.6 fails when picld is running in degraded mode

Already powered on: HPCI at IO13
Powering on: EXB at EX16
Already powered on: EXB at EX16
Powering on: HPCI at IO16
Already powered on: HPCI at IO16
Note: Slot0 boards[17:0] assigned to domain but not present: 00010

Significant contents of .postrc (domain)
/etc/opt/SUNWSMS/SMS1.4.1/config/A/.postrc:
# Domain-specific .postrc file for domain A
level 16
verbose 70

Failed to get root; libfru error: Node not foundpcs_fru_libopen() failure. 
Exitcode = 43: Error in SSC software execution

or

Failed to open PICL; libfru error: General LIBFRU FAILUREpcs_fru_libopen() failure.
Exitcode = 43: Error in SSC software execution
# SMI Sun Fire 12/15/20/25K POST log closed Fri Nov 9 04:11:29 2012

 
Relief/Workaround

Use the 'ps -ef | grep picld' command or check the Main SC's /var/adm/messages file to determine if the picld daemon is running in degraded mode.

1) Use "ps -ef" command

# ps -ef | grep picld
root 9960 1 7 14:44:15 ? 0:02 /usr/lib/picl/picld -l -r

2) System controller messages file.

Apr 29 04:46:09 oacis12k1-sc1 picld[26995]: [ID 995911 daemon.crit] Received signal: Segmentation Fault
Apr 29 04:46:09 oacis12k1-sc1 picld[26995]: [ID 161310 daemon.crit] WARNING: Starting up in DEGRADED mode <<<<<<<<<<<<<<<<

To solve this issue, stop and restart picld (requires root user privledges on the SC), and restart SMS daemons

Stop and Restart picld

If SC on Solaris 8 and 9, do these

# /etc/init.d/picld stop
# /etc/init.d/picld start

If SC on Solaris 10, do these

# svcs picl
STATE          STIME    FMRI
online         Apr_06   svc:/system/picl:default
# svcadm restart picl
# svcs picl
STATE          STIME    FMRI
online         10:48:09 svc:/system/picl:default
#

Stop/Start SMS

Before restarting SMS, please turn OFF failover. If failover is turned ON, the Spare SC will take over the Main Role

sms-svc> setfailover off
sms-svc> su - root

# /etc/init.d/sms stop
# /etc/init.d/sms start

15133725 - picld will receive a signal 'Segmentation Fault' and restart in DEGRADED mode

Attachments

This solution has no attachment