Sun SPARC[TM] Enterprise M3000, M4000, M5000, M8000, M9000: Solaris Hotplug Framework

Asset ID:	1-79-1456060.1
Update Date:	2017-10-11
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1456060.1 : Sun SPARC[TM] Enterprise M3000, M4000, M5000, M8000, M9000: Solaris Hotplug Framework - S10U9 or 142909-17

Applies to:

Sun SPARC Enterprise M9000-32 Server - Version Not Applicable and later
Sun SPARC Enterprise M3000 Server - Version Not Applicable and later
Sun SPARC Enterprise M8000 Server - Version Not Applicable and later
Sun SPARC Enterprise M5000 Server - Version Not Applicable and later
Sun SPARC Enterprise M9000-64 Server - Version Not Applicable and later
Information in this document applies to any platform.

Purpose

This docs describe the Solaris Hotplug Framework is introduced in S10U9 or 142909-17.
The goal is to provide a common framework; ie. generic and common hotplug interfaces.

Details

The Solaris Hotplug Framework is introduced in S10U9 or 142909-17.
The goal is to provide a common framework; ie. generic and common hotplug interfaces.

Solaris Hotplug Framework - Block Diagram

From a userland perspective, the framework is composed of :

an SMF service : svc:/system/hotplug
- because of the Service Management Facility policy, it's disabled by default
a daemon : /usr/sbin/hotplugd to centrally manage hotplug operations
a CLI : hotplug(1M) - /usr/sbin/hotplug to support physical and virtual hotplugging
a cfgadm plugin : /usr/lib/cfgadm/cfgadm_shp.so to support physical hotplugging of connectors
a libhotplug(3LIB) library : /lib/libhotplug.so.1 shared by hotplug(1M) and cfgadm

Userland Hotplug Overview

This means that cfgadm is no longer the only interface.
cfgadm has a specific plugin so is compatible with the new framework but will return the following error until the hotplug service is enabled :

cfgadm: Configuration administration not supported: Error: hotplug service is 
probably not running, please use 'svcadm enable hotplug' to enable the service. 
See cfgadm_shp(1M) for more details.

The hotplug service is disabled by default :

bash-3.00# svcs  hotplug
STATE          STIME    FMRI
disabled       11:15:00 svc:/system/hotplug:default

bash-3.00# svcs -l hotplug
fmri         svc:/system/hotplug:default
name         hotplug daemon
enabled      false
state        disabled
next_state   none
state_time   Wed Sep 29 11:15:00 2010
logfile      /var/svc/log/system-hotplug:default.log
restarter    svc:/system/svc/restarter:default
contract_id  
dependency   require_all/none svc:/system/device/local (online)
dependency   require_all/none svc:/system/filesystem/local (online)

To enable the hotplug service :

bash-3.00# svcadm enable hotplug

bash-3.00# svcs  hotplug
STATE          STIME    FMRI
online         11:16:27 svc:/system/hotplug:default

bash-3.00# svcs -l hotplug
fmri         svc:/system/hotplug:default
name         hotplug daemon
enabled      true
state        online
next_state   none
state_time   Wed Sep 29 11:16:27 2010
logfile      /var/svc/log/system-hotplug:default.log
restarter    svc:/system/svc/restarter:default
contract_id  130 
dependency   require_all/none svc:/system/device/local (online)
dependency   require_all/none svc:/system/filesystem/local (online)

bash-3.00# ps -eaf | grep hotplug
    root 16665     1   0 11:16:27 ?           0:00 /usr/lib/hotplugd

There is no change applied to RCM so the current interaction via the RCM scripts and hotplug/cfgadm should have no impact.

Also the hotplug controller drivers continue to generate events via syseventd; no change on this part.

Impact

The impact on servicing the pci cards installed in an OPL domain running S10U9 or 142909-17 is minimum.

The cfgadm can still be used to service the pci cards installed in the slots in the IOU.
The only impact is that the hotplug SMF service must be enabled in order to use the hotplug feature.
However, enabling the service is effective immediately and does not require any reboot.

The M-series Service Manual should be updated with the following note :

Caution- If you use PCI Hot Plug (PHP) function on the servers with Oracle Solaris 10 9/10, or 142909-17 or later installed, enable the hotplug service as follows.
# svcadm enable hotplug

We'll work on updating the EIS checklist for OPL.

From an ESG perspective, only the OPL systems are 'impacted'.

On a Starcat domain, no need to enable the hotplug service. The pci bus is still managed by the cfgadm_pci(1m) plugin and not the new cfgadm_shp(1m) plugin.

The hotplug command can now be used to change the status of the card but also to get some information about some cards.

The details available depends on the type of the card.

The new hotplug command also provides more granularity as it's possible to offline one port of the connector.

Similar to showdevices on the XSCF, the hotplug command (via the verbose mode) can report if the connector/port is being used or not. This may be helpful to determine a

condition that prevent an hotplug operation from succeeding.

State model

cfgadm : Receptacle / Occupant

empty <-> disconnected <-> connected <-> unconfigured <-> configured

hotplug : Connector / Port

empty <-> present <-> powered <-> enabled

Empty: No component plugged to the connector.
Present: Component already plugged to the connector.
Powered: The connector is powered.
Enabled: The connector is fully functional.

cfgadm	hotplug
Receptacle / Occupant	Connector / Port
connected / configured	Enabled / Online
connected / configured	Enabled / Offline
connected / unconfigured	Powered / -
disconnected / unconfigured	Present / -

Note that the new Solaris Hotplug Framework introduced a "virtual hotplug" feature that is not discussed here.

The hotplug(1m) command

The manual page is attached.

bash-3.00# hotplug
Usage:  hotplug  <subcommand> [<args>]

Subcommands:
   list       [-l] [-v] [<path> [<connection>]]
   online     <path> <port>
   offline    [-f] [-q] <path> <port>
   enable     <path> <connector>
   disable    [-f] [-q] <path> <connector>
   poweron    <path> <connector>
   poweroff   [-f] [-q] <path> <connector>
   get        -o <options> <path> <connector>
   set        -o <options> <path> <connector>

List the connectors and ports or with a verbose output.

Using the verbose mode, the hotplug command can provide some information of a connector/port that may be in use.
Example :

bash-3.00# hotplug list -v /pci@0,600000/pci@0/pci@9 iou#0-pci#1
[iou#0-pci#1]  (ENABLED)
<pci.0,0>  (ONLINE)
LSILogic,sas@0
     tape
     disk
     smp@w508002000000377f
     sd@1,0
          { mounted filesystem "/Stef" }
     sd@2,0
          { contains metadb(s) }

Example with an Emulex LPe11002-S card

bash-3.00# prtdiag -v
[...]
04  PCIe  71      2, fc20, 10df       2,  0,  0  okay     4,    4  SUNW,emlxs-pciex10df,fc20      LPe11002-S          
    /pci@47,700000/SUNW,emlxs@0                                                                         
04  PCIe  71      2, fc20, 10df       2,  0,  1  okay     4,    4  SUNW,emlxs-pciex10df,fc20      LPe11002-S          
    /pci@47,700000/SUNW,emlxs@0,1

bash-3.00# cfgadm -al
[...]
iou#0-pci#7                    connected    configured   ok         Location: iou#0-pci#7
Jan  1  1970 fibre/hp     n        /devices/pci@47,700000:iou#0-pci#7
[...]
bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug list -l /pci@47,700000
/pci@47,700000
/pci@47,700000 [iou#0-pci#7] (ENABLED)
/pci@47,700000 <pci.0,0> (ONLINE)
/pci@47,700000/SUNW,emlxs@0
/pci@47,700000/SUNW,emlxs@0/fp
/pci@47,700000/SUNW,emlxs@0/fp/disk
/pci@47,700000/SUNW,emlxs@0/fp/tape
/pci@47,700000/SUNW,emlxs@0/fp@0,0
/pci@47,700000 <pci.0,1> (ONLINE)
/pci@47,700000/SUNW,emlxs@0,1
/pci@47,700000/SUNW,emlxs@0,1/fp
/pci@47,700000/SUNW,emlxs@0,1/fp/disk
/pci@47,700000/SUNW,emlxs@0,1/fp/tape
/pci@47,700000/SUNW,emlxs@0,1/fp@0,0

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (ENABLED)
     <pci.0,0>  (ONLINE)
     SUNW,emlxs@0
          fp
               disk
               tape
          fp@0,0
     <pci.0,1>  (ONLINE)
     SUNW,emlxs@0,1
          fp
               disk
               tape
          fp@0,0

bash-3.00# hotplug get -o help /pci@47,700000 iou#0-pci#7
fault_led=on|off|blink
power_led=on|off|blink
attn_led=on|off|blink
active_led=on|off|blink
card_type=type description
board_type=type description
slot_condition=type description

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
fault_led=off
power_led=on
attn_led=off
active_led=off
card_type=fibre
board_type=pci hotplug
slot_condition=ok

bash-3.00# hotplug poweroff /pci@47,700000 iou#0-pci#7

Sep 29 09:21:39 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0/fp@0,0 (fp3) offline
Sep 29 09:21:41 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0 (emlxs2) offline
Sep 29 09:21:41 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1/fp@0,0 (fp6) offline
Sep 29 09:21:43 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1 (emlxs3) offline

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (PRESENT)

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    unknown      disconnected unconfigured unknown

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
ERROR: unsupported property name or value.
(Properties may depend upon connector state.)

bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=off,fault=off,attn=off,active=off

bash-3.00# hotplug poweron /pci@47,700000 iou#0-pci#7

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (POWERED)

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    unknown      connected    unconfigured unknown

bash-3.00# hotplug enable /pci@47,700000 iou#0-pci#7

Sep 29 09:26:23 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [13.02D8]emlxs2: NOTICE: 200: Adapter initialization. 
(Firmware update not needed.)
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.175F]emlxs2: NOTICE: 100: Driver attach. 
(Emulex-S s10-64 sparc v2.50o (2010.01.08.09.45))
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1762]emlxs2: NOTICE: 100: Driver attach.
(LPe11002-S Dev_id:fc20 Sub_id:fc22 Id:26)
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.176F]emlxs2: NOTICE: 100: Driver attach.
(Firmware:2.82a4 (Z3F2.82A4) Boot:5.03a8 Fcode:3.10a3)
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1798]emlxs2: NOTICE: 100: Driver attach. 
(SLI:3 MSI:2 NPIV:0 FCA)
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.179E]emlxs2: NOTICE: 100: Driver attach. 
(WWPN:10000000C961A0EA WWNN:20000000C961A0EA)
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 px: [ID 236367 kern.info] PCI Express-device: SUNW,emlxs@0, emlxs2
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] emlxs2 is /pci@47,700000/SUNW,emlxs@0
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0 (emlxs2) online
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] fp3 is /pci@47,700000/SUNW,emlxs@0/fp@0,0
Sep 29 09:26:33 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0/fp@0,0 (fp3) online
Sep 29 09:26:36 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [13.02D8]emlxs3: NOTICE: 200: Adapter initialization. 
(Firmware update not needed.)
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.175F]emlxs3: NOTICE: 100: Driver attach. 
(Emulex-S s10-64 sparc v2.50o (2010.01.08.09.45))
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1762]emlxs3: NOTICE: 100: Driver attach. 
(LPe11002-S Dev_id:fc20 Sub_id:fc22 Id:26)
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.176F]emlxs3: NOTICE: 100: Driver attach. 
(Firmware:2.82a4 (Z3F2.82A4) Boot:5.03a8 Fcode:3.10a3)
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1798]emlxs3: NOTICE: 100: Driver attach. 
(SLI:3 MSI:2 NPIV:0 FCA)
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.179E]emlxs3: NOTICE: 100: Driver attach. 
(WWPN:10000000C961A0EB WWNN:20000000C961A0EB)
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 px: [ID 236367 kern.info] PCI Express-device: SUNW,emlxs@0,1, emlxs3
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] emlxs3 is /pci@47,700000/SUNW,emlxs@0,1
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1 (emlxs3) online
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] fp6 is /pci@47,700000/SUNW,emlxs@0,1/fp@0,0
Sep 29 09:26:46 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1/fp@0,0 (fp6) online

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (ENABLED)
     <pci.0,0>  (ONLINE)
     SUNW,emlxs@0
          fp
               disk
               tape
          fp@0,0
     <pci.0,1>  (ONLINE)
     SUNW,emlxs@0,1
          fp
               disk
               tape
          fp@0,0

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    fibre/hp     connected    configured   ok

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
fault_led=off
power_led=on
attn_led=off
active_led=off
card_type=fibre
board_type=pci hotplug
slot_condition=ok

bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug offline /pci@47,700000 pci.0,1

Sep 29 09:35:05 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1/fp@0,0 (fp6) offline
Sep 29 09:35:07 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1 (emlxs3) offline

bash-3.00# hotplug list -l /pci@47,700000
/pci@47,700000
/pci@47,700000 [iou#0-pci#7] (ENABLED)
/pci@47,700000 <pci.0,0> (ONLINE)
/pci@47,700000/SUNW,emlxs@0
/pci@47,700000/SUNW,emlxs@0/fp
/pci@47,700000/SUNW,emlxs@0/fp/disk
/pci@47,700000/SUNW,emlxs@0/fp/tape
/pci@47,700000/SUNW,emlxs@0/fp@0,0
/pci@47,700000 <pci.0,1> (OFFLINE)
/pci@47,700000/SUNW,emlxs
/pci@47,700000/SUNW,emlxs/fp
/pci@47,700000/SUNW,emlxs/fp/disk
/pci@47,700000/SUNW,emlxs/fp/tape
bash-3.00# hotplug list -v /pci@47,700000

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    fibre/hp     connected    configured   ok

bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
fault_led=off
power_led=on
attn_led=off
active_led=off
card_type=fibre
board_type=pci hotplug
slot_condition=ok

bash-3.00# hotplug online /pci@47,700000 pci.0,1

Sep 29 09:46:43 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [13.02D8]emlxs3: NOTICE: 200: Adapter initialization. 
(Firmware update not needed.)
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.175F]emlxs3: NOTICE: 100: Driver attach. 
(Emulex-S s10-64 sparc v2.50o (2010.01.08.09.45))
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1762]emlxs3: NOTICE: 100: Driver attach. 
(LPe11002-S Dev_id:fc20 Sub_id:fc22 Id:26)
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.176F]emlxs3: NOTICE: 100: Driver attach. 
(Firmware:2.82a4 (Z3F2.82A4) Boot:5.03a8 Fcode:3.10a3)
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.1798]emlxs3: NOTICE: 100: Driver attach. 
(SLI:3 MSI:2 NPIV:0 FCA)
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 emlxs: [ID 349649 kern.info] [ B.179E]emlxs3: NOTICE: 100: Driver attach. 
(WWPN:10000000C961A0EB WWNN:20000000C961A0EB)
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 px: [ID 236367 kern.info] PCI Express-device: SUNW,emlxs@0,1, emlxs3
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] emlxs3 is /pci@47,700000/SUNW,emlxs@0,1
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1 (emlxs3) online
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 genunix: [ID 936769 kern.info] fp6 is /pci@47,700000/SUNW,emlxs@0,1/fp@0,0
Sep 29 09:46:53 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1/fp@0,0 (fp6) online

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    fibre/hp     connected    configured   ok

bash-3.00# cfgadm -c disconnect iou#0-pci#7

Sep 29 09:49:38 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0/fp@0,0 (fp3) offline
Sep 29 09:49:40 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0 (emlxs2) offline
Sep 29 09:49:40 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1/fp@0,0 (fp6) offline
Sep 29 09:49:43 v4u-m9000a-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@47,700000/SUNW,emlxs@0,1 (emlxs3) offline

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    fibre/hp     connected    configured   ok

bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=off,fault=off,attn=off,active=off

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
ERROR: unsupported property name or value.
(Properties may depend upon connector state.)

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (PRESENT)

bash-3.00# cfgadm -c configure iou#0-pci#7
bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (ENABLED)
     <pci.0,0>  (ONLINE)
     SUNW,emlxs@0
          fp
               disk
               tape
          fp@0,0
     <pci.0,1>  (ONLINE)
     SUNW,emlxs@0,1
          fp
               disk
               tape
          fp@0,0

bash-3.00# hotplug set -o attn_led=on /pci@47,700000 iou#0-pci#7

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
fault_led=off
power_led=on
attn_led=on
active_led=off
card_type=fibre
board_type=pci hotplug

bash-3.00# hotplug set -o attn_led=off /pci@47,700000 iou#0-pci#7

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@47,700000 iou#0-pci#7
fault_led=off
power_led=on
attn_led=off
active_led=off
card_type=fibre
board_type=pci hotplug
slot_condition=ok

bash-3.00# hotplug disable /pci@47,700000 iou#0-pci#7

bash-3.00# cfgadm -x led iou#0-pci#7
Ap_Id                   Led
iou#0-pci#7             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug list -v /pci@47,700000
pci@47,700000
     [iou#0-pci#7]  (POWERED)

bash-3.00# cfgadm -al | grep iou#0-pci#7
iou#0-pci#7                    unknown      connected    unconfigured unknown

Example with a F20 card

bash-3.00# prtdiag -v
[...]
00  PCIe  0       8,   58, 1000     119,  0,  0  N/A      8,    8  LSILogic,sas-pciex1000,58      LSI,1068E           
    /pci@0,600000/pci@0/pci@9/LSILogic,sas@0

bash-3.00# format
Searching for disks...

c3t1d0: configured with capacity of 22.89GB
c3t2d0: configured with capacity of 22.89GB

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@1,0
       2. c3t1d0 <ATA-MARVELLSD88SA02-DP0N cyl 23435 alt 2 hd 16 sec 128>
          /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@1,0
       3. c3t2d0 <ATA-MARVELLSD88SA02-DP0N cyl 23435 alt 2 hd 16 sec 128>
          /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@2,0

bash-3.00# cfgadm -al | grep iou#0-pci#1
iou#0-pci#1                    scsi/hp      connected    configured   ok

bash-3.00# cfgadm -x led iou#0-pci#1
Ap_Id                   Led
iou#0-pci#1             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug list -v pci@0,600000/pci@0/pci@9            
pci@9
     [iou#0-pci#1]  (ENABLED)
     <pci.0,0>  (ONLINE)
     LSILogic,sas@0
          tape
          disk
          smp@w508002000000377f
          sd@1,0
          sd@2,0


bash-3.00# hotplug list -v pci@0,600000/pci@0/pci@9 iou#0-pci#1
[iou#0-pci#1]  (ENABLED)
<pci.0,0>  (ONLINE)
LSILogic,sas@0
     tape
     disk
     smp@w508002000000377f
     sd@1,0
     sd@2,0

bash-3.00# hotplug disable /pci@0,600000/pci@0/pci@9 iou#0-pci#1

Sep 29 10:21:29 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/smp@w508002000000377f (smp2) offline
Sep 29 10:21:29 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@1,0 (sd43) offline
Sep 29 10:21:29 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@2,0 (sd44) offline
Sep 29 10:21:29 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1) offline

bash-3.00# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@1,0
Specify disk (enter its number):

bash-3.00# cfgadm -x led iou#0-pci#1
Ap_Id                   Led
iou#0-pci#1             power=on,fault=off,attn=off,active=off

bash-3.00# cfgadm -al | grep iou#0-pci#1
iou#0-pci#1                    unknown      connected    unconfigured unknown

bash-3.00# hotplug list -l pci@0,600000/pci@0/pci@9 iou#0-pci#1
pci@0,600000/pci@0/pci@9 [iou#0-pci#1] (POWERED)

bash-3.00# hotplug poweroff /pci@0,600000/pci@0/pci@9 iou#0-pci#1

bash-3.00# hotplug list -l /pci@0,600000/pci@0/pci@9 iou#0-pci#1
/pci@0,600000/pci@0/pci@9 [iou#0-pci#1] (PRESENT)

bash-3.00# cfgadm -al | grep iou#0-pci#1
iou#0-pci#1                    unknown      disconnected unconfigured unknown

bash-3.00# cfgadm -x led iou#0-pci#1
Ap_Id                   Led
iou#0-pci#1             power=off,fault=off,attn=off,active=off

bash-3.00# hotplug enable /pci@0,600000/pci@0/pci@9 iou#0-pci#1

Sep 29 10:26:35 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03  initiator SCSI ID now 7
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03  Rev. 8 LSI, Inc. 1068E found.
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03  mpt1 supports power management.
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03 scsi: [ID 243001 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:35 v4u-m5000b-dom00-gmp03  DMA restricted below 4GB boundary due to errata
Sep 29 10:26:36 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:36 v4u-m5000b-dom00-gmp03  mpt1 Firmware version v1.1b.3.0 (IT)
Sep 29 10:26:36 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:36 v4u-m5000b-dom00-gmp03  mpt1: IOC Operational.
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 scsi: [ID 365881 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03  NVDATA default version 0x2d00, persistent version 0x2d00
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 scsi: [ID 243001 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1):
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03  mpt1: Initiator WWNs: 0x5080020000d31320-0x5080020000d31327
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 pcieb: [ID 586369 kern.info] PCIE-device: LSILogic,sas@0, mpt1
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 genunix: [ID 936769 kern.info] mpt1 is /pci@0,600000/pci@0/pci@9/LSILogic,sas@0
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0 (mpt1) online
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 scsi: [ID 456205 kern.info] smp2 at mpt1: wwn 508002000000377f
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 genunix: [ID 936769 kern.info] smp2 is /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/smp@w508002000000377f
Sep 29 10:26:48 v4u-m5000b-dom00-gmp03 genunix: [ID 408114 kern.info] /pci@0,600000/pci@0/pci@9/LSILogic,sas@0/smp@w508002000000377f (smp2) online

bash-3.00# cfgadm -x led iou#0-pci#1
Ap_Id                   Led
iou#0-pci#1             power=on,fault=off,attn=off,active=off

bash-3.00# hotplug get -o fault_led,power_led,attn_led,active_led,card_type,board_type,slot_condition /pci@0,600000/pci@0/pci@9 iou#0-pci#1
fault_led=off
power_led=on
attn_led=off
active_led=off
card_type=scsi
board_type=pci hotplug
slot_condition=ok

bash-3.00# hotplug list -v /pci@0,600000/pci@0/pci@9 iou#0-pci#1
[iou#0-pci#1]  (ENABLED)
<pci.0,0>  (ONLINE)
LSILogic,sas@0
     tape
     disk
     smp@w508002000000377f
     sd@1,0
          { mounted filesystem "/Stef" }
     sd@2,0
          { contains metadb(s) }

bash-3.00# cfgadm -c unconfigure iou#0-pci#1
cfgadm: Component system is busy, try again: 
     Resource               Information        
------------------  ---------------------------
/dev/dsk/c3t1d0s0   mounted filesystem "/Stef" 
/dev/dsk/c3t2d0s0   contains metadb(s)

bash-3.00# hotplug disable /pci@0,600000/pci@0/pci@9 iou#0-pci#1
ERROR: devices or resources are busy.
/pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@1,0:
   { mounted filesystem "/Stef" }
/pci@0,600000/pci@0/pci@9/LSILogic,sas@0/sd@2,0:
   { contains metadb(s) }

Issues

There is a known issue which prevent hotplug feature to work properly in the following scenario:

(see Bug 18529361 : Hotplug Slot cannot configure any cards after boot Solaris)

Occurrence

This issue can occur in only Solaris 11 on the following platforms:

Sun SPARC Enterprise(R) M3000 and M3000-E

Not affected.

Sun SPARC Enterprise(R) M4000 and Sun SPARC Enterprise(R) M5000 IOU0

The following PCIe slots are affected:
/pci[LSB1]1,700000 (PCIe Slot 2)
/pci[LSB2]2,600000 (PCIe Slot 3)
/pci[LSB2]3,700000 (PCIe Slot 4)

Note: When a Physical System Board (PSB) is in UniMode, [LSB1] it is the same as [LSB2]. When the PSB is placed in QuadMode, [LSB1] it is NOT the same as [LSB2] when the resources are in the same domain. It is possible however that when in QuadMode, [LSB1] could be the same as [LSB2] when the resources are in different domains.

Sun SPARC Enterprise(R) M5000 IOU1

The following PCIe slots are affected:
/pci[LSB3]1,700000 (PCIe Slot 2)
/pci[LSB4]2,600000 (PCIe Slot 3)
/pci[LSB4]3,700000 (PCIe Slot 4)

Note: Just as with IOU0, when a Physical System Board (PSB) is in UniMode, the IOU1 [LSB3] is the same as [LSB4]. When the PSB is placed in QuadMode, IOU1 [LSB3] is NOT the same as [LSB4] when the resources are in the same domain. It is possible however that when in QuadMode, IOU1 [LSB3] could be the same as [LSB4] when the resources are in different domains.

Sun SPARC Enterprise(R) M8000, Sun SPARC Enterprise(R) M9000-32, and Sun SPARC Enterprise(R) M9000-64

The following PCIe slots are affected:
/pci[LSB1]0,600000 (PCIe Slot 0)
/pci[LSB1]1,700000 (PCIe Slot 1)
/pci[LSB2]2,600000 (PCIe Slot 2)
/pci[LSB2]3,700000 (PCIe Slot 3)
/pci[LSB3]4,600000 (PCIe Slot 4)
/pci[LSB3]5,700000 (PCIe Slot 5)
/pci[LSB4]6,600000 (PCIe Slot 6)
/pci[LSB4]7,700000 (PCIe Slot 7)

Note: When the Physical System Board (PSB) is in UniMode, [LSB1] through [LSB4] will all have the same value. When in QuadMode, [LSB1] through [LSB4] will not be the same value when the resources are in the same domain. It is possible however that when in QuadMode, there could be the equivalent values for [LSB1] through [LSB4] when the resources are in different domains.

Sun SPARC(R) External I/O Expansion Unit

Not affected.

x86 Platform

Not affected.

Symptoms

If the described issue occurs, output similar to the following will be seen on the console while performing a hotplug operation using the command 'cfgadm' or 'hotplug' as follows:

e.g., hotplug operation on M5000, iou#1-pci#2

# hotplug poweron iou#1-pci#2
# hotplug enable iou#1-pci#2

Nov 26 11:06:45 m5000-b-dom-00 pcicfg: WARNING: failed to allocate
1000000 bytes for dev ethernet BASE register [0x10]
ERROR: hardware or driver specific failure

Workaround/Resolution:

Before system boot, insert an HBA into each PCIe slot where future hotplug operations might be performed. Only slots occupied during system boot will support future hotplug operations.

A final Solaris resolution will not be released.

Solaris 10 and Solaris 12 are not impacted

References

Attachments

This solution has no attachment