Control Domain Unable To Recognize FC HBA In PCIe Slot Correctly - It Shows "assigned-device" In The Path But No LDOMs

Asset ID:	1-72-2215548.1
Update Date:	2016-12-20
Keywords:

Solution Type Problem Resolution Sure

Solution 2215548.1 : Control Domain Unable To Recognize FC HBA In PCIe Slot Correctly - It Shows "assigned-device" In The Path But No LDOMs

Applies to:

SPARC T4-1 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

T4-1 system has two FC HBA installed in slot PCIE0 and PCIE1.

$ prtdiag -v

-----------
/SYS/MB/RISER0/PCIE0PCIE SUNW,qlc-pciex1077,2532 QLE2562 5.0GTx4
/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0
/SYS/MB/RISER0/PCIE0PCIE SUNW,qlc-pciex1077,2532 QLE2562 5.0GTx4
/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1

/SYS/MB/RISER1/PCIE1PCIE SUNW,assigned-device-pciex1077,2532 5.0GTx4
/pci@400/pci@2/pci@0/pci@8/SUNW,assigned-device@0
/SYS/MB/RISER1/PCIE1PCIE SUNW,assigned-device-pciex1077,2532 5.0GTx4
/pci@400/pci@2/pci@0/pci@8/SUNW,assigned-device@0,1

Only the HBA with "SUNW,qlc" in its path is recognized by the OS:

$ luxadm -e port
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0/fp@0,0:devctl NOT CONNECTED
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:devctl NOT CONNECTED

OBP cannot "see" the HBA with "SUNW, assigned-device"

{0} ok probe-scsi-all

Unit 0 Removable Read Only device /pci@400/pci@2/pci@0/pci@4/scsi@0
FCode Version 1.00.63Target 9
Unit 0 Disk HITACHI H106030SDSUN300G A2B0 585937500 Blocks, 300 GB
SASDeviceName 5000cca03c29bd94 SASAddress 5000cca03c29bd95 PhyNum 0
Target a
Unit 0 Removable Read Only device TEAC DV-W28SS-V 1.0B
SATA device PhyNum 6
Target b
Unit 0 Disk HITACHI H106030SDSUN300G A2B0 585937500 Blocks, 300 GB
SASDeviceName 5000cca03c29ced4 SASAddress 5000cca03c29ced5 PhyNum 2

/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0,1
QLogic QLE2562 Host Adapter FCode(SPARC): 2.03 06/30/08
Wait for link up -
Firmware version 4.03.02
Fibre Channel Link down
Possible causes: No cable, incorrect connection mode or data rate
SFP state: 8Gb Present

/pci@400/pci@1/pci@0/pci@8/SUNW,qlc@0
QLogic QLE2562 Host Adapter FCode(SPARC): 2.03 06/30/08
Wait for link up -
Firmware version 4.03.02
Fibre Channel Link down
Current firmware state - Waiting for Login
SFP state: 8Gb Present
/pci@400/pci@1/pci@0/pci@4/scsi@0
FCode Version 1.00.63Target 9
Unit 0 Disk HITACHI H106030SDSUN300G A2B0 585937500 Blocks, 300 GB
SASDeviceName 5000cca03c29d1a0 SASAddress 5000cca03c29d1a1 PhyNum 0
Target a
Unit 0 Disk HITACHI H106030SDSUN300G A2B0 585937500 Blocks, 300 GB
SASDeviceName 5000cca03c29c5e8 SASAddress 5000cca03c29c5e9 PhyNum 2

Changes

A new Solaris OS was installed on the server

Cause

The clue was the "SUNW,assigned-device" itself.

This means that the PCIe slot where this HBA is installed, was previously configured as an end point device (Direct IO) on a guest LDOM

But the Explorer (v8.11) which now collects LDOM data by default, has no LDOM configuration data.

The ILOM console logs, shows that there was an LDOM named ldm1 which was configured with DIO

<ILOM-snapshot-dir>/ilom/console1.txt

# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
pci_0 BUS pci_0 primary IOV
niu_0 NIU niu_0 primary
/SYS/MB/RISER0/PCIE0 PCIE pci_0 primary OCC
/SYS/MB/RISER1/PCIE1 PCIE pci_0 ldm1 OCC
/SYS/MB/RISER2/PCIE2 PCIE pci_0 primary OCC
/SYS/MB/RISER0/PCIE3 PCIE pci_0 primary OCC
/SYS/MB/RISER1/PCIE4 PCIE pci_0 ldm1 OCC

The domain was stopped but not completely removed before reinstalling a new OS on the control domain

root # ldm stop-domain ldm1
Remote graceful shutdown or reboot capability is not available on ldm1
LDom ldm1 stopped

Note:
Proper procedure to remove a guest ldom:

1. In the control domain, list the ldoms:

control-domain # ldm ls-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
ldm1 active -n---- 5001 8 8G 0.1% 7h 26m

2. Shutdown the domain, and from the control domain:

control-domain# ldm stop-domain ldm1
LDom ldm1 stopped

3. . Un-bind the domain from the hypervisor.

control-domain # ldm unbind-domain ldm1

control-domain # ldm ls-domain

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
ldm1 inactive ------ 8 8G

4. Remove the domain

control-domain # ldm remove-domain ldm1

Since the DIO guest domain was not completely removed, the booting config was still set to dio:

From the ILOM console log:

> NOTICE: Booting config = dio

Solution

The server has to restored to factory default using either of the 2 procedures:

1. From the primary domain.

Select the factory default configuration.

primary# ldm set-config factory-default

Stop the control domain.

primary# shutdown -i1 -g0 -y

Powercycle the system to load the factory default configuration.

-> stop /SYS
-> start /SYS

2. From the SP.

Restore the factory default configuration from the service processor.

-> set /HOST/bootmode config=factory-default

Powercycle the system to load the factory default configuration.

-> reset /SYS

Attachments

This solution has no attachment