How to Configure FC HBA NPIV With Vsan / Vhba on Guest LDom

Asset ID:	1-71-2108444.1
Update Date:	2018-04-18
Keywords:

Solution Type Technical Instruction Sure

Solution 2108444.1 : How to Configure FC HBA NPIV With Vsan / Vhba on Guest LDom

Applies to:

SPARC T4-2 - Version All Versions and later
Solaris Operating System - Version 11.3 and later
Information in this document applies to any platform.

Goal

Note: Unless source and destination devices support direct-connect NPIV, a NPIV capable FC Switch is usually required between the FC HBA and the storage device.

The objective of this document is to show how we can use NPIV feature on a control domain, in combination with new feature vsan / vhba, available on Solaris 11.3 or greater.

vsan (primary or io domain) <<<——>>> vhba (guest)

About vhba and vsan , see this nice blog:
Virtual HBA in Oracle VM Server for SPARC
https://blogs.oracle.com/jsavit/entry/virtual_hba_in_oracle_vm

Implementing vHBA
It requires Oracle VM Server for SPARC 3.3, which is delivered with Oracle Solaris 11.3 in the control domain.
The guest domains using vHBA must run Solaris 11.3, as Solaris 10 does not have the SCSA interface described above.
There are no special hardware requirements other than having a physical HBA with LUNs.
It runs on any supported SPARC server - in this example on a SPARC T2.

Oracle® VM Server for SPARC 3.3 Release Notes
--> "Virtual SCSI HBA, which requires the Oracle Solaris 11.3 OS"

Solaris 11.3 provides Oracle VM Server for SPARC 3.3 , documentation is available here:
Oracle VM Server for SPARC 3.3 Documentation Library
--> see Chapter 11 "UsingVirtual SCSI Host Bus Adapters" of the "Oracle®VMServer for SPARC 3.3 Administration Guide"

About NPIV , see this document as a example:
How to enable NPIV - Oracle Emulex FC HBA : fcinfo shows NPIV Not Supported (Doc ID 1609077.1)

In this document : we will create and use a NPIV ports on a FC HBA to present different devices or LUNs on each WWN port:

1. Only a single tape device to a guest domain guest01, so the guest domain can do backup directly through the SAN, using vsan / vhba

For a guest domain, there are several ways to do this:
- assign the FC HBA via Direct IO to the guest domain

- create FC SR-IOV VFs and assign to the guest domain ,see:
How to Create and Assign FC SR-IOV Virtual Functions (VFs) on an Emulex FC HBA (Doc ID 1999091.1)

- assign virtual FC HBA as vsan (new in Solaris 11.3) to the guest domain, see also this doc:
LUNs Not Recognized on a Guest LDOM Using vHBAs. (Doc ID 2093753.1)

2. A group of LUNs / Disk devices using different vsan

Note. There is another approach to use NPIV with ldoms, see https://blogs.oracle.com/vmserver/entry/using_npiv_with_oracle_vm
but that was written on 2012, when vsan did not exist,
on step 3 we create a vdisk for each LUN mapped to the NPIV port.
"Once you have identified which LUNs are visible through the NPIV port, you can virtualize and assign them to a LDoms guest domain using regular LDoms commands (ldm add-vdsdev and ldm add-vdisk)."

Potentially relevant bug
Bug 24712380 - ldm ls-hba fails when Service Domains have large numbers of devices

Solution

Summary of actions performed :

1. On the service control domain there are two FC HBA Dual port , with two physical ports, so two WWNs.
2. We create NPIV ports on top of one FC HBA port, that way we have a new WWN for each NPIV port, so the FC HBA can have multiple virtual ports instead of a single physical port.
Note. From the SAN/Storage point of view, each WWN is an FC HBA initiator where we can map LUNs from SAN disk arrays or tape libraries, for the storage there is no difference between a physical port and the NPIV port (each WWN is unique and different)
3. Then we create a vsan/vhba for each NPIV port --> that way we can assign one NPIV port to a different guest domain.

Note1. The guest domain will see only the LUNs mapped (disks or tapes) to the WWN of that particular NPIV port.

Note2. With vsan/vhba the guest domain OS only see the LUNs present on the FC HBA, Solaris OS on guest domain does not see WWNs, or FC HBAs, or controllers (luxadm or cfgadm does not work) , Solaris just see the LUNs (disks with format, or tapes on /dev/rmt) mapped to the WWN of that FC HBA.

Note3. Please, be aware that Solaris 11.3 SRU 7.6.0, integrates the following RFE:
BUG 20951004 - VHBA SHOULD SUPPORT HBAS WHEN MPXIO IS ENABLED IN SERVICE DOMAIN

This means that *only for disk devices* , if you have Solaris 11.3 SRU 7.6.0 or greater
you don't need to disable MPxIO on the service domain on the HBA ports configured as vHBAs
But for tape devices, this is still needed : mpxio must be disabled on the service domain for the HBA ports configured as vHBAs, as explained on this bug:

Bug 24368313 - vHBAs not detecting tape drives on a guest LDOM

If tape devices are not detected by guest LDom and mpxio is disabled on the service domain, you would need "ldm rescan-vhba" to force vhba to discover the devices.

Example

1) Initially , on the service control domain, we have mapped a tape device to the physical port WWN 2100000e1e2c9f10 (c8,qlc0)

C# INST# PORT WWN MODEL FCODE STATUS DEVICE PATH
-- ----- -------- ----- ----- ------ -----------
c8 qlc0 2100000e1e2c9f10 7101674 (QLE8362) 4.02 CONNECTED /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0
c9 qlc1 2100000e1e2c9f11 7101674 (QLE8362) 4.02 CONNECTED /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1

c10 qlc2 21000024ff5afe11 7101674 (QLE8362) 4.02 CONNECTED /pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0
c11 qlc3 21000024ff5afe11 7101674 (QLE8362) 4.02 CONNECTED /pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0,1

root@cdom01:/# more cfgadm-al-o_show_FCP_dev.out
Ap_Id Type Receptacle Occupant Condition
c8 fc-fabric connected configured unknown
c8::2002000e1110b93c,0 tape connected configured unknown
c8::2002000e1110b93c,1 med-changer connected configured unknown

c9 fc-private connected configured unknown
c9::21000024ff5afe11,0 disk connected configured unknown
c9::21000024ff5afe11,10 disk connected configured unknown

c10 fc-fabric connected configured unknown
c10::2100000e1e125841,0 disk connected configured unknown
c10::2100000e1e125841,1 disk connected configured unknown
c10::2100000e1e125841,5 disk connected configured unknown

c11 fc-private connected configured unknown
c11::21000024ff5afe10,0 disk connected configured unknown
c11::21000024ff5afe10,10 disk connected configured unknown

c8 = qlc0 (fp1) -> /devices/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0:devctl
================================================================================
Port_ID Port WWN Device Description Type
------- -------- ------------------ ----
20500 2002000e1110b93c -> BDT Tape (Tape device)
20100 2100000e1e2c9f10 -> QLogic HBA: Port 1 (Unknown Type,Host Bus Adapter)

c10 = qlc2 (fp4) -> /devices/pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,0:devctl
================================================================================
Port_ID Port WWN Device Description Type
------- -------- ------------------ ----
10c00 2100000e1e125841 -> QLogic HBA: Port 1 (Disk device)
10100 2100000e1e2ca020 -> QLogic HBA: Port 1 (Unknown Type,Host Bus Adapter)

Jan 29 12:08:01 cdom01 qlc: [ID 628150 kern.info] NOTICE: Qlogic qlc(0,0,0): Link OFFLINE
Jan 29 12:08:01 cdom01 qlc: [ID 628150 kern.info] NOTICE: Qlogic qlc(0,0,0): Link ONLINE
Jan 29 12:22:52 cdom01 scsi: [ID 583861 kern.info] sgen0 at fp1: unit-address w2002000e1110b93c,1: 20500
Jan 29 12:22:52 cdom01 genunix: [ID 936769 kern.info] sgen0 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/sgen@w2002000e1110b93c,1
Jan 29 12:22:52 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/sgen@w2002000e1110b93c,1 (sgen0) online
Jan 29 12:22:52 cdom01 scsi: [ID 583861 kern.info] st0 at fp1: unit-address w2002000e1110b93c,0: 20500
Jan 29 12:22:52 cdom01 genunix: [ID 936769 kern.info] st0 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/st@w2002000e1110b93c,0
Jan 29 12:22:52 cdom01 scsi: [ID 365881 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/st@w2002000e1110b93c,0 (st0):
Jan 29 12:22:52 cdom01 <HP Ultrium LTO 3>
Jan 29 12:22:52 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/st@w2002000e1110b93c,0 (st0) online

2) There are no vsan on the primary domain for this FC HBA

root@cdom01:~# ldm list-hba -l primary
NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]

/SYS/MB/PCIE5/HBA0,1/PORT0,0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1/fp@0,0]

....

3) Create the vsan:

root@cdom01:~# ldm add-vsan /SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0 primary
/SYS/MB/PCIE5/HBA0/PORT0,0 resolved to device: /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0
root@cdom01:~#

4) Check that there is a new vsan on the primary domain

root@cdom01:~# ldm list-hba -l primary
NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]
/SYS/MB/PCIE5/HBA0,1/PORT0,0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1/fp@0,0]

....

5) Create the vhba

root@cdom01:~# ldm add-vhba p0_vhba-pci5 pci5_port0 tapedisk

6) Set the timeout to 90

root@cdom01:~# ldm set-vhba timeout=90 p0_vhba-pci5 tapedisk

Note. We create another vsan on another FC HBA port, where we have some LUNs mapped from a disk storage array:

ldm add-vsan /SYS/MB/PCIE8/HBA0/PORT0,0 pci8_port0 primary
ldm add-vhba p0_vhba-pci8 pci8_port0 tapedisk
ldm set-vhba timeout=90 p0_vhba-pci8 tapedisk

7) The following commands show current configuration:

root@cdom01:~# ldm list -o hba,san
NAME
primary

VSAN
NAME TYPE DEVICE IPORT
pci8_port0 VSAN vsan@4 [/pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,0]
pci5_port0 VSAN vsan@6 [/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]

------------------------------------------------------------------------------
NAME
tapedisk

VHBA
NAME VSAN DEVICE TOUT SERVER
p0_vhba-pci8 pci8_port0 90 primary
p0_vhba-pci5 pci5_port0 90 primary

root@cdom01:~# ldm list-hba -d primary

NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/sgen@w2002000e1110b93c,1 pci5_port0
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/st@w2002000e1110b93c,0 pci5_port0
/SYS/MB/PCIE5/HBA0,1/PORT0,0
c9t21000024FF5AFE11d10s0
c9t21000024FF5AFE11d0s0

/SYS/MB/PCIE8/HBA0/PORT0,0 pci8_port0
c10t2100000E1E125841d5s0 pci8_port0
c10t2100000E1E125841d1s0 pci8_port0
c10t2100000E1E125841d0s0 pci8_port0
c10t2100000E1E125841d2s0 pci8_port0
/SYS/MB/PCIE8/HBA0,1/PORT0,0
c11t21000024FF5AFE10d10s0
c11t21000024FF5AFE10d0s0

root@cdom01:/ldom/iso# ldm list-hba -t -d primary
NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0
init-port 2100000e1e2c9f10
Transport Protocol FABRIC
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/sgen@w2002000e1110b93c,1 pci5_port0
tport 2002000e1110b93c
lun 1
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0/st@w2002000e1110b93c,0 pci5_port0
tport 2002000e1110b93c
lun 0

/SYS/MB/PCIE5/HBA0,1/PORT0,0
init-port 2100000e1e2c9f11
Transport Protocol FABRIC
c9t21000024FF5AFE11d10s0
tport 21000024ff5afe11
lun 10
c9t21000024FF5AFE11d0s0
tport 21000024ff5afe11
lun 0

/SYS/MB/PCIE8/HBA0/PORT0,0 pci8_port0
init-port 2100000e1e2ca020
Transport Protocol FABRIC
c10t2100000E1E125841d5s0 pci8_port0
tport 2100000e1e125841
lun 5
c10t2100000E1E125841d1s0 pci8_port0
tport 2100000e1e125841
lun 1
c10t2100000E1E125841d0s0 pci8_port0
tport 2100000e1e125841
lun 0
c10t2100000E1E125841d2s0 pci8_port0
tport 2100000e1e125841
lun 2

/SYS/MB/PCIE8/HBA0,1/PORT0,0
init-port 2100000e1e2ca021
Transport Protocol FABRIC
c11t21000024FF5AFE10d10s0
tport 21000024ff5afe10
lun 10
c11t21000024FF5AFE10d0s0
tport 21000024ff5afe10
lun 0

root@cdom01:~# ldm list-domain -l
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 16 16G 0.2% 0.2% 5d 12h 20m

...

VSAN
NAME TYPE DEVICE IPORT
pci8_port0 VSAN vsan@4 [/pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,0]
pci5_port0 VSAN vsan@6 [/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]

------------------------------------------------------------------------------
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
tapedisk bound ------ 5002 16 32G

...

VHBA
NAME VSAN DEVICE TOUT SERVER
p0_vhba-pci8 pci8_port0 vhba@0 90 primary
p0_vhba-pci5 pci5_port0 vhba@1 90 primary

Note. If we bind and start the guest domain tapedisk, from the OBP we can see the vhbas

root@cdom01:/# ldm bind tapedisk
root@cdom01:/# ldm start tapedisk
LDom tapedisk started
root@cdom01:/# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 16 16G 0.1% 0.1% 7d 11h 7m
tapedisk active -t---- 5002 16 32G 6.2% 6.2% 17s
root@cdom01:/# telnet 0 5002
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.

Connecting to console "tapedisk" in group "tapedisk" ....
Press ~? for control options ..

{0} ok

{0} ok devalias
p0_vhba-pci5 /virtual-devices@100/channel-devices@200/scsi@1 <<---
p0_vhba-pci8 /virtual-devices@100/channel-devices@200/scsi@0 <<---
vdisk_iso_ai /virtual-devices@100/channel-devices@200/disk@1
vdisk1 /virtual-devices@100/channel-devices@200/disk@0
vnet9 /virtual-devices@100/channel-devices@200/network@0
net /virtual-devices@100/channel-devices@200/network@0
disk /virtual-devices@100/channel-devices@200/disk@0
virtual-console /virtual-devices/console@1
name aliases
{0} ok

8) Create two NPIV port, one each fc hba port

root@cdom01:~# fcadm create-npiv-port -p c0007d6bef267821 -n c0007d6bef267820 2100000e1e2c9f10
root@cdom01:~# fcadm create-npiv-port -p c0007d6bef123456 -n c0007d6bef123455 2100000e1e2ca020

Check the new NPIV ports are present:

root@cdom01:~# fcinfo hba-port
HBA Port WWN: 2100000e1e2c9f10
Port Mode: Initiator
Port ID: 20100
OS Device Name: /dev/cfg/c8
Manufacturer: QLogic Corp.
Model: 7023303
Firmware Version: 7.05.01
FCode/BIOS Version: BIOS: 3.19; fcode: 4.02; EFI: 5.36;
Serial Number: 463916R+1535279857
Driver Name: qlc
Driver Version: 150505-5.03b
Type: N-port
State: online
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 2000000e1e2c9f10
Max NPIV Ports: 254
NPIV port list:
Virtual Port1:
Node WWN: c0007d6bef267820
Port WWN: c0007d6bef267821

....

HBA Port WWN: 2100000e1e2ca020
Port Mode: Initiator
Port ID: 10100
OS Device Name: /dev/cfg/c10
Manufacturer: QLogic Corp.
Model: 7023303
Firmware Version: 7.05.01
FCode/BIOS Version: BIOS: 3.19; fcode: 4.02; EFI: 5.36;
Serial Number: 463916R+1535279874
Driver Name: qlc
Driver Version: 150505-5.03b
Type: N-port
State: online
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 2000000e1e2ca020
Max NPIV Ports: 254
NPIV port list:
Virtual Port1:
Node WWN: c0007d6bef123455
Port WWN: c0007d6bef123456

Expected messages when NPIV ports are created:

Feb 8 21:34:14 cdom01 qlc: [ID 648178 kern.info] Qlogic qlc(0) NPIV(1) WWPN=c0007d6bef267821 : WWNN=c0007d6bef267820
Feb 8 21:34:15 cdom01 fcsm: [ID 517869 kern.info] NOTICE: fcsm(9): attached to path /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1
Feb 8 21:34:15 cdom01 genunix: [ID 936769 kern.info] fp9 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1
Feb 8 21:34:15 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1 (fp9) online
Feb 8 21:35:36 cdom01 qlc: [ID 648178 kern.info] Qlogic qlc(2) NPIV(1) WWPN=c0007d6bef123456 : WWNN=c0007d6bef123455
Feb 8 21:35:36 cdom01 fcsm: [ID 517869 kern.info] NOTICE: fcsm(10): attached to path /pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,1
Feb 8 21:35:36 cdom01 genunix: [ID 936769 kern.info] fp10 is /pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,1
Feb 8 21:35:36 cdom01 genunix: [ID 408114 kern.info] /pci@3c0/pci@1/pci@0/pci@7/SUNW,qlc@0/fp@0,1 (fp10) online

On the FC switch where c8 , qlc0 (fp1) is connected to portID 020100 we see the new NPIV port as another fc hba device

1 1 020100 id N16 Online FC F-Port 1 N Port + 1 NPIV public

brocade6505-tvp540-a:admin> nsshow
{
Type Pid COS PortName NodeName TTL(sec)
N 020100; 3;21:00:00:0e:1e:2c:9f:10;20:00:00:0e:1e:2c:9f:10; na
FC4s: IPFC FCP
PortSymb: [8] "qlc(0,0)"
NodeSymb: [5] ":qlc0"
Fabric Port Name: 20:01:50:eb:1a:6e:d7:bd
Permanent Port Name: 21:00:00:0e:1e:2c:9f:10
Port Index: 1
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No

N 020101; 3;c0:00:7d:6b:ef:26:78:21;c0:00:7d:6b:ef:26:78:20; na <<<--- the new NPIV port
FC4s: IPFC FCP
PortSymb: [8] "qlc(0,1)"
NodeSymb: [18] "cdom01:qlc0"
Fabric Port Name: 20:01:50:eb:1a:6e:d7:bd
Permanent Port Name: 21:00:00:0e:1e:2c:9f:10
Port Index: 1
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No

Then make a new zone on the FC switch for the NPIV port WWN c0:00:7d:6b:ef:26:78:21 and a SE6140 storage and map two new LUNs

When we run cfgenable on the FC switch (to make effective zonning changes)
the new LUNs from SE6140 storage array are discovered automatically on primary domain

Feb 8 22:51:34 cdom01 scsi: [ID 583861 kern.info] ssd8 at fp9: unit-address w201600a0b811fbd4,1: 21300
Feb 8 22:51:34 cdom01 genunix: [ID 936769 kern.info] ssd8 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,1
Feb 8 22:51:34 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,1 (ssd8) online
Feb 8 22:51:34 cdom01 scsi: [ID 583861 kern.info] ssd9 at fp9: unit-address w201600a0b811fbd4,0: 21300
Feb 8 22:51:34 cdom01 genunix: [ID 936769 kern.info] ssd9 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,0
Feb 8 22:51:34 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,0 (ssd9) online
Feb 8 22:51:34 cdom01 scsi: [ID 583861 kern.info] ssd10 at fp9: unit-address w201700a0b811fbd4,1: 21200
Feb 8 22:51:34 cdom01 genunix: [ID 936769 kern.info] ssd10 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,1
Feb 8 22:51:36 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,1 (ssd10) online
Feb 8 22:51:36 cdom01 scsi: [ID 583861 kern.info] ssd11 at fp9: unit-address w201700a0b811fbd4,0: 21200
Feb 8 22:51:36 cdom01 genunix: [ID 936769 kern.info] ssd11 is /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,0
Feb 8 22:51:37 cdom01 genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,0 (ssd11) online

11. c14t201600A0B811FBD4d0 <SUN-CSM200_R-0760-50.00GB> vol1
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,0
12. c14t201600A0B811FBD4d1 <SUN-CSM200_R-0760 cyl 510 alt 2 hd 64 sec 64> npiv_tes
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201600a0b811fbd4,1
13. c14t201700A0B811FBD4d0 <drive type unknown>
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,0
14. c14t201700A0B811FBD4d1 <drive type unknown>
/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1/ssd@w201700a0b811fbd4,1

c14 fc-fabric connected configured unknown
c14::201600a0b811fbd4,0 disk connected configured unknown
c14::201600a0b811fbd4,1 disk connected configured unknown
c14::201700a0b811fbd4,0 disk connected configured unknown
c14::201700a0b811fbd4,1 disk connected configured unknown

The new NPIV port is seen as "/SYS/MB/PCIE5/HBA0/PORT0,1"

root@cdom01:~# ldm list-hba -l primary
NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]

/SYS/MB/PCIE5/HBA0/PORT0,1 <<<----
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1]

/SYS/MB/PCIE5/HBA0,1/PORT0,0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1/fp@0,0]

9) Then create the new vsan and vhba on top of that NPIV port

ldm add-vsan /SYS/MB/PCIE5/HBA0/PORT0,1 pci5_port0_npiv1 primary
ldm add-vhba p0npiv1_vhba-pci5 pci5_port0_npiv1 tapedisk
ldm set-vhba timeout=90 p0pniv1_vhba-pci5 tapedisk

Checking new vsan and vhba are created:

root@cdom01:~# ldm list -o hba
....
NAME
tapedisk

VHBA
NAME VSAN DEVICE TOUT SERVER
p0_vhba-pci8 pci8_port0 vhba@0 90 primary
p0_vhba-pci5 pci5_port0 vhba@1 90 primary
p0npiv1_vhba-pci5 pci5_port0_npiv1 vhba@2 90 primary

root@cdom01:~# ldm list-hba -l primary
NAME VSAN
---- ----
/SYS/MB/PCIE5/HBA0/PORT0,0 pci5_port0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0]

/SYS/MB/PCIE5/HBA0/PORT0,1 pci5_port0_npiv1 <<<--- New one
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,1]

/SYS/MB/PCIE5/HBA0,1/PORT0,0
[/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1/fp@0,0]
....

Notice, when we run "ldm add-vhba p0npiv1_vhba-pci5 pci5_port0_npiv1 tapedisk" , on the guest domain the two LUNs
were automatically discovered (notice here we have mpxio enabled on the guest domain)

Feb 8 22:59:52 guest01 cnex: [ID 799930 kern.info] channel-device: vhba4
Feb 8 22:59:52 guest01 genunix: [ID 936769 kern.info] vhba4 is /virtual-devices@100/channel-devices@200/scsi@2
Feb 8 22:59:52 guest01 genunix: [ID 408114 kern.info] /virtual-devices@100/channel-devices@200/scsi@2 (vhba4) online
Feb 8 22:59:54 guest01 scsi: [ID 583861 kern.info] vhba5 at vhba4: initiator-port 84f9efe200000005 scsi-iport 0
Feb 8 22:59:54 guest01 genunix: [ID 936769 kern.info] vhba5 is /virtual-devices@100/channel-devices@200/scsi@2/iport@0
Feb 8 22:59:54 guest01 genunix: [ID 408114 kern.info] /virtual-devices@100/channel-devices@200/scsi@2/iport@0 (vhba5) online

Feb 8 23:00:04 guest01 scsi: [ID 583861 kern.info] sd4 at scsi_vhci0: unit-address g600a0b80002983fe00001c4f56b093ba: f_asym_lsi
Feb 8 23:00:04 guest01 genunix: [ID 936769 kern.info] sd4 is /scsi_vhci/disk@g600a0b80002983fe00001c4f56b093ba
Feb 8 23:00:05 guest01 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c4f56b093ba (sd4) online
Feb 8 23:00:05 guest01 genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c4f56b093ba (sd4) multipath status: degraded: path 5 vhba5/disk@w201700a0b811fbd4,0 is standby
Feb 8 23:00:05 guest01 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c4f56b093ba (sd4) multipath status: optimal: path 6 vhba5/disk@w201600a0b811fbd4,0 is online: Load balancing: round-robin
Feb 8 23:00:05 guest01 scsi: [ID 583861 kern.info] sd5 at scsi_vhci0: unit-address g600a0b80002983fe00001c7356b1dab5: f_asym_lsi
Feb 8 23:00:05 guest01 genunix: [ID 936769 kern.info] sd5 is /scsi_vhci/disk@g600a0b80002983fe00001c7356b1dab5
Feb 8 23:00:05 guest01 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c7356b1dab5 (sd5) online
Feb 8 23:00:05 guest01 genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c7356b1dab5 (sd5) multipath status: degraded: path 7 vhba5/disk@w201600a0b811fbd4,1 is online
Feb 8 23:00:05 guest01 scsi: [ID 967866 kern.warning] WARNING: scsi_enumeration_failed: vhba5 probe@w201700a0b811fbd4,0,d enumeration failed during tran_tgt_init
Feb 8 23:00:05 guest01 scsi: [ID 967866 kern.warning] WARNING: scsi_enumeration_failed: vhba5 probe@w201600a0b811fbd4,0,d enumeration failed during tran_tgt_init
Feb 8 23:00:05 guest01 genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600a0b80002983fe00001c7356b1dab5 (sd5) multipath status: optimal: path 8 vhba5/disk@w201700a0b811fbd4,1 is standby

Note. As a best practice, from the control domain you can save this LDom configuration with command "ldm add-spconfig my_new_config_name" , as explained on:
Configuration, Save & Restore Setup and Troubleshooting of Oracle VM Server for SPARC (LDom) (Doc ID 1464421.1)

10) On the guest domain the disks can be seen with format :

root@guest01:~# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t600A0B80002983FE00001C4F56B093BAd0 <SUN-CSM200_R-0760-50.00GB> vol1
/scsi_vhci/disk@g600a0b80002983fe00001c4f56b093ba
1. c0t600A0B80002983FE00001C7356B1DAB5d0 <SUN-CSM200_R-0760 cyl 510 alt 2 hd 64 sec 64> npiv_tes
/scsi_vhci/disk@g600a0b80002983fe00001c7356b1dab5
2. c0t600144F0B09928A0661956AB7B9F0001d0 <SUN-COMSTAR-1.0 cyl 4998 alt 2 hd 256 sec 64> T5-2
/scsi_vhci/disk@g600144f0b09928a0661956ab7b9f0001

3. c0t600144F0B09928A066195510922D0001d0 <SUN-COMSTAR-1.0-9.77GB>
/scsi_vhci/disk@g600144f0b09928a066195510922d0001
4. c0t600144F0B09928A066195510923D0003d0 <SUN-COMSTAR-1.0-341.80GB> gerh-os
/scsi_vhci/disk@g600144f0b09928a066195510923d0003
5. c0t600144F0B09928A06619551092370002d0 <SUN-COMSTAR-1.0 cyl 632 alt 2 hd 254 sec 254>
/scsi_vhci/disk@g600144f0b09928a06619551092370002

6. c1d0 <Unknown-Unknown-0001-39.06GB>
/virtual-devices@100/channel-devices@200/disk@0
Specify disk (enter its number):

Note. Commands like luxadm , fcinfo, cfgadm will show nothing when only vsan / vhba are configured:

root@guest01:~# luxadm -e port
root@guest01:~# fcinfo hba-port
No Adapters Found.
root@guest01:~## cfgadm -al
cfgadm: Configuration administration not supported

Only mpathadm command will show mpxio path information:

root@guest01:~# mpathadm show LU
Logical Unit: /dev/rdsk/c0t600144F0B09928A0661956AB7B9F0001d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: COMSTAR
Revision: 1.0
Name Type: unknown type
Name: 600144f0b09928a0661956ab7b9f0001
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000002
Target Port Name: w2100000e1e125841
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 0
Explicit Failover: no
Access State: active optimized
Target Ports:
Name: w2100000e1e125841
Relative ID: 2

Logical Unit: /dev/rdsk/c0t600144F0B09928A06619551092370002d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: COMSTAR
Revision: 1.0
Name Type: unknown type
Name: 600144f0b09928a06619551092370002
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000002
Target Port Name: w2100000e1e125841
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 0
Explicit Failover: no
Access State: active optimized
Target Ports:
Name: w2100000e1e125841
Relative ID: 2

Logical Unit: /dev/rdsk/c0t600144F0B09928A066195510923D0003d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: COMSTAR
Revision: 1.0
Name Type: unknown type
Name: 600144f0b09928a066195510923d0003
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000002
Target Port Name: w2100000e1e125841
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 0
Explicit Failover: no
Access State: active optimized
Target Ports:
Name: w2100000e1e125841
Relative ID: 2

Logical Unit: /dev/rdsk/c0t600144F0B09928A066195510922D0001d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: COMSTAR
Revision: 1.0
Name Type: unknown type
Name: 600144f0b09928a066195510922d0001
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000002
Target Port Name: w2100000e1e125841
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 0
Explicit Failover: no
Access State: active optimized
Target Ports:
Name: w2100000e1e125841
Relative ID: 2

Logical Unit: /dev/rdsk/c0t600A0B80002983FE00001C4F56B093BAd0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: CSM200_R
Revision: 0760
Name Type: unknown type
Name: 600a0b80002983fe00001c4f56b093ba
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000005
Target Port Name: w201700a0b811fbd4
Override Path: NA
Path State: OK
Disabled: no

Initiator Port Name: 84f9efe200000005
Target Port Name: w201600a0b811fbd4
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 5
Explicit Failover: yes
Access State: standby
Target Ports:
Name: w201700a0b811fbd4
Relative ID: 0

ID: 6
Explicit Failover: yes
Access State: active
Target Ports:
Name: w201600a0b811fbd4
Relative ID: 0

Logical Unit: /dev/rdsk/c0t600A0B80002983FE00001C7356B1DAB5d0s2
mpath-support: libmpscsi_vhci.so
Vendor: SUN
Product: CSM200_R
Revision: 0760
Name Type: unknown type
Name: 600a0b80002983fe00001c7356b1dab5
Asymmetric: yes
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 84f9efe200000005
Target Port Name: w201600a0b811fbd4
Override Path: NA
Path State: OK
Disabled: no

Initiator Port Name: 84f9efe200000005
Target Port Name: w201700a0b811fbd4
Override Path: NA
Path State: OK
Disabled: no

Target Port Groups:
ID: 7
Explicit Failover: yes
Access State: active
Target Ports:
Name: w201600a0b811fbd4
Relative ID: 0

ID: 8
Explicit Failover: yes
Access State: standby
Target Ports:
Name: w201700a0b811fbd4
Relative ID: 0

root@guest01:~# mpathadm show initiator-port 84f9efe200000005
Initiator Port: 84f9efe200000005
Transport Type: parallel SCSI
OS Device File: /devices/virtual-devices@100/channel-devices@200/scsi@2/iport@0

root@guest01:~# mpathadm show initiator-port 84f9efe200000002
Initiator Port: 84f9efe200000002
Transport Type: parallel SCSI
OS Device File: /devices/virtual-devices@100/channel-devices@200/scsi@0/iport@0

The tape device can be seen on /dev/rmt :

root@guest01:/dev/rmt# ls -l
total 24
lrwxrwxrwx 1 root root 94 Feb 4 16:09 0 -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0b -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:b
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0bn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:bn
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0c -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:c
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0cb -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:cb
lrwxrwxrwx 1 root root 97 Feb 4 16:09 0cbn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:cbn
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0cn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:cn
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0h -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:h
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0hb -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:hb
lrwxrwxrwx 1 root root 97 Feb 4 16:09 0hbn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:hbn
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0hn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:hn
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0l -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:l
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0lb -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:lb
lrwxrwxrwx 1 root root 97 Feb 4 16:09 0lbn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:lbn
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0ln -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:ln
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0m -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:m
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0mb -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:mb
lrwxrwxrwx 1 root root 97 Feb 4 16:09 0mbn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:mbn
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0mn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:mn
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0n -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:n
lrwxrwxrwx 1 root root 95 Feb 4 16:09 0u -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:u
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0ub -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:ub
lrwxrwxrwx 1 root root 97 Feb 4 16:09 0ubn -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:ubn
lrwxrwxrwx 1 root root 96 Feb 4 16:09 0un -> ../../devices/virtual-devices@100/channel-devices@200/scsi@1/iport@0/tape@w2002000e1110b93c,0:un

root@guest01:/dev/rmt# mt status
HP Ultrium LTO 3 tape drive:
sense key(0x0)= No Additional Sense residual= 0 retries= 0
file no= 0 block no= 0
root@guest01:/dev/rmt#

For additional troubleshooting:

The following commands will show at kernel level what LUNs each of vhba and vsan know about:

On guest domain:

# echo "::vhba" | mdb -k

On the service control domain:

# echo "::vsan" | mdb -k

References

<NOTE:1609077.1> - How to enable NPIV - Oracle Emulex FC HBA : fcinfo shows NPIV Not Supported
<NOTE:1999091.1> - How to Create and Assign FC SR-IOV Virtual Functions (VFs) on an Emulex FC HBA
<BUG:20951004> - VHBA SHOULD SUPPORT HBAS WHEN MPXIO IS ENABLED IN SERVICE DOMAIN
<NOTE:2093753.1> - LUNs Not Recognized on a Guest LDOM Using vHBAs.
<BUG:24368313> - VHBAS NOT DETECTING TAPE DRIVES ON A GUEST LDOM

Attachments

This solution has no attachment