LDOM Virtual Function Configuration Lost For Fibre Channel HBAs - How To Create FC HBA VF With User-allocated WWN

Asset ID:	1-72-2243834.1
Update Date:	2017-12-13
Keywords:

Solution Type Problem Resolution Sure

Solution 2243834.1 : LDOM Virtual Function Configuration Lost For Fibre Channel HBAs - How To Create FC HBA VF With User-allocated WWN

Applies to:

SPARC T4-2 - Version All Versions and later
SPARC T5-2 - Version All Versions and later
SPARC T7-2 - Version All Versions and later
Solaris Operating System - Version 11 11/11 and later
Oracle Solaris on SPARC (64-bit)

Symptoms

The virtual function configuration was lost for Fibre Channel HBAs.

Changes

The Fibre Channel HBA drivers and firmware were updated.

Cause

After creating Fibre Channel virtual functions and saving the configuration, the pwwn and nwwn of the new FC VFs aren't saved in the LDOM zeus database file (/var/opt/SUNWldm/ldom-db.xml).

Port and node world wide names are only saved to the constraints database if they are explicitly set by the user when creating the virtual function. Auto-allocated WWN's are not saved in the DB.

The overall design philosophy employed is that auto-allocated values are not considered constraints, and as such do not need to be saved.

Solution

It is best to manually assign the WWNs when creating virtual functions to ensure a predictable configuration of the SAN storage.

The following text was added to the ldm set-io description to hopefully alleviate future issues:

Fibre Channel virtual functions are permitted to have the following types of WWNs:

- Auto-allocated WWNs: Automatically generates values in the range of 00:14:4F:F8:00:00 - 00:14:4F:FB:FF:FF.

The prefix for a port-wwn is 10:00:00:14:4F:FC:00:01.
The prefix for a node-wwn is 20:00:00:14:4F:FC:00:01.
When auto-allocated WWNs are used, collision detection is enabled and the information is not saved in the constraints database or in XML, so NO recovery is possible.

- User-allocated WWNs: Uses user-allocated port-wwn and node-wwn values. For manual allocation values must be in the range of 00:14:4F:FC:00:00 - 00:14:4F:FF:FF:FF

When user-allocated WWNs are used, no collision detection is performed, but the value is saved for recovery.

The CLI will present a warning to the consumer that auto-allocated WWNs are not saved in case of reconfiguration.

Use the user-allocated method to guarantee that the same WWNs are used when Fibre Channel virtual functions are destroyed and re-created.

Example of user-allocated VF

1. Initially we have a 16GB FC HBA with no VF created :

From fcinfo hba-port , this is the physical FC HBA and port WWNs :

HBA Port WWN: 10000090fa2ee19f <<<----
Port Mode: Initiator
Port ID: 20800
OS Device Name: /dev/cfg/c18
Manufacturer: Emulex
Model: 7101684
Firmware Version: 7101684 10.6.230.0
FCode/BIOS Version: Boot:10.6.230.0 Fcode:4.03a5
Serial Number: 4925382+131100002M
Driver Name: emlxs
Driver Version: 3.2.00.0 (2017.02.08.15.00)
Type: N-port
State: online
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 20000090fa2ee19f
NPIV Not Supported
HBA Port WWN: 10000090fa2ee1a0 <<<----
Port Mode: Initiator
Port ID: 20000
OS Device Name: /dev/cfg/c19
Manufacturer: Emulex
Model: 7101684
Firmware Version: 7101684 10.6.230.0
FCode/BIOS Version: Boot:10.6.230.0 Fcode:4.03a5
Serial Number: 4925382+131100002M
Driver Name: emlxs
Driver Version: 3.2.00.0 (2017.02.08.15.00)
Type: N-port
State: online
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 20000090fa2ee1a0
NPIV Not Supported

2. Then we enable IOV on the pci bus

root@t5-2-tvp540-s:~# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
root@t5-2-tvp540-s:~# ldm set-io iov=on pci_1
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
root@t5-2-tvp540-s:~#

3. Now we will create one VF with port WWN 10:00:00:14:4F:FD:AB:01 on the FC HBA port /SYS/MB/PCIE5/IOVFC.PF0

root@t5-2-tvp540-s:~# ldm create-vf port-wwn=10:00:00:14:4F:FD:AB:01 node-wwn=20:00:00:14:4F:FD:AB:01 /SYS/MB/PCIE5/IOVFC.PF0
Created new vf: /SYS/MB/PCIE5/IOVFC.PF0.VF0
root@t5-2-tvp540-s:~# date
Mon Nov 6 14:18:33 GMT 2017

Notice. This action will impact IO on that FC HBA port, as the FC HBA port is restarted / reset to make changes effective (see below messages as a result of creating the VF)

Nov 6 14:18:17 t5-2-tvp540-s genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g60080e500023eab00000212159ffd5cd (ssd10) multipath status: optimal: path 6 fp15/ssd@w20430080e523eab0,0 is offline
Nov 6 14:18:17 t5-2-tvp540-s genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g60080e500023eab00000212159ffd5cd (ssd10) multipath status: optimal: path 7 fp15/ssd@w20420080e523eab0,0 is offline
Nov 6 14:18:17 t5-2-tvp540-s last message repeated 1 time
Nov 6 14:18:17 t5-2-tvp540-s genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g60080e500023eab00000212159ffd5cd (ssd10) multipath status: optimal: path 6 fp15/ssd@w20430080e523eab0,0 is offline
Nov 6 14:18:17 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ 5.0401]emlxs4: NOTICE: 710: Link down.
Nov 6 14:18:18 t5-2-tvp540-s genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0/fp@0,0 (fp15) offline
Nov 6 14:18:21 t5-2-tvp540-s genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0 (emlxs4) offline
Nov 6 14:18:21 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [14.0145]emlxs4: NOTICE: 200: Adapter initialization. (Networking is not supported in SLI4, turning it off)
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [14.061D]emlxs4: NOTICE: 730: Link reset.
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ B.1BB9]emlxs4: NOTICE: 100: Driver attach. (emlxs s11-64 sparc v3.2.00.0 (2017.02.08.15.00))
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ B.1BBC]emlxs4: NOTICE: 100: Driver attach. (7101684 Dev_id:e200 Sub_id:e20e Id:48)
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ B.1BC9]emlxs4: NOTICE: 100: Driver attach. (Firmware:10.6.230.0 (xe201.grp) Boot:10.6.230.0 Fcode:4.03a5)
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ B.1BF3]emlxs4: NOTICE: 100: Driver attach. (SLI:4(FIP) MSIX:8 NPIV:0 FCA FC)
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ B.1C01]emlxs4: NOTICE: 100: Driver attach. (WWPN:10000090FA2EE19F WWNN:20000090FA2EE19F)
Nov 6 14:18:23 t5-2-tvp540-s pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0, emlxs4
Nov 6 14:18:23 t5-2-tvp540-s genunix: [ID 936769 kern.info] emlxs4 is /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0
Nov 6 14:18:23 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [14.1540]emlxs4: NOTICE: 423: Adapter notice. (SLI Port Async Event: Physical media detected)
Nov 6 14:18:25 t5-2-tvp540-s pcieb: [ID 586369 kern.info] PCIE-device: fibre-channel@0,2, emlxs-1
Nov 6 14:18:25 t5-2-tvp540-s genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0 (emlxs4) online
Nov 6 14:18:25 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ 5.06F4]emlxs4: NOTICE: 730: Link reset. (Resetting link...)
Nov 6 14:18:25 t5-2-tvp540-s emlxs: [ID 349649 kern.info] [ 5.0743]emlxs4: NOTICE: 730: Link reset.
Nov 6 14:18:26 t5-2-tvp540-s fcsm: [ID 517869 kern.info] NOTICE: fcsm(15): attached to path /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0/fp@0,0
Nov 6 14:18:26 t5-2-tvp540-s genunix: [ID 936769 kern.info] fp15 is /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0/fp@0,0
Nov 6 14:18:26 t5-2-tvp540-s genunix: [ID 408114 kern.info] /pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0/fp@0,0 (fp15) online
Nov 6 14:18:31 t5-2-tvp540-s genunix: [ID 530209 kern.info] /scsi_vhci/ssd@g60080e500023eab00000212159ffd5cd (ssd10) multipath status: optimal: path 6 fp15/ssd@w20430080e523eab0,0 is online: Load balancing: round-robin
Nov 6 14:18:31 t5-2-tvp540-s genunix: [ID 530209 kern.info] /scsi_vhci/ssd@g60080e500023eab00000212159ffd5cd (ssd10) multipath status: optimal: path 7 fp15/ssd@w20420080e523eab0,0 is online: Load balancing: round-robin

Note. On this example we create an additional second VF with port WWN 10:00:00:14:4F:FD:AB:02 on the other FC HBA port /SYS/MB/PCIE5/IOVFC.PF1

root@t5-2-tvp540-s:~# ldm create-vf port-wwn=10:00:00:14:4F:FD:AB:02 node-wwn=20:00:00:14:4F:FD:AB:02 /SYS/MB/PCIE5/IOVFC.PF1
Created new vf: /SYS/MB/PCIE5/IOVFC.PF1.VF0

Note. Example error, when the WWN selected is out of range

# ldm set-io port-wwn=10:00:00:14:4f:f8:d4:cd node-wwn=20:00:00:14:4f:f8:d4:cd /SYS/MB/PCIE5/IOVFC.PF0.VF0

Port WWNs beginning with 0x1000 are reserved for internally
generated names and cannot be used for manual assignment.

4. Check VF are created:

root@t5-2-tvp540-s:~# ldm list-io -l /SYS/MB/PCIE5
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/MB/PCIE5 PCIE pci_1 primary OCC
[pci@340/pci@1/pci@0/pci@4]
SUNW,emlxs@0/fp/disk
SUNW,emlxs@0/fp/st
SUNW,emlxs@0/fp@0,0
SUNW,emlxs@0,1/fp/disk
SUNW,emlxs@0,1/fp/st
SUNW,emlxs@0,1/fp@0,0
fibre-channel@0,2
fibre-channel@0,a
/SYS/MB/PCIE5/IOVFC.PF0 PF pci_1 primary
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0]
maxvfs = 8
/SYS/MB/PCIE5/IOVFC.PF1 PF pci_1 primary
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,1]
maxvfs = 8
/SYS/MB/PCIE5/IOVFC.PF0.VF0 VF pci_1 <<------
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,2]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fd:ab:01
node-wwn = 20:00:00:14:4f:fd:ab:01
bw-percent = 0
/SYS/MB/PCIE5/IOVFC.PF1.VF0 VF pci_1 <<------
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,a]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fd:ab:02
node-wwn = 20:00:00:14:4f:fd:ab:02
bw-percent = 0

We can also check that new entries are created on /var/opt/SUNWldm/ldom-db.xml for these two VFs

<iov_pf_devices>
<iov_pf_instance>
<iov_pf_name>/SYS/MB/PCIE5/IOVFC.PF0</iov_pf_name>
<iov_pf_class_name>iov-fibrechannel</iov_pf_class_name>
<iov_vf_devices>
<iov_vf_device_instance>
<iov_vf_name>/SYS/MB/PCIE5/IOVFC.PF0.VF0</iov_vf_name>
<iov_vf_class_props>
<port-wwn>1152921591848217345</port-wwn> <<<-----------
<node-wwn>2305843096455064321</node-wwn>
<bandwidth-percentile>0</bandwidth-percentile>
</iov_vf_class_props>
</iov_vf_device_instance>
</iov_vf_devices>
</iov_pf_instance>
<iov_pf_instance>
<iov_pf_name>/SYS/MB/PCIE5/IOVFC.PF1</iov_pf_name>
<iov_pf_class_name>iov-fibrechannel</iov_pf_class_name>
<iov_vf_devices>
<iov_vf_device_instance>
<iov_vf_name>/SYS/MB/PCIE5/IOVFC.PF1.VF0</iov_vf_name>
<iov_vf_class_props>
<port-wwn>1152921591848217346</port-wwn> <<<-----------
<node-wwn>2305843096455064322</node-wwn>
<bandwidth-percentile>0</bandwidth-percentile>
</iov_vf_class_props>
</iov_vf_device_instance>
</iov_vf_devices>
</iov_pf_instance>
</iov_pf_devices>

5. Now we can assign these VF to a guest LDom , on this example we will assign to guest LDom testzgz

root@t5-2-tvp540-s:/# ldm add-io /SYS/MB/PCIE5/IOVFC.PF0.VF0 testzgz
root@t5-2-tvp540-s:/# ldm add-io /SYS/MB/PCIE5/IOVFC.PF1.VF0 testzgz

root@t5-2-tvp540-s:~# ldm list-io -l /SYS/MB/PCIE5
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/MB/PCIE5 PCIE pci_1 primary OCC
[pci@340/pci@1/pci@0/pci@4]
SUNW,emlxs@0/fp/disk
SUNW,emlxs@0/fp/st
SUNW,emlxs@0/fp@0,0
SUNW,emlxs@0,1/fp/disk
SUNW,emlxs@0,1/fp/st
SUNW,emlxs@0,1/fp@0,0
fibre-channel@0,2
fibre-channel@0,a
/SYS/MB/PCIE5/IOVFC.PF0 PF pci_1 primary
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0]
maxvfs = 8
/SYS/MB/PCIE5/IOVFC.PF1 PF pci_1 primary
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,1]
maxvfs = 8
/SYS/MB/PCIE5/IOVFC.PF0.VF0 VF pci_1 testzgz <<<-----
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,2]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fd:ab:01
node-wwn = 20:00:00:14:4f:fd:ab:01
bw-percent = 0
/SYS/MB/PCIE5/IOVFC.PF1.VF0 VF pci_1 testzgz <<<-----
[pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,a]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fd:ab:02
node-wwn = 20:00:00:14:4f:fd:ab:02
bw-percent = 0

From the control domain "zgz" these looks like any other FC HBA

root@zgz:~# luxadm -e port
/devices/pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,2/fp@0,0:devctl NOT CONNECTED
/devices/pci@340/pci@1/pci@0/pci@4/SUNW,emlxs@0,a/fp@0,0:devctl NOT CONNECTED

From the Brocade switch, we can check existance with swicthshow and nsshow command:

From switchshow

0 0 020000 id N16 Online FC F-Port 1 N Port + 2 NPIV public

8 8 020800 id N16 Online FC F-Port 1 N Port + 2 NPIV public

brocade6505-tvp540-a:admin> nsshow
{
Type Pid COS PortName NodeName TTL(sec)
N 020000; 2,3;2f:ff:00:10:a0:e1:2e:fa;27:ff:00:20:a0:e1:2e:fa; na
NodeSymb: [255] "LANCER_BASE_PORT..............................................................................................................................................................................................................................................."
Fabric Port Name: 20:00:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:a0:e1:2e:fa
Port Index: 0
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No
N 020001; 3;10:00:00:90:fa:2e:e1:a0;20:00:00:90:fa:2e:e1:a0; na
FC4s: IPFC FCP
PortSymb: [34] "Emulex PPN-10:00:00:90:fa:2e:e1:a0"
NodeSymb: [52] "Emulex 7101684 FV10.6.230.0 DV3.2.00.0 t5-2-tvp540-s"
Fabric Port Name: 20:00:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:a0:e1:2e:fa
Port Index: 0
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No
N 020002; 3;10:00:00:14:4f:fd:ab:02;20:00:00:14:4f:fd:ab:02; na
FC4s: IPFC FCP
PortSymb: [34] "Emulex PPN-10:00:00:14:4f:fd:ab:02"
NodeSymb: [40] "Emulex 7101684 FV10.6.230.0 DV3.0.02 zgz"
Fabric Port Name: 20:00:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:a0:e1:2e:fa
Port Index: 0
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No

N 020800; 2,3;2f:ff:00:10:9f:e1:2e:fa;27:ff:00:20:9f:e1:2e:fa; na
NodeSymb: [255] "LANCER_BASE_PORT..............................................................................................................................................................................................................................................."
Fabric Port Name: 20:08:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:9f:e1:2e:fa
Port Index: 8
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No
N 020801; 3;10:00:00:90:fa:2e:e1:9f;20:00:00:90:fa:2e:e1:9f; na
FC4s: IPFC FCP
PortSymb: [34] "Emulex PPN-10:00:00:90:fa:2e:e1:9f"
NodeSymb: [52] "Emulex 7101684 FV10.6.230.0 DV3.2.00.0 t5-2-tvp540-s"
Fabric Port Name: 20:08:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:9f:e1:2e:fa
Port Index: 8
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No
N 020802; 3;10:00:00:14:4f:fd:ab:01;20:00:00:14:4f:fd:ab:01; na
FC4s: IPFC FCP
PortSymb: [34] "Emulex PPN-10:00:00:14:4f:fd:ab:01"
NodeSymb: [40] "Emulex 7101684 FV10.6.230.0 DV3.0.02 zgz"
Fabric Port Name: 20:08:50:eb:1a:6e:d7:bd
Permanent Port Name: 2f:ff:00:10:9f:e1:2e:fa
Port Index: 8
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
LSAN: No

References

<BUG:24934990> - RFE: SET-IO: ALLOW "FORCE LOAD" OF RESERVED FC VF WWNS
<BUG:24913592> - FC-VF: PWWN/NWWN ARE NOT SAVED IN THE ZEUS DATABASE FILE ONCE CREATED/SAVED
<BUG:19023382> - DOC ERROR: EXAMPLE 6-25 MODIFYING A FIBRE CHANNEL SR-IOV VIRTUAL FUNCTION

Attachments

This solution has no attachment