SFP Not Working In 16Gb FC HBA Emulex Card - WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)

Asset ID:	1-72-1670748.1
Update Date:	2016-03-23
Keywords:

Solution Type Problem Resolution Sure

Solution 1670748.1 : SFP Not Working In 16Gb FC HBA Emulex Card - WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)

Applies to:

Emulex FC HBA - Version All Versions and later
SPARC T5-8 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Solaris 11.1.16.5.0 T5-8 server with two Dual 16Gigabit/Sec Fibre Channel / 10Gbps FCoE Host Adapter , Emulex LPe16002-M6-O Ganymede-E , Part Number 7023036

HOST NAME : server1
HOST TYPE : SPARC T5-8
HOST OS : Solaris 11.1 SPARC
HOST UPTIME: 61 day(s), 22:49, 9 users, load average: 1.80, 1.25, 1.11

C# INST# PORT WWN MODEL FCODE STATUS DEVICE PATH
-- ----- -------- ----- ----- ------ -----------
c8 emlxs0 4.03a1 NOT CONNECTED /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0
c9 emlxs1 4.03a1 NOT CONNECTED /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1
c10 emlxs2 4.03a1 NOT CONNECTED /pci@600/pci@1/pci@0/pci@e/SUNW,emlxs@0
c11 emlxs3 4.03a1 NOT CONNECTED /pci@600/pci@1/pci@0/pci@e/SUNW,emlxs@0,1

After inserting the new delivered SFP optics for the emulex FC HBA card these are not working as expected

Card:
  Manufacturer: Emulex
  Model: 7101684
  Firmware Version: 7101684 1.1.43.8
  FCode/BIOS Version: Boot:1.1.43.8 Fcode:4.03a1
SFP:
Oracle PN: 7023053

Error message in /var/adm/messages

May 7 15:21:09 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs3:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 7 15:21:09 server1 emlxs: [ID 986832 kern.warning] WARNING: emlxs3: Incompatible optics - Replace with compatible optics for card to function.

Card has been disabled and enabled using the hotplug command.

hotplug disable PCIEx
hotplug enable PCIEx

I can confirm FC HBA is an Oracle Emulex FC HBA and also SFPs PN 7023053 are correct PN for this FC HBA, from Oracle handbook:

Mixing Transceiver types is not supported. Use 16Gbps SFP+ or 10Gbps SFP+.
The 10Gbps SFP+ and 16Gbps SFP+ transceivers are sold and FRU'd as pairs.

Use 10Gbps Shortwave SFP+ 7023046, FRU Pair 7023057, Option 7101688.
Use 16Gbps Shortwave SFP+ 7023053, FRU Pair 7023059, Option 7101686.

Fibre Channel PCI Identifiers are Vendor ID 0x10df and Device ID 0xe200.
Ethernet PCI Identifiers are Vendor ID 0x10df and Device ID 0xe220.
FCoE PCI Identifiers are Vendor ID 0x10df and Device ID 0xe260.
Vital Product Data VPD=PN is 7023036.

From prtdiag-v.out we can see they are inserted into PCIE2 and PCIE5

======================================== IO Devices =======================================
Slot + Bus Name + Model Max Speed Cur Speed
Status Type Path /Width /Width
-------------------------------------------------------------------------------------------

/SYS/RCSA/PCIE2 PCIE SUNW,emlxs-pciex10df,e200 7101684 8.0GT/x8 8.0GT/x8
  /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0
/SYS/RCSA/PCIE2 PCIE SUNW,emlxs-pciex10df,e200 7101684 8.0GT/x8 8.0GT/x8
  /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1

/SYS/RCSA/PCIE5 PCIE SUNW,emlxs-pciex10df,e200 7101684 8.0GT/x8 8.0GT/x8
  /pci@600/pci@1/pci@0/pci@e/SUNW,emlxs@0
/SYS/RCSA/PCIE5 PCIE SUNW,emlxs-pciex10df,e200 7101684 8.0GT/x8 8.0GT/x8
  /pci@600/pci@1/pci@0/pci@e/SUNW,emlxs@0,1

On explorer I can see instances were offline, when the hotplug command was executed

hotplug disable PCIEx

May 8 08:58:00 server1 su[34226]: [ID 366847 auth.info] 'su oracle' succeeded for root on /dev/???
May 8 08:58:17 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0 (fp9) offline
May 8 08:58:17 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0 (fp9) offline
May 8 08:58:20 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0 (emlxs0) offline
May 8 08:58:20 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0 (emlxs0) offline
May 8 08:58:20 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0 (fp8) offline
May 8 08:58:20 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0 (fp8) offline
May 8 08:58:22 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1 (emlxs1) offline
May 8 08:58:22 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1 (emlxs1) offline

And and some seconds later, they were online, when hotplug command :

hotplug enable PCIEx

May 8 08:58:58 server1 emlxs: [ID 349649 kern.info] [14.0140]emlxs0: NOTICE: 200: Adapter initialization. (Networking is not supported in SLI4, turning it off)
May 8 08:58:58 server1 emlxs: [ID 349649 kern.info] [14.0140]emlxs0: NOTICE: 200: Adapter initialization. (Networking is not supported in SLI4, turning it off)
May 8 08:58:59 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs0:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:58:59 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs0:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B76]emlxs0: NOTICE: 100: Driver attach. (Emulex-S s11-64 sparc v2.90.14.0 (2013.12.07.13.30))
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B76]emlxs0: NOTICE: 100: Driver attach. (Emulex-S s11-64 sparc v2.90.14.0 (2013.12.07.13.30))
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B79]emlxs0: NOTICE: 100: Driver attach. (7101684 Dev_id:e200 Sub_id:e20e Id:48)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B79]emlxs0: NOTICE: 100: Driver attach. (7101684 Dev_id:e200 Sub_id:e20e Id:48)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B86]emlxs0: NOTICE: 100: Driver attach. (Firmware:1.1.43.8 (xe201.grp) Boot:1.1.43.8 Fcode:4.03a1)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1B86]emlxs0: NOTICE: 100: Driver attach. (Firmware:1.1.43.8 (xe201.grp) Boot:1.1.43.8 Fcode:4.03a1)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1BB0]emlxs0: NOTICE: 100: Driver attach. (SLI:4(FIP) MSIX:8 NPIV:0 FCA FC)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1BB0]emlxs0: NOTICE: 100: Driver attach. (SLI:4(FIP) MSIX:8 NPIV:0 FCA FC)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1BBE]emlxs0: NOTICE: 100: Driver attach. (WWPN:10000090FA4B4205 WWNN:20000090FA4B4205)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ B.1BBE]emlxs0: NOTICE: 100: Driver attach. (WWPN:10000090FA4B4205 WWNN:20000090FA4B4205)
May 8 08:59:09 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0, emlxs0
May 8 08:59:09 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0, emlxs0
May 8 08:59:09 server1 genunix: [ID 936769 kern.info] emlxs0 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0
May 8 08:59:09 server1 genunix: [ID 936769 kern.info] emlxs0 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0
May 8 08:59:09 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0, emlxs0
May 8 08:59:09 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0, emlxs0
May 8 08:59:09 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0 (emlxs0) online
May 8 08:59:09 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0 (emlxs0) online
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ 5.06F0]emlxs0: NOTICE: 730: Link reset. (Resetting link...)
May 8 08:59:09 server1 emlxs: [ID 349649 kern.info] [ 5.06F0]emlxs0: NOTICE: 730: Link reset. (Resetting link...)
May 8 08:59:10 server1 emlxs: [ID 349649 kern.info] [ 5.073F]emlxs0: NOTICE: 730: Link reset.
May 8 08:59:10 server1 emlxs: [ID 349649 kern.info] [ 5.073F]emlxs0: NOTICE: 730: Link reset.
May 8 08:59:10 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs0:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:10 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs0:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:26 server1 fcsm: [ID 517869 kern.info] NOTICE: fcsm(9): attached to path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0
May 8 08:59:26 server1 fcsm: [ID 517869 kern.info] NOTICE: fcsm(9): attached to path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0
May 8 08:59:26 server1 genunix: [ID 936769 kern.info] fp9 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0
May 8 08:59:26 server1 genunix: [ID 936769 kern.info] fp9 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0
May 8 08:59:26 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0 (fp9) online
May 8 08:59:26 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0 (fp9) online
May 8 08:59:26 server1 emlxs: [ID 349649 kern.info] [14.0140]emlxs1: NOTICE: 200: Adapter initialization. (Networking is not supported in SLI4, turning it off)
May 8 08:59:26 server1 emlxs: [ID 349649 kern.info] [14.0140]emlxs1: NOTICE: 200: Adapter initialization. (Networking is not supported in SLI4, turning it off)
May 8 08:59:28 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs1:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:28 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs1:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B76]emlxs1: NOTICE: 100: Driver attach. (Emulex-S s11-64 sparc v2.90.14.0 (2013.12.07.13.30))
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B76]emlxs1: NOTICE: 100: Driver attach. (Emulex-S s11-64 sparc v2.90.14.0 (2013.12.07.13.30))
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B79]emlxs1: NOTICE: 100: Driver attach. (7101684 Dev_id:e200 Sub_id:e20e Id:48)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B79]emlxs1: NOTICE: 100: Driver attach. (7101684 Dev_id:e200 Sub_id:e20e Id:48)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B86]emlxs1: NOTICE: 100: Driver attach. (Firmware:1.1.43.8 (xe201.grp) Boot:1.1.43.8 Fcode:4.03a1)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1B86]emlxs1: NOTICE: 100: Driver attach. (Firmware:1.1.43.8 (xe201.grp) Boot:1.1.43.8 Fcode:4.03a1)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1BB0]emlxs1: NOTICE: 100: Driver attach. (SLI:4(FIP) MSIX:8 NPIV:0 FCA FC)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1BB0]emlxs1: NOTICE: 100: Driver attach. (SLI:4(FIP) MSIX:8 NPIV:0 FCA FC)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1BBE]emlxs1: NOTICE: 100: Driver attach. (WWPN:10000090FA4B4206 WWNN:20000090FA4B4206)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ B.1BBE]emlxs1: NOTICE: 100: Driver attach. (WWPN:10000090FA4B4206 WWNN:20000090FA4B4206)
May 8 08:59:38 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0,1, emlxs1
May 8 08:59:38 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0,1, emlxs1
May 8 08:59:38 server1 genunix: [ID 936769 kern.info] emlxs1 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1
May 8 08:59:38 server1 genunix: [ID 936769 kern.info] emlxs1 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1
May 8 08:59:38 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0,1, emlxs1
May 8 08:59:38 server1 pcieb: [ID 586369 kern.info] PCIE-device: SUNW,emlxs@0,1, emlxs1
May 8 08:59:38 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1 (emlxs1) online
May 8 08:59:38 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1 (emlxs1) online
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ 5.06F0]emlxs1: NOTICE: 730: Link reset. (Resetting link...)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ 5.06F0]emlxs1: NOTICE: 730: Link reset. (Resetting link...)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ 5.073F]emlxs1: NOTICE: 730: Link reset.
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [ 5.073F]emlxs1: NOTICE: 730: Link reset.
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs1:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:38 server1 emlxs: [ID 349649 kern.info] [14.14C1]emlxs1:WARNING: 424: Adapter warning. (SLI Port Async Event: Unsupported physical media detected)
May 8 08:59:46 server1 sshd[34454]: [ID 800047 auth.info] Received disconnect from 10.87.168.203: 11: Session Shutdown
May 8 08:59:46 server1 sshd[34559]: [ID 800047 auth.info] SSH: Server;Ltype: Version;Remote: 10.87.168.203-43663;Protocol: 2.0;Client: libssh2_1.4.3
May 8 08:59:47 server1 sshd[34559]: [ID 800047 auth.info] Accepted publickey for bmcbppm from 10.87.168.203 port 43663 ssh2
May 8 08:59:54 server1 fcsm: [ID 517869 kern.info] NOTICE: fcsm(8): attached to path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0
May 8 08:59:54 server1 fcsm: [ID 517869 kern.info] NOTICE: fcsm(8): attached to path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0
May 8 08:59:54 server1 genunix: [ID 936769 kern.info] fp8 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0
May 8 08:59:54 server1 genunix: [ID 936769 kern.info] fp8 is /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0
May 8 08:59:54 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0 (fp8) online
May 8 08:59:54 server1 genunix: [ID 408114 kern.info] /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0,1/fp@0,0 (fp8) online

Unfortunately we get the same warning error

"(SLI Port Async Event: Unsupported physical media detected)" --> this is an indication that the FC HBA recognizes SFPs as not supported

I would expect to see a Link UP message, but there is not,
probably because of the previous error "SLI Port Async Event: Unsupported physical media detected"

Solaris recognize the FC HBA as configured, all looks fine, emlxs and fp driver binds to it, except for the Link that does not come UP, from cfgadm -alv :

PCIE2 connected configured ok Location: PCIE2
May 8 08:58 fibre/hp n /devices/pci@300/pci@1/pci@0/pci@c:PCIE2

PCIE5 connected configured ok Location: PCIE5
May 7 15:20 fibre/hp n /devices/pci@600/pci@1/pci@0/pci@e:PCIE5

On new fcinfo I can see it recognize the 16GB capability, and present a high number of "Invalid Tx Word Count: " on two ports,
as if only two FC cables were connected, but failed to link up

HBA Port WWN: 10000090fa4b3cbb
  Port Mode: Initiator
  Port ID: 0
  OS Device Name: /dev/cfg/c10
  Manufacturer: Emulex
  Model: 7101684
  Firmware Version: 7101684 1.1.43.8
  FCode/BIOS Version: Boot:1.1.43.8 Fcode:4.03a1
  Serial Number: 4925382+13340000MG
  Driver Name: emlxs
  Driver Version: 2.90.14.0 (2013.12.07.13.30)
  Type: unknown
  State: offline
  Supported Speeds: 4Gb 8Gb 16Gb
  Current Speed: not established
  Node WWN: 20000090fa4b3cbb
  Link Error Statistics:
  Link Failure Count: 8
  Loss of Sync Count: 1258
  Loss of Signal Count: 4
  Primitive Seq Protocol Error Count: 0
  Invalid Tx Word Count: 8352<<---???
  Invalid CRC Count: 0
  NPIV Not Supported
HBA Port WWN: 10000090fa4b3cbc
  Port Mode: Initiator
  Port ID: 0
  OS Device Name: /dev/cfg/c11
  Manufacturer: Emulex
  Model: 7101684
  Firmware Version: 7101684 1.1.43.8
  FCode/BIOS Version: Boot:1.1.43.8 Fcode:4.03a1
  Serial Number: 4925382+13340000MG
  Driver Name: emlxs
  Driver Version: 2.90.14.0 (2013.12.07.13.30)
  Type: unknown
  State: offline
  Supported Speeds: 4Gb 8Gb 16Gb
  Current Speed: not established
  Node WWN: 20000090fa4b3cbc
  Link Error Statistics:
  Link Failure Count: 0
  Loss of Sync Count: 0
  Loss of Signal Count: 0
  Primitive Seq Protocol Error Count: 0
  Invalid Tx Word Count: 0
  Invalid CRC Count: 0
  NPIV Not Supported
HBA Port WWN: 10000090fa4b4205
  Port Mode: Initiator
  Port ID: 0
  OS Device Name: /dev/cfg/c8
  Manufacturer: Emulex
  Model: 7101684
  Firmware Version: 7101684 1.1.43.8
  FCode/BIOS Version: Boot:1.1.43.8 Fcode:4.03a1
  Serial Number: 4925382+13340000OG
  Driver Name: emlxs
  Driver Version: 2.90.14.0 (2013.12.07.13.30)
  Type: unknown
  State: offline
  Supported Speeds: 4Gb 8Gb 16Gb
  Current Speed: not established
  Node WWN: 20000090fa4b4205
  Link Error Statistics:
  Link Failure Count: 0
  Loss of Sync Count: 0
  Loss of Signal Count: 0
  Primitive Seq Protocol Error Count: 0
  Invalid Tx Word Count: 0
  Invalid CRC Count: 0
  NPIV Not Supported
HBA Port WWN: 10000090fa4b4206
  Port Mode: Initiator
  Port ID: 0
  OS Device Name: /dev/cfg/c9
  Manufacturer: Emulex
  Model: 7101684
  Firmware Version: 7101684 1.1.43.8
  FCode/BIOS Version: Boot:1.1.43.8 Fcode:4.03a1
  Serial Number: 4925382+13340000OG
  Driver Name: emlxs
  Driver Version: 2.90.14.0 (2013.12.07.13.30)
  Type: unknown
  State: offline
  Supported Speeds: 4Gb 8Gb 16Gb
  Current Speed: not established
  Node WWN: 20000090fa4b4206
  Link Error Statistics:
  Link Failure Count: 0
  Loss of Sync Count: 0
  Loss of Signal Count: 0
  Primitive Seq Protocol Error Count: 0
  Invalid Tx Word Count: 25352423 <<---???
  Invalid CRC Count: 0
  NPIV Not Supported

When the SFPs are not supported by the FC HBA (ie third party SFPs)
we may get this error on the console when the server boot:

WARNING: emlxs0: Incompatible optics - Replace with compatible optics for card to function.
WARNING: emlxs1: Incompatible optics - Replace with compatible optics for card to function.

Changes

The new 16GB Emulex FC HBAs were installed initially in the T5-8 server with non valid / supported SFPs.
Later, while Solaris was up and running, customer removed and inserted new supported 16GB SFPs (Oracle PN: 7023053), but the FC HBA did not recognized it, Solaris server reboot was not done.

Cause

The Emulex FC HBA reads the VPD of the new SFP when the FC HBA is powered on.
If the hotplug commands did not power off the FC HBA, that could explain why new SFPs are not recognized : (SLI Port Async Event: Unsupported physical media detected)

This FC HBA is operating in the correct mode (Fiber Channel) , so we can discard this is a wrong operation mode,

from Oracle Handbook. "Fibre Channel PCI Identifiers are Vendor ID 0x10df and Device ID 0xe200."

This is also explained on Bug 17628896 : ILOM NEEDS TO DISTINGUISH GANYMEDE FC OR FCOE IN PCI DEVICES LIST,

@ { {0x10DF, 0xE200, 0x10DF, 0xE20E}, ... "7101684", "Sun Storage 16 Gb Fibre Channel PCIe Universal HBA, Emulex" }, /* (Emulex Ganymede - FC) */
@ { {0x10DF, 0xE220, 0x10DF, 0xE20E}, ... "7101684", "Sun Storage 16 Gb Fibre Channel PCIe Universal HBA, Emulex" }, /* (Emulex Ganymede - NIC) */
@ { {0x10DF, 0xE260, 0x10DF, 0xE20E}, ... "7101684", "Sun Storage 16 Gb Fibre Channel PCIe Universal HBA, Emulex" }, /* (Emulex Ganymede - FCoE) */
@ .
@ So we have three different flavors , but each flavor (FC/NIC/FCoE) have the same description.

where we have: {vendor-id, device-id, subsystem-vendor-id , subsystem-id}

We can confirm operation mode on prtpicl -v by looking at the device-id value:

  SUNW,emlxs (scsi-fcp, 2ac00002304)
  :DeviceID 0
  :UnitAddress 0
  :vendor-id 0x10df
  :device-id 0xe200 --> this shows this is FC mode
  :revision-id 0x10
  :subsystem-vendor-id 0x10df
  :subsystem-id 0xe20e
  :class-code 0xc0400
  :cache-line-size 0x10
  :interrupts 00 00 00 01
  :port-type PCIE-Endpoint
  :#vfs 0x8
  :initial-vfs 0x8
  :total-vfs 0x8
  :first-vf-offset 0x2
  :vf-stride 0x1
  :fcode-rom-offset 0xfe00
  :firmware-version 1.1.43.8
  :fcode-version 4.03a1
  :manufacture Emulex
  :device_type scsi-fcp

emlxs driver is binded to it:
  :devfs-path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0
  :driver-name emlxs
  :binding-name pciex10df,e200
  :bus-addr 0
  :instance 0
  :_class scsi-fcp
  :name SUNW,emlxs

same as correspondent fp instance:
  fp (devctl, 2ac00002356)
  :numaio-lgrps 0x1
  :scsi-enumeration 0
  :scsi-options 0x107ff8
  :scsi-reset-delay 0xbb8
  :scsi-watchdog-tick 0xa
  :scsi-selection-timeout 0xfa
  :scsi-tag-age-limit 0x2
  :scsi-poll-timeout 0xa
  :scsi-transport-timeout 0x3c
  :scsi-initiator-id 0x7
  :initiator-interconnect-type FABRIC
  :mpxio-disable no
  :load-ulp-list (2ac0000236aTBL)
  | 1 |
  | fcp |
  :scsi-binding-set fcp
  :class fibre-channel
  :port 0
  :initiator-port 10000090fa4b4205
  :initiator-node 20000090fa4b4205
  :pm-hardware-state needs-suspend-resume
  :pm-components (2ac0000235fTBL)
  | NAME=FC Port |
  | 0=Port Down |
  | 1=Port Up |
  :devfs-path /pci@300/pci@1/pci@0/pci@c/SUNW,emlxs@0/fp@0,0
  :driver-name fp
  :binding-name fp
  :bus-addr 0,0
  :instance 9
  :_class devctl
  :name fp

Based on the FC HBA Installation Guide (Sun Storage 16 Gb Fibre Channel PCIe Universal Host Bus Adapter, Emulex , Installation Guide For HBA Model 7101684)
from page 19 and 20 "Changing the Operating Protocol Mode" :

Note – The transceivers must be plugged into the HBA before the HBA is booted.
The Sun Storage 16 Gb Fibre Channel PCIe Universal HBA, Emulex does not support changing the operating mode by “hot swapping” the transceivers.

Note - Only supported transceivers are allowed. If invalid or unapproved transceivers are plugged in and the board is booted,
the links are not allowed to come up.

New FINISAR SFPs are supported with latest Universal HBA firmware 10.6.230.0 or greater, see Document ID 2119504.1

Solution

The following commands were enough to make it working on a Solaris 11.1 T5-8 server (with the supported 16GB SFPs already installed), the removal of the FC HBA PCIE card was not necessary.

hotplug disable PCIEx
cfgadm -c disconnect pciex
hotplug enable PCIEx

Notice, on this case, the FC HBA was already operating in FC mode (The default operating mode is two-port 16 Gb FC)

On other servers, it may be required to unconfigure the cards using hotplug disable and cfgadm disconnect
physically remove the card, swapped the sfp's and then inserted the cards back in the box.

NOTE. If above does not work, a Solaris server reboot may be required, read carefully all steps and possible scenarios

described on page 19 and 20 "Changing the Operating Protocol Mode"

of (Sun Storage 16 Gb Fibre Channel PCIe Universal Host Bus Adapter, Emulex , Installation Guide For HBA Model 7101684)

References

<NOTE:1960235.1> - How To Verify Transceiver Type Installed In Oracle Sun Storage Branded Emulex 16Gb Fibre Channel / 10Gb FCoE Universal HBA On Solaris 11.2 And Later
<NOTE:1581956.1> - Sun Storage 16 Gb Fibre Channel PCIe Universal Host Bus Adapter (Emulex) Operating Mode Configuration
<NOTE:2119504.1> - emlxs WARNING: 424: Adapter warning. (SLI Port Async Event: Physical media not detected)

Attachments

This solution has no attachment