Unable to establish FCoE connection using second port of CNA to Cisco Nexus 5548 switch

Asset ID:	1-72-2087929.1
Update Date:	2015-12-14
Keywords:

Solution Type Problem Resolution Sure

Solution 2087929.1 : Unable to establish FCoE connection using second port of CNA to Cisco Nexus 5548 switch

Applies to:

SPARC T5-2 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Customer has 3 Servers.
Each Server has 2 Emulex CNA cards.
On each server, 1 port from each CNA card is connected to a Cisco 5548 switch using FCoE.
The Cisco 5548 is connected to a FC switch, which is then connected to a FC Storage Array.
The FC Storage Array presents FC LUNs to the Server via FCoE CNA ports.
This configuration is up and running without fault.

The issue is that the customer is unable to establish a FC connection to the FC switch beyond the Nexus Cisco 5548 switch using the remaining (second) port on each CNA card.
Due to this they are unable to present Storage from the FC Array to the second CNA port.

In messages numerous events of the following type are logged against the new data path:

Nov 16 03:10:46 xxxxxxxxx emlxs: [ID 349649 kern.info] [ 5.062D]emlxs7: NOTICE: 720: Link up. (10Gb, fabric, initiator)
Nov 16 03:10:46 xxxxxxxxx emlxs: [ID 349649 kern.info] [ 5.03EF]emlxs7: NOTICE: 710: Link down.

and:

messages.0:Nov 15 23:38:44 xxxxxxxxx fctl: [ID 517869 kern.warning] WARNING: fp(8)::Topology Unknown, Offlining the port..
messages.0:Nov 16 02:55:20 xxxxxxxxx fctl: [ID 517869 kern.warning] WARNING: fp(12)::Topology Unknown, Offlining the port..

From the fptrace the following is logged:

[Mon Nov 16 2015 14:02:00:725:272:222] 112=>fp(9)::fp_fabric_login Begin; port=6
002615e000, job=600288e76c8
[Mon Nov 16 2015 14:02:00:725:327:992] 113=>fp(9)::fp_fabric_login End; port=600
2615e000, job=600288e76c8
[Mon Nov 16 2015 14:02:00:823:914:945] 114=>fp(9)::fp_flogi_intr; port=6002615e0
00, pkt=600287f4d90, state=3
[Mon Nov 16 2015 14:02:00:823:953:457] 115=>fp(9)::fp_ulp_statec fired; Port=600
2615e000,state=0, len=0
[Mon Nov 16 2015 14:02:03:821:319:232] 116=>fp(8)::fp_fabric_login Begin; port=6
0027fe3000, job=600288e7738
[Mon Nov 16 2015 14:02:03:821:348:190] 117=>fp(8)::fp_fabric_login End; port=600
27fe3000, job=600288e7738
[Mon Nov 16 2015 14:02:04:042:985:737] 118=>fp(8)::fp_flogi_intr; port=60027fe30
00, pkt=600288ecd90, state=3
[Mon Nov 16 2015 14:02:04:042:998:945] 119=>fp(8)::Topology Unknown, Offlining t
he port..

Cause

This is a limitation with how Cisco implements FCoE with Port Channels and vPC.

See the following image for supported topologies:

FCoE Port Channel Topology

Cisco does not support binding of Virtual Fiber Channel (vfc) interfaces from a server with multiple links to the same Nexus 5x00 (either in Port-Channel or vPC).
VFC can only be bound to 1 interface.

Port Channels were also in use on these CNA interfaces and the configuration being implemented was not a supported topology.
Because of this limitation in the FCoE section of the data path the customer was unable to establish a FC connection on the second CNA port, hence the numerous Link up/Link down events.

Solution

Customer should engage Cisco directly for assistance in verifying that the topology being implemented is supported for the version of firmware installed on the Cisco Nexus 5548.

References

<BUG:17867961> - VLAN ID BIGGER THAN 255 IS NOT SET APPROPRIATELY IN FIP SOLICITATION FRAME
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices

Attachments

This solution has no attachment