Brocade 8Gb Switches may experience high er_bad

Asset ID:	1-72-1117643.1
Update Date:	2016-06-27
Keywords:

Solution Type Problem Resolution Sure

Solution 1117643.1 : Brocade 8Gb Switches may experience high er_bad_os counts

Applies to:

Brocade 300 Switch - Version Not Applicable and later
Information in this document applies to any platform.
Applies to:
Brocade Switches with Fabric OS v6.x and Oracle/STK 8G devices but could effect Oracle/STK devices below 8G (see note below).

Symptoms

The following symptoms maybe a result of the fillword parameter being incorrectly set on the Brocade Switch if they occur against an 8Gb (or above) device.

1. Rapidly increasing "er_bad_os" counts logged by the Brocade Switch.

2. The following Warnings being written to the Brocade fabric log.

2016/06/12-00:03:06, [AN-1003], 5577, FID 128, WARNING, SANB-ERPDR, Latency bottleneck on F-Port 8. 0.00 pct. of 300 secs. affected. Avg. delay 0 us. Avg. slowdown 0.
2016/06/12-00:03:10, [AN-1005], 5579, FID 128, INFO, SANB-ERPDR, Port 8 has Latency bottleneck cleared.

2016/06/03-11:18:25, [C2-1014], 5523, CHASSIS, WARNING, Brocade300, Link Reset on Port S0,P8(15) vc_no=0 crd(s)lost=6 auto trigger.
2016/06/03-17:44:29, [C2-1014], 5527, CHASSIS, WARNING, Brocade300, Link Reset on Port S0,P8(15) vc_no=0 crd(s)lost=6 auto trigger.

3. The following events being written to the host messages file.

Jun 12 00:04:23 server01 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jun 12 00:04:23 server01 /scsi_vhci/ssd@g6000b08414bxxxxxxx23632373500000 (ssd4): Command Timeout on path fp4/ssd@w2100000e1e27XXXX,0
Jun 12 00:04:23 server01 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jun 12 00:04:23 server01 /scsi_vhci/ssd@g6000b08414bxxxxxxx23632373500000 (ssd4): Command Timeout on path fp4/ssd@w2100000e1e27XXXX,0
Jun 12 00:04:23 server01 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Jun 12 00:04:23 server01 /scsi_vhci/ssd@g6000b08414bxxxxxxx23632373500007

This behavior was observed on a Solaris 11.2 GA T5-2 primary domain server with 16GB Oracle FC HBAs connected to Brocade 300 switches to access an Oracle FS-1 storage array.

Changes

The "portcfgfillword" command has changed in FOS:

In Fabric OS v6.3.0 ---the "portcfgfillword" command was added and the fillword could be hard set to:
0 |-idle-idle Sets IDLE mode in the Link Init and IDLE as the fill word (default), or
1 | -arbff-arbff Sets ARB(ff) in the Link Init and ARB(ff) as the fill word.

In Fabric OS v6.3.1 --- the "portcfgfillword" command was modified to allow a setting of one of the following:

MODE:
0 | -idle-idle Sets IDLE mode in the Link Init and IDLE as the fill word (default).
1 | -arbff-arbff Sets ARB(ff) in the Link Init and ARB(ff) as the fill word.
2 | -idlef-arbff Sets IDLE mode in the Link Init and ARB(ff) as the fill word.
3 | -aa-then-ia Attempts hardware arbff-arbff (mode 1) first. If the attempt fails to go into active state, this command executes software idle-arb (mode 2).

Mode 3 is the preferable to modes 1 and 2 as it captures more cases.

Cause

Solution

1. Check the fillword mode setting for a particular port use the portcfgshow command:

admin> portcfgshow 1/0
Area Number: 9
Speed Level: AUTO(HW)
Fill Word: 0(Idle-Idle)

2) If appropriate, change the fillword mode setting using the portcfgfillword command:

Portcfgfillword <port number> <Fillword value>

For example.

portCfgFillWord 1/0 3

Note:
- For existing fabrics in the field with FOS v6.3.0x, one needs to set the portcfgfillword setting to option 1.
- For switches with FOS v6.3.1x and later, Engineering would recommend that they set the portcfgfillword for a port to option 3.
- With option 1, it will do ONLY ARB(ff) for the fill word, which is what the 8G devices expect.
- With option 3, this is "kind of" an auto-negotiate setting so that you can have any device connected to it.

Note: This command disables and re-enables the port.

Important - potential problem - for non-8G devices
** With mode 0 & 1 used and not mode 3 - if a customer ever connects a non 8G device to that port
they would have the same issue.

Attachments

This solution has no attachment