CRC Errors On ISL 8GB Port On Brocade300 - Vc_translation_link_init = 0 And Fillword = 0 = IDLE

Asset ID:	1-72-1613159.1
Update Date:	2014-01-14
Keywords:

Solution Type Problem Resolution Sure

Solution 1613159.1 : CRC Errors On ISL 8GB Port On Brocade300 - Vc_translation_link_init = 0 And Fillword = 0 = IDLE_IDLE

Applies to:

Brocade 300 Switch - Version All Versions and later
Information in this document applies to any platform.

Symptoms

We have a lot of CRC and Enc Out errors on ISL ports 14 and 15, 8GB long wave SFP used to connect 4 Brocade 300 switches FOS 6.4.3c

This shows the type of SFP 8GB Long Wave used:

sfpshow -all       :

=============
Port 14:
=============
Identifier: 3    SFP
Connector:   7    LC
Transceiver: 5401001200000000 200,400,800_MB/s SM lw Long_dist
Encoding:    1    8B10B
Baud Rate:   85   (units 100 megabaud)
Length 9u:   10   (units km)
Length 9u:   100 (units 100 meters)
Length 50u: 0    (units 10 meters)
Length 62.5u:0    (units 10 meters)
Length Cu:   0    (units 1 meter)
Vendor Name: BROCADE
Vendor OUI: 00:05:1e
Vendor PN:   57-1000027-01
Vendor Rev: A
Wavelength: 1310 (units nm)
Options:     003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max:      0
BR Min:      0
Serial No:   UDN11317B100137
Date Code:   130429
DD Type:     0x68
Enh Options: 0xb0
Status/Ctrl: 0xa2
Alarm flags[0,1] = 0x5, 0x40
Warn Flags[0,1] = 0x5, 0x40
                                          Alarm                  Warn
                                      low        high       low         high
Temperature: 40      Centigrade     -15         100        -10         95
Current:     27.906 mAmps          5.000       80.000     10.000      70.000
Voltage:     3267.7 mVolts         2970.0      3600.0     3100.0      3470.0
RX Power:    -3.4    dBm (456.7 uW) 25.1   uW   1412.5 uW 39.8   uW   891.2 uW
TX Power:    -2.6    dBm (550.7 uW) 89.1   uW   1412.5 uW 141.3 uW   891.3 uW

Changes

Usually the crc and enc out errors are caused by dirty cables, sfps or patch panels.
However, customer state that the sfps have been changed and all the cables have been checked and have no issues.

Cause

1) There is a defect that can explain CRC errors observed:

The sfps in ports 14 and 15 begin with s/n UDN1xxxx.
There is a defect (DEFECT000414944) which these particular sfps caused CRC errors on 8G links.
This has been corrected in FOS v 6.4.3e, v7.0.2a and v7.1.0. (Check the release notes)

2) In addition to that, there are some port parameter that are recommended by Brocade:

-----------
All of the 8GB ports do not require the use of fillword 3.

2GB and 4GB devices should be set to 0.

ISLs should use fillword 0 and if VC_Init is Enabled CR Recovery and QOS must be disabled.

Storage devices even 8GB do not require fillword 3, recommend to start with 0.

8GB HBAs may have to use fillword 3 but this is not an always have to rule.

The Enc_Out or Enc_In should be resolved by making these changes and checking the porterrshow output.

Once the Enc_Out and Enc_In errors are minimized the CRCs will be minimized as well.
--------------

3) Another possible cause , that affect only to Brocade 300, is described by Brocade Technical Support Bulletin TSB-2012-011-O, it is a best practice,

and it does not apply on our case as all SFPs are 8GB (short wave and long wave)

Solution

1. Customer upgraded to latest v7.1.1a with applications up and running, on live systems, first will apply patch 149139-02 to go to v7.0.2c and then 150481-02: v7.1.1a

For more information about patches and FOS versions see:
Oracle Fibre Channel (FC) SAN Switch Firmware Patch Matrix for Firmware Upgrade - Brocade - Cisco - McData - Qlogic (Doc ID 1507746.1)

*** With this only action CRC error decreased significantly.

2. After upgrade, customer recreated ISL port (less than 1KM distance between sites)

- They enabled on ISL port 15 on all switches:

Wed Dec 18 17:55:09 2013 admin, 172.17.5.107, portdisable 15
Wed Dec 18 17:55:22 2013 admin, 172.17.5.107, portcfglongdistance 15 L0
Wed Dec 18 17:55:35 2013 admin, 172.17.5.107, portcfgfillword 15 0
Wed Dec 18 17:55:46 2013 admin, 172.17.5.107, portcfgcreditrecovery --disable 15
Wed Dec 18 17:55:54 2013 admin, 172.17.5.107, portenable 15

*** With this second action no more crc or enc out errors were observed.

At the end, customer enabled a second ISL port 14 :

Wed Dec 18 18:27:40 2013 admin, 172.17.5.107, portdisable 14
Wed Dec 18 18:27:50 2013 admin, 172.17.5.107, portcfglongdistance 14 L0
Wed Dec 18 18:27:57 2013 admin, 172.17.5.107, portcfgfillword 14 0
Wed Dec 18 18:28:07 2013 admin, 172.17.5.107, portcfgcreditrecovery --disable 14
Wed Dec 18 18:28:29 2013 admin, 172.17.5.107, portenable 14

3. Final ISL and port configuration was:

islshow :
1: 14-> 14 10:00:00:27:f8:98:e0:48 1 brocade1-A sp: 8.000G bw: 8.000G
2: 15-> 15 10:00:00:27:f8:98:e0:48 1 brocade1-A sp: 8.000G bw: 8.000G

On port 14 and 15 :
Long Distance disabled (L0) , VC Link Init 0 , Fillword 0 and Credit Recovery disabled

portcfgshow :

Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
----------------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+---
Speed AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN
Fill Word(On Active) 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0
Fill Word(Current) 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0
AL_PA Offset 13 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Trunk Port ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. LE .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. ON .. ..
Locked L_Port - - - - - - - - - - - - - - - -
Locked G_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
LOS TOV enable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
QOS Port AE AE AE AE AE AE AE AE AE AE AE AE AE .. .. AE
Mirror Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Rate Limit .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Credit Recovery ON ON ON ON ON ON ON ON ON ON ON ON ON ON .. ..
Fport Buffers .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Port Auto Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
CSCTL mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Fault Delay 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

No more errors observed:

porterrshow :
  frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs
  tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err
  0: 607.1m 327.2m 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  1: 3.7g 999.6m 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  2: 1.2g 3.5g 0 0 0 0 0 0 105 1 0 0 0 0 0 0 0 0
  3: 3.3g 2.4g 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0
  4: 1.5m 18 0 0 0 0 0 0 50 0 0 0 0 0 0 0 0 0
  5: 321.7m 279.9m 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  6: 281.4m 321.7m 0 0 0 0 0 0 50 0 0 0 0 0 0 0 0 0
  7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  9: 2.4g 2.6g 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14: 7.6m 2.3m 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15: 11.3m 3.6m 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0

References

<NOTE:1003754.1> - Brocade: What logs are required to troubleshoot a Brocade switch?
<NOTE:1519015.1> - How to Collect Brocade FC Switch logs
<NOTE:1449866.2> - Information Center: Brocade 300 Switch
<NOTE:1507746.1> - Oracle Fibre Channel (FC) SAN Switch Firmware Patch Matrix for Firmware Upgrade - Brocade - Cisco - McData - Qlogic

Attachments

This solution has no attachment