Brocade - High Increase of Time TX Credit Zero tim_txcrd_z

Asset ID:	1-72-2024681.1
Update Date:	2016-11-07
Keywords:

Solution Type Problem Resolution Sure

Solution 2024681.1 : Brocade - High Increase of Time TX Credit Zero tim_txcrd_z - Not Enough Buffer Credits

Applies to:

Brocade 300 Switch - Version All Versions and later
Information in this document applies to any platform.

Symptoms

There are two Brocades 300 (FOS 7.1.1a) on a Fabric (each switch on a different site) connected by two ISL (8GB , port 14 and 15) configured as L0

Index Port Address Media Speed       State   Proto
==================================================

14 14   010e00   id    N8       Online      FC E-Port 10:00:00:27:f8:93:XX:XX "brocade2"
15 15   010f00   id    N8       Online      FC E-Port 10:00:00:27:f8:93:XX:XX "brocade2" (upstream)

brocade1:admin> islshow
1: 14-> 15 10:00:00:27:f8:93:XX:XX 3 brocade2 sp: 8.000G bw: 8.000G
2: 15-> 14 10:00:00:27:f8:93:XX:XX 3 brocade2 sp: 8.000G bw: 8.000G

The distance between both switches on the fabric was 800 meter, but it has changed recently to 1400 meters.
Since then, there is a high increase of Time TX Credit Zero , tim_txcrd_z is increasing very quickly on ISL E ports 14 and 15

Example with port 14:

brocade1:admin> date; portstatsshow 14 | grep tim_txcrd_z
Tue Jun 2 09:10:52 Localtime 2015
tim_txcrd_z 271710676 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0 0 488631 8426304
tim_txcrd_z_vc 4- 7: 9771658 253024083 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0

Some time later (78 secs on this case) collect the same output:

brocade1:admin> date; portstatsshow 14 | grep tim_txcrd_z
Tue Jun 2 09:12:10 Localtime 2015
tim_txcrd_z             271964488   Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0           0           496423 8585307
tim_txcrd_z_vc 4- 7: 9858675     253024083   0 0
tim_txcrd_z_vc 8-11: 0           0           0 0
tim_txcrd_z_vc 12-15: 0           0           0 0

The average number of ticks per second on this port 14 is

271964488 - 271710676 = 253812 --> /78 secs = 3254 ticks/sec on port 14

This rate of tim_txcrd_z is not normal , this can lead to perfomance issues.

Q1. What can we do to lower the rate of tim_txcrd_z ? If the proposed action is to increase the Buffer Credits, what are recommended values?
Q2. How can we increase credit buffers on port 14 and 15?
In order to increase the buffer credits on port 14 and 15, you should change the port configuration to long distance mode.

When trying to change to long distance LE mode got an error.

brocade1:admin> portcfglongdistance 14 LE 1
portcfglongdistance vc_translation_link_init conflicts with
portcfgfillword configuration. Valid configurations are:
vc_translation_link_init = 0, fillword = IDLE_IDLE or
vc_translation_link_init = 1 fillword=ARB_ARB, IDLE_ARB, AA_THEN_IA

This error is because these configuration values needs to be changed before running portcfglongdistance, this is explained on Brocade documentation

- "Fabric OS Administrator’s Guide FOS 7.2.0" Chapter 5 "Buffer-to-Buffer Credits and Credit Recovery" and Chapter 25 "Managing Long-Distance Fabrics" , from page 90

To configure an 8 Gbps (and 8 Gbps only) connection between a device and a switch, use the
portCfgFillWord command. This command provides the following configuration options:
• Mode Link Init/Fill Word
• Mode 0 IDLE/IDLE
• Mode 1 ARBF/ARBF
• Mode 2 IDLE/ARBF
• Mode 3 If ARBF/ARBF fails, use IDLE/ARBF
ATTENTION
Although this setting only affects devices logged in at 8 Gbps, changing the mode is disruptive
regardless of the speed at which the port is operating.

- "Fabric OS Command Reference Supporting Fabric OS v7.1.0" see command portCfgLongDistance page 707

vc_translation_link_init
Specifies the fill words used on long distance links. When set to 1, the link uses
ARB fill words (default). When set to 0, the link uses IDLE fill words. The IDLE fill
word option is not compatible with QoS configured links and Credit Recovery
enabled links. You must disable these features before configuring long distance
IDLE fill words.

Use command portcfgshow to see configured values on each port, ie on this case:

portcfgshow :

Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
----------------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+---
Speed AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN
Fill Word(On Active) 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0
Fill Word(Current) 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0
AL_PA Offset 13 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Trunk Port ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. LE .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. ON .. ..
Locked L_Port - - - - - - - - - - - - - - - -
Locked G_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
LOS TOV enable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
QOS Port AE AE AE AE AE AE AE AE AE AE AE AE AE .. AE AE
Mirror Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Rate Limit .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Credit Recovery ON ON ON ON ON ON ON ON ON ON ON ON ON ON .. ..
Fport Buffers .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Port Auto Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
CSCTL mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Fault Delay 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Other than that, there are no other errors on these ports, ie on brocade1 switch:

porterrshow        :
          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy c3timeout    pcs
       tx     rx      in    err    g_eof shrt   long   eof     out   c3    fail    sync   sig                 tx    rx    err

14:    2.0g 758.5m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
15:    2.0g 758.8m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

HW infrastructure between both sites has been checked, fibre attenuation is between expected values, no problem found here.

There are long distance SFP on these ports, no issue found with SFPs, ie:

=============
Port 14:
=============
Identifier: 3    SFP
Connector:   7    LC
Transceiver: 5401001200000000 2,4,8_Gbps SM lw Long_dist
Encoding:    1    8B10B
Baud Rate:   85   (units 100 megabaud)
Length 9u:   10   (units km)
Length 9u:   100 (units 100 meters)
Length 50u (OM2): 0    (units 10 meters)
Length 50u (OM3): 0    (units 10 meters)
Length 62.5u:0    (units 10 meters)
Length Cu:   0    (units 1 meter)
Vendor Name: BROCADE
Vendor OUI: 00:05:1e
Vendor PN:   57-1000027-01
Vendor Rev: A
Wavelength: 1310 (units nm)
Options:     003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max:      0
BR Min:      0
Serial No:   UDN11317B100137
Date Code:   130429
DD Type:     0x68
Enh Options: 0xb0
Status/Ctrl: 0xa2
Alarm flags[0,1] = 0x5, 0x0
Warn Flags[0,1] = 0x5, 0x0

Changes

The distance between both switches on the fabric was 800 meter, but it has changed recently to 1400 meters.

Cause

The Buffer Credit on port 14 and 15 are 8, as these are configured as E ports L0

Site1

brocade1:admin> portbuffershow
User Port Lx Max/Resv Avg Buffer Usage & FrameSize Buffer Needed Link Remaining
Port Type Mode Buffers Tx Rx Usage Buffers Distance Buffers
---- ---- ---- ------- ---------------------------- ------ ------- --------- ----------
  0 F - 8 2(2112) 1(1280) 8 - -
  1 F - 8 1(1848) - (1004) 8 - -
  2 F - 8 - ( 796) 1(1992) 8 - -
  3 F - 8 - ( 964) 1(1648) 8 - -
  4 F - 8 - ( - ) - ( - ) 8 - -
  5 F - 8 - ( - ) - ( - ) 8 - -
  6 F - 8 - ( - ) - ( - ) 8 - -
  7 F - 8 - ( 72) 1(1740) 8 - -
  8 - 8 - ( - ) - ( - ) 0 - -
  9 - 8 - ( - ) - ( - ) 0 - -
  10 - 8 - ( - ) - ( - ) 0 - -
  11 - 8 - ( - ) - ( - ) 0 - -
  12 - 8 - ( - ) - ( - ) 0 - -
  13 LE 46 - ( - ) - ( - ) 0 - -
  14 E - 8 1(1932) - ( 792) 17 17 <2km
  15 E - 8 1(1924) - ( 812) 17 17 <2km
  16 - 8 - ( - ) - ( - ) 0 - -
  17 - 8 - ( - ) - ( - ) 0 - -
  18 - 8 - ( - ) - ( - ) 0 - -
  19 - 8 - ( - ) - ( - ) 0 - -
  20 - 8 - ( - ) - ( - ) 0 - -
  21 - 8 - ( - ) - ( - ) 0 - -
  22 - 8 - ( - ) - ( - ) 0 - -
  23 - 8 - ( - ) - ( - ) 0 - - 428

Site2

brocade2:admin> portbuffershow

User Port Lx Max/Resv Avg Buffer Usage & FrameSize Buffer Needed Link Remaining
Port Type Mode Buffers Tx Rx Usage Buffers Distance Buffers
---- ---- ---- ------- ---------------------------- ------ ------- --------- ----------
  0 F - 8 1(1852) - ( 56) 8 - -
  1 F - 8 1(1904) - ( 140) 8 - -
  2 F - 8 - ( - ) - ( - ) 8 - -
  3 F - 8 - ( - ) - ( - ) 8 - -
  4 F - 8 - ( - ) - ( - ) 8 - -
  5 F - 8 - ( - ) - ( - ) 8 - -
  6 F - 8 - ( - ) - ( - ) 8 - -
  7 - 8 - ( - ) - ( - ) 0 - -
  8 - 8 - ( - ) - ( - ) 0 - -
  9 - 8 - ( - ) - ( - ) 0 - -
  10 - 8 - ( - ) - ( - ) 0 - -
  11 - 8 - ( - ) - ( - ) 0 - -
  12 - 8 - ( - ) - ( - ) 0 - -
  13 LE 46 - ( - ) - ( - ) 0 - -
  14 E - 8 - ( 52) 1(1852) 17 17 <2km
  15 E - 8 - ( 64) 1(1804) 17 17 <2km
  16 - 8 - ( - ) - ( - ) 0 - -
  17 - 8 - ( - ) - ( - ) 0 - -
  18 - 8 - ( - ) - ( - ) 0 - -
  19 - 8 - ( - ) - ( - ) 0 - -
  20 - 8 - ( - ) - ( - ) 0 - -
  21 - 8 - ( - ) - ( - ) 0 - -
  22 - 8 - ( - ) - ( - ) 0 - -
  23 - 8 - ( - ) - ( - ) 0 - - 428

As the distance on the ISL link has increased, the problem is that we have not enough buffer credits on the ISL ports.

On this case, tim_txcrd_z counters are increasing due to incorrectly configured ports.

Solution

Proceeding with below plan of action will be good starting point with resolving all issues.

Schedule maintenance window for below plan of action and perform that when mentioned fabric is redundant.

Use the portCfgLongDistance command to support long-distance links and to allocate sufficient
numbers of full-size frame buffers on a specific port. Changes made by this command are
persistent across switch reboots and power cycles.

Extended Mode (LE) — LE configures the distance for an E_Port when that distance is greater
than 5 km and up to 10 km. LE does not require an Extended Fabrics license. The baseline for
the buffer credit calculation is one buffer credit per km at 2 Gbps. This allocation yields the
following values for 10 km:
- 10 buffer credits per port at 2 Gbps.
- 20 buffer credits per port at 4 Gbps.
- 40 buffer credits per port at 8 Gbps.
- 50 buffer credits per port at 10 Gbps
- 80 buffer credits per port at 16 Gbps

(1) Configuring an extended ISL

Before configuring an extended ISL, ensure that the following conditions are met:

a/ The ports on both ends of the ISL are operating at the same port speed, and can be configured
for the same distance_level without compromising local switch performance.

NOTE
A long-distance link also can be configured to be part of a trunk group. Two or more
long-distance links in a port group form a trunk group when they are configured for the same
speed and distance, and their link distances are nearly equal. For information on trunking
concepts and configurations, refer to Chapter 22, “Managing Trunking Connections”.

b/ Only qualified Brocade SFPs are used. Only Brocade-branded or certain Brocade-qualified
SFPs are supported.

(2) Configuration Steps

From page 589 "Configuring an extended ISL" of "Fabric OS Administrator’s Guide FOS 7.2.0"

1. Connect to the switch and log in using an account assigned to the admin role.

2. Enter the switchDisable command.

3. Enter the configure command to set the switch fabric-wide configurations. You can set the
following fabric-wide settings:
(* = multiplication symbol)

Field Type Default Range
Domain Number 1 Varies
R_A_TOV Number 10000 E_D_TOV * 2 to 120000
E_D_TOV Number 2000 1000 to R_A_TOV/2
WAN_TOV Number 0 0 to R_A_TOV/4
MAX_HOPS Number 7 7 to 19

4. For 8-Gbps platforms only, enter the portCfgFillword command to set ARB as the fill word.
portcfgfillword [slot/]port, mode
The mode parameter in this command should be set to 3 if the vc_translation_link_init
parameter in the portCfgLongDistance command (in the next step) is set to 1.

5. Enter the portCfgLongDistance command.

portcfglongdistance [slot/]port [distance_level] [vc_translation_link_init] [-distance desired_distance]
Please review "FOS Command Reference Guide for more details regarding portCfgLongDistance CLI command p.737.

6. Repeat step 4 and step 5 for the remote extended ISL port. Both the local and remote
extended ISL ports must be configured to the same distance_level. When the connection is
initiated, the fabric will reconfigure.

If you are not comfortable with performing above action plan, you can engage Oracle ACS to take care of that.

About how to contact and the many benefits to using Oracle ACS, see Oracle Advanced Customer Support Services

(3) On this particular case, as there was a second redundant fabric (with another pair of switches) , we configured successfully on the fly (no switchdisable) the ports on the first fabric

just disabling the port , change configuration parameters and enabling again, on each port of each switch of the ISL link, ie for port 14 :

portdisable 14
portcfgfillword 14 3
portcfgcreditrecovery --enable 14
portcfglongdistance 14 LE 1
portenable 14

With this new configuration, there are 46 buffer credits available on each E port.

brocade2:admin> portbuffershow
User Port Lx Max/Resv Avg Buffer Usage & FrameSize Buffer Needed Link Remaining
Port Type Mode Buffers Tx Rx Usage Buffers Distance Buffers
---- ---- ---- ------- ---------------------------- ------ ------- --------- ----------
0 F - 8 1(1712) - ( 116) 8 - -
1 F - 8 1(1644) - ( 848) 8 - -
2 F - 8 1(1032) 1(1828) 8 - -
3 F - 8 - ( - ) - ( - ) 8 - -
4 F - 8 - ( - ) - ( - ) 8 - -
5 F - 8 - ( - ) - ( - ) 8 - -
6 F - 8 - ( - ) - ( - ) 8 - -
7 - 8 - ( - ) - ( - ) 0 - -
8 - 8 - ( - ) - ( - ) 0 - -
9 - 8 - ( - ) - ( - ) 0 - -
10 - 8 - ( - ) - ( - ) 0 - -
11 - 8 - ( - ) - ( - ) 0 - -
12 - 8 - ( - ) - ( - ) 0 - -
13 LE 46 - ( - ) - ( - ) 0 - -
14 E LE 46 6(1208) 9(1648) 46 46 10km
15 E LE 46 8(1456) 7(1400) 46 46 10km
16 - 8 - ( - ) - ( - ) 0 - -
17 - 8 - ( - ) - ( - ) 0 - -
18 - 8 - ( - ) - ( - ) 0 - -
19 - 8 - ( - ) - ( - ) 0 - -
20 - 8 - ( - ) - ( - ) 0 - -
21 - 8 - ( - ) - ( - ) 0 - -
22 - 8 - ( - ) - ( - ) 0 - -
23 - 8 - ( - ) - ( - ) 0 - - 370

After 48 hours the value of tim_txcrd_z was only of 1413

tim_txcrd_z 1413 Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc 0- 3: 0 0 1413 0
tim_txcrd_z_vc 4- 7: 0 0 0 0
tim_txcrd_z_vc 8-11: 0 0 0 0
tim_txcrd_z_vc 12-15: 0 0 0 0

phy_stats_clear_ts 06-23-2015 Localtime Tue 09:38:09 Timestamp of phy_port stats clear
lgc_stats_clear_ts 06-23-2015 Localtime Tue 09:38:09 Timestamp of lgc_port stats clear

References

<NOTE:1613159.1> - CRC Errors On ISL 8GB Port On Brocade300 - Vc_translation_link_init = 0 And Fillword = 0 = IDLE_IDLE
<NOTE:1518936.1> - How to Display FC Switches B2B (Buffer to Buffer) Credits

Attachments

This solution has no attachment