Brocade 48000 - Blade model FR4-18i - Link Timeout in Internal Port (slot 7, port 23) Resulted in Blade Fault

Asset ID:	1-72-1547171.1
Update Date:	2013-08-14
Keywords:

Solution Type Problem Resolution Sure

Solution 1547171.1 : Brocade 48000 - Blade model FR4-18i - Link Timeout in Internal Port (slot 7, port 23) Resulted in Blade Fault

Applies to:

Brocade 48000 Director - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk/Tape Storage Area Networks

Data Replication from primary to secondary site stopped working March 5th,

we got this errors on Brocade 48000 (FOS v6.4.0b) switch sw2, from ge ports used for replication, blade on slot 7, model FR4-18i

2013/03/05-06:06:47, [PORT-1009], 152424, SLOT 6 | FID 128, INFO, sw2, GigE Port (ID: 128) has been disabled
2013/03/05-06:06:47, [PORT-1009], 152425, SLOT 6 | FID 128, INFO, sw2, GigE Port (ID: 129) has been disabled

2013/03/05-06:07:22, [PORT-1008], 152426, SLOT 6 | FID 128, INFO, sw2, GigE Port (ID: 128) has been enabled
2013/03/05-06:07:22, [PORT-1008], 152427, SLOT 6 | FID 128, INFO, sw2, GigE Port (ID: 129) has been enabled

- Blade 7 (model FR4-18i) in sw2 was found to be faulty and replaced
- Blade 7 in sw2 appeared to be online per errdump messages

2013/03/06-17:07:05, [BL-1031], 152447, SLOT 6 | FID 128, CRITICAL, sw2, Link timeout in internal port (slot 7, port 23) resulted in blade fault. Use slotpoweroff/slotpoweron to recover the blade.
2013/03/06-17:07:05, [EM-1034], 152448, SLOT 6 | CHASSIS, ERROR, SilkWorm48000, Slot 7 set to faulty, rc=20015.

2013/03/06-21:15:05, [EM-1069], 152451, SLOT 6 | CHASSIS, INFO, SilkWorm48000, Slot 7 is being powered off.
2013/03/06-21:15:05, [FW-1440], 152452, SLOT 6 | FID 128, INFO, sw2, Slot 7 state has changed to FW_FRU_ABSENT.
2013/03/06-21:15:06, [EM-1050], 152453, SLOT 6 | CHASSIS, INFO, SilkWorm48000, FRU Slot 7 removal detected.

2013/03/06-21:16:22, [EM-1049], 152454, SLOT 6 | CHASSIS, INFO, SilkWorm48000, FRU Slot 7 insertion detected.
2013/03/06-21:16:22, [EM-1070], 152455, SLOT 6 | CHASSIS, INFO, SilkWorm48000, Slot 7 is being powered on.
2013/03/06-21:17:20, [BM-1002], 152456, SLOT 6 | CHASSIS, INFO, SilkWorm48000, Connection established between CP and blade in slot 7.
2013/03/06-21:23:20, [BL-1017], 152457, SLOT 6 | CHASSIS, INFO, SilkWorm48000, Slot 7 Initializing...
2013/03/06-21:23:21, [BL-1018], 152458, SLOT 6 | CHASSIS, INFO, SilkWorm48000, Slot 7 Initialization completed.
2013/03/06-21:23:27, [SNMP-1008], 152459, SLOT 6 | FID 128, INFO, sw2, The last device change happened at : Wed Mar 6 21:23:23 2013

slotshow -m       :

Slot   Blade Type     ID    Model Name     Status
--------------------------------------------------
1     AP BLADE     24     FR4-18i        ENABLED
2     SW BLADE     18     FC4-32         ENABLED
3     UNKNOWN                            VACANT
4     UNKNOWN                            VACANT
5     CP BLADE     16     CP256          ENABLED
6     CP BLADE     16     CP256          ENABLED
7     AP BLADE     24     FR4-18i        ENABLED
8     SW BLADE     18     FC4-32         ENABLED
9     UNKNOWN                            VACANT
10     UNKNOWN                            VACANT

- From swicthshow command, we can see the ge ports were online:

Index Slot Port Address Media Speed State     Proto
===================================================

       1 ge0            id    1G   Online    FCIP
       1 ge1            id    1G   Online    FCIP

       7 ge0            id    1G   Online    FCIP
       7 ge1            id    1G   Online    FCIP

- But portcmd still not working :

portcmd --ping 7/ge0 -s 10.0.1.142 -d 10.0.2.241
portcmd --ping 7/ge1 -s 10.0.1.143 -d 10.0.2.242

- Found Blade 7 in sw2 listed as faulty again per Mar 6 customer switch query, and two other ports from Blade slot 7 remain Offline when they should not:

192 7 16 04c000 -- -- Offline VE <<< offline
200 7 24 04c800 -- -- Offline VE <<< offline

Cause

Brocade defect 315258

Fault of the blade in slot 7 of Brocade 48000 sw2 noted in the latest data is due to defect 315258, which is described in detail in Brocade TECHNICAL SUPPORT BULLETIN TSB-2010-096:

If an FR4-18i blade is connected to the back-end link where the error was detected, the FR4-18i blade will be faulted.

Solution

As a workaround, execute the following commands on both switches (on primary and secondary site) in order to power the FR4-18i blades off and on:

slotpoweroff 7
slotpoweron 7

This is solved by upgrading FOS :

Changes made in Fabric OS v6.3.2a, v6.4.0c and v 6.4.1 under Defect 315258 corrected this issue

This was escalated to Brocade , case 114880

For more information, consult Brocade TECHNICAL SUPPORT BULLETIN TSB-2010-096.

Attachments

This solution has no attachment