Multiple Fans Reported as Failed on T5220 & T5240

Asset ID:	1-72-1490986.1
Update Date:	2018-04-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1490986.1 : Multiple Fans Reported as Failed on T5220 & T5240

Applies to:

Sun SPARC Enterprise T5140 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise T5120 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise T5240 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise T5220 Server - Version All Versions to All Versions [Release All Releases]
Sun SPARC Enterprise T5240 Server
Information in this document applies to any platform.

Symptoms

Fan removal or failure occurs on multiple fans or SP powers off HOST and on HOST power up [or while running Snapshot (normal mode data_set=full)] SP powers off HOST quickly.

Example 1 - Fan module issue Only (FMx)

Post Status: Passed all devices
ID Time FRU Class Fault
1 Jun 21 04:27:55 /SYS/FANBD0/FM1 SP detected fault: Required FAN at FANBD0/FM1 is not present.< ---------
2 Jun 15 15:38:47 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F0 has reached low non-recoverable threshold.
3 Jun 15 15:38:42 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F1 has reached low non-recoverable threshold.
4 Jun 15 15:38:57 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F0 has reached low non-recoverable threshold.
5 Jun 15 15:38:53 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F1 has reached low non-recoverable threshold.
6 Jun 21 04:27:11 /SYS/FANBD0/FM2 SP detected fault: Required FAN at FANBD0/FM2 is not present. < ---------
sc>
console msg 20min after 2nd reset Post Status: Passed all devices
ID Time FRU Class Fault
1 Jun 21 04:27:55 /SYS/FANBD0/FM1 SP detected fault: Required FAN at FANBD0/FM1 is not present. < ----------
2 Jun 15 15:38:47 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F0 has reached low non-recoverable threshold.
3 Jun 15 15:38:42 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F1 has reached low non-recoverable threshold.
4 Jun 15 15:38:57 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F0 has reached low non-recoverable threshold.

Fault event description:Faulty fan module is at location /SYS/FANBD0/FM2 < ----------
SunHwTrapFanFault
sunHwTrapSystemIdentifier =
sunHwTrapChassisId = BEL0845FTL
sunHwTrapProductName = T5240
sunHwTrapSuspectComponentName = /SYS/FANBD0/FM2
sunHwTrapFaultClass = system.fault
sunHwTrapFaultCertainty = 100
sunHwTrapFaultMessageID =
sunHwTrapFaultUUID =

################################################################################

From example 1 - the issue is with Fan(s) module 1 and 2, they should be replaced and no other components!

################################################################################

Video - How to replace T5120/T5220 Fan module (1:26)

Example 2 - Fan board issue (FANBDx)

##### Tx000/showfaults_-v #####
Last POST Run: Wed Jun 3 13:33:38 2015

Post Status: Passed all devices
ID Time FRU Class Fault
1 Jun 03 13:27:43 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F0 has reached low non-recoverable threshold.
2 Jun 03 13:27:30 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F1 has reached low non-recoverable threshold.
3 Jun 03 13:28:10 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F0 has reached low non-recoverable threshold.
4 Jun 03 13:27:55 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F1 has reached low non-recoverable threshold.

Jun 03 13:27:11: IPMI |critical: "ID = 3bb : 06/03/2015 : 13:27:11 : Fan : /FB0/FM0/F0/TACH : Lower Non-recoverable going low : reading 0 <= threshold 2400 RPM"
Jun 03 13:27:14: IPMI |critical: "ID = 3bc : 06/03/2015 : 13:27:14 : Fan : /FB0/FM1/F1/TACH : Lower Non-recoverable going low : reading 0 <= threshold 2400 RPM"

SYS/FANBD0/FM0/F0 TACH failed (0rpm )
SYS/FANBD0/FM0/F1 TACH failed (0rpm )
SYS/FANBD0/FM1/F0 TACH failed (0rpm )
SYS/FANBD0/FM1/F1 TACH failed (0rpm )

--------------------------------------------------------------------------------
Fan Status:
--------------------------------------------------------------------------------
Fans (Speeds Revolution Per Minute):
Sensor Status Speed Warn Low
--------------------------------------------------------------------------------
/SYS/FANBD0/FM0/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM0/F1/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM1/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM1/F1/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM2/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM2/F1/TACH FAILED 0 4000 2400

Component : /SYS/FANBD0
Time Stamp : 2015-06-19T06:14:10-04:00
New_Status : 0x10 (PROXIED FAULT)
Old_Status : 0x10 (PROXIED FAULT)
Initiator : SCAPP
Component : 49

################################################################################

From example 2 - the issue is with fan board 0, (FANBD0), and only FANBD0 should be replaced!

################################################################################

Example 3: both Fan boards failed
Tx000/showenvironment
/SYS/LOCATE /SYS/SERVICE /SYS/ACT
OFF ON ON

/SYS/PS_FAULT /SYS/TEMP_FAULT /SYS/FAN_FAULT
OFF OFF ON

Sensor Status Speed Warn Low
/SYS/FANBD0/FM0/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM0/F1/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM1/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM1/F1/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM2/F0/TACH FAILED 0 4000 2400
/SYS/FANBD0/FM2/F1/TACH FAILED 0 4000 2400
/SYS/FANBD1/FM0/F0/TACH FAILED 0 4000 2400
/SYS/FANBD1/FM0/F1/TACH FAILED 0 4000 2400
/SYS/FANBD1/FM1/F0/TACH FAILED 0 4000 2400
/SYS/FANBD1/FM1/F1/TACH FAILED 0 4000 2400

/SYS/FANBD0 Dialog 5017695-03 E03NTD 109 (36 degrees C) 0x10 (PROXIED FAULT)
/SYS/FANBD1 FOXCONN 5017695-04 E09NE8 94 (21 degrees C) 0x10 (PROXIED FAULT)

Component : /SYS/FANBD0
Time Stamp : Thu, Dec 21 2000 12:17:06 GMT
New_Status : 0x10 (PROXIED FAULT)
Old_Status : 0x10 (PROXIED FAULT)
Initiator : SCAPP
Component : 50
Message : TACH at /SYS/FANBD0/FM2/F0 has exceeded low non-recoverable threshold.

Component : /SYS/FANBD1
Time Stamp : Thu, Dec 21 2000 12:18:14 GMT
New_Status : 0x10 (PROXIED FAULT)
Old_Status : 0x10 (PROXIED FAULT)
Initiator : SCAPP
Component : 53
Message : TACH at /SYS/FANBD1/FM1/F0 has exceeded low non-recoverable threshold.

##### Tx000/showfaults_-v #####
Last POST Run: Mon Oct 23 13:57:23 2000

Post Status: Passed all devices
ID Time FRU Class Fault
1 Dec 21 12:16:01 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F0 has exceeded low non-recoverable threshold.
2 Dec 21 12:15:57 /SYS/FANBD0/FM0 SP detected fault: TACH at /SYS/FANBD0/FM0/F1 has exceeded low non-recoverable threshold.
3 Dec 21 12:16:34 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F0 has exceeded low non-recoverable threshold.
4 Dec 21 12:16:30 /SYS/FANBD0/FM1 SP detected fault: TACH at /SYS/FANBD0/FM1/F1 has exceeded low non-recoverable threshold.
5 Dec 21 12:17:06 /SYS/FANBD0/FM2 SP detected fault: TACH at /SYS/FANBD0/FM2/F0 has exceeded low non-recoverable threshold.
6 Dec 21 12:17:02 /SYS/FANBD0/FM2 SP detected fault: TACH at /SYS/FANBD0/FM2/F1 has exceeded low non-recoverable threshold.
7 Dec 21 12:17:41 /SYS/FANBD1/FM0 SP detected fault: TACH at /SYS/FANBD1/FM0/F0 has exceeded low non-recoverable threshold.
8 Dec 21 12:17:38 /SYS/FANBD1/FM0 SP detected fault: TACH at /SYS/FANBD1/FM0/F1 has exceeded low non-recoverable threshold.
9 Dec 21 12:18:14 /SYS/FANBD1/FM1 SP detected fault: TACH at /SYS/FANBD1/FM1/F0 has exceeded low non-recoverable threshold.
10 Dec 21 12:18:10 /SYS/FANBD1/FM1 SP detected fault: TACH at /SYS/FANBD1/FM1/F1 has exceeded low non-recoverable threshold. ################################################################################

From example 3 - The issue is with both Fan(s) board 0 and 1, in this case both fan boards and connector board

Caution - Please refer to update under "Cause" session, not always we need to have connector brd replaced even when both fan brd's showing as an issue

################################################################################

Cause

When both fan boards (and/or multiple fans) are shown as failed, these errors could be due to a known issue related to less than optimal connection of the fan board(s) and connector board.

Both fan boards are showing as failed, likely due to corrosion residue on the fan board connector and/or the fretting.

See known issue HF1: SP Powers Off HOST. On subsequent HOST Power Up the SP Powers Off HOST Quickly. Snapshot SP Event and/or FMA logs show Fan Brd and/or Multiple FAN Module Failures. Closed 6925325 Fan: A unique characteristic of this Issue is that PATA/SATA Connector Board Connector (Pins) and/or Fan Brd Edge Connector show corrosion residue, and/or smokey Film (This is NOT a Thermal Event) and/or Fan Board Edge Connector Finger Fretting (distortion, lifting). The connector manufacturer has resolved this problem so new systems will not be affected.
FIX: Replace the Connector Board and Both Fan Boards

NOTE: If we only see one FANBD with alerts no need to change both FANBD and connector board, instead only change affected FANBD.
Still have FSE to inspect for any corrosion residue, if found any then replace both FANBDs and the connector board.

Solution

Open an SR for an FE to verify the failure which may require fan board and connector board replacements. To help avoid time delays when opening a new Oracle SR, please gather and upload all related data, refer to Doc ID 1475104.1

The fan board and connector board will need replacement if corrosion residue, smokey film and/or finger fretting are found.

A unique characteristic of this Issue is that PATA/SATA Connector Board Connector (Pins) and/or Fan Brd Edge Connector show corrosion residue, and/or smokey Film (This is NOT a Thermal Event) and/or Fan Board Edge Connector Finger Fretting (distortion, lifting). The connector manufacturer has resolved this problem so new systems will not be affected.

References

<BUG:15621864> - SUNBT6925325 ELWOOD CHASSIS FAN BOARD INTERMITTENTS CAUSING FANS TO "DISSAP
<NOTE:1475104.1> - Troubleshooting data needed for T5xx0 servers

Attachments

This solution has no attachment