All Nodes lose IB communication when halting or shutting down IB LEAF switch for maintenance

Asset ID:	1-71-1960598.1
Update Date:	2016-05-31
Keywords:

Solution Type Technical Instruction Sure

Solution 1960598.1 : All Nodes lose IB communication when halting or shutting down IB LEAF switch for maintenance

Applies to:

Exadata X4-2 Hardware - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
Oracle Exadata Hardware - Version 11.1.0.7 to 11.2.3.1.1 [Release 11.1 to 11.2]
Information in this document applies to any platform.

Goal

Why did all nodes lose communication and reboot when shutting down IB leaf switch ibb01?

Solution

Shutting down or halting an infiniband switch from the Linux CLI will leave the physical links in the UP state but not responding to SM requests . The active bonds may not fail over resulting
in communication failure and node eviction.

Correct way to halt or shutdown an NM2 IB switch is to remove the AC power. See Bug 18795339 - IBSWITCH ISSUE: EXADATA X4-2: CSSD CRASHED DURING IBSWITCH REBOOT

Only use supported commands at the switch Linux CLI. See supported commands in the "help" output.

Note: To reboot a switch, this must be done from the ILOM using /SP/reset

Attachments

This solution has no attachment