How to update the InfiniBand HCA firmware on Exalytics X4-4, X5-4 and X6-4 systems

Asset ID:	1-71-2172279.1
Update Date:	2017-06-15
Keywords:

Solution Type Technical Instruction Sure

Solution 2172279.1 : How to update the InfiniBand HCA firmware on Exalytics X4-4, X5-4 and X6-4 systems

Applies to:

Exalytics In-Memory Machine X4-4 - Version All Versions to All Versions [Release All Releases]
Exalytics In-Memory Machine X5-4 - Version All Versions to All Versions [Release All Releases]
Exalytics In-Memory Machine X6-4 - Version All Versions to All Versions [Release All Releases]
x86 64 bit

Goal

How to update the InfiniBand HCA firmware on Exalytics X4-4, X5-4 and X6-4 systems

Solution

To update the firmware on an InfiniBand HCA used in an Exalytics X4-4, X5-4 or X6-4 server the following steps should be followed.

First find the pci address of the HCA using the lspci command

[root@exalytics0 ~]# lspci | grep InfiniBand
20:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]

Exalytics X4-4, X5-4, and X6-4 systems should have the InfiniBand HCA card located in pci slot 4 so the lspci command should show the address in use as "20:00.0"

Use this address in the mstflint query command to check the current firmware level of the card

[root@exalytics0 ~]# mstflint -d 20:00.0 q
Image type: FS2
FW Version: 2.11.1280
Device ID: 4099
HW Access Key: Disabled
Description: Node Port1 Port2 Sys image
GUIDs: 0010e00001661ff4 0010e00001661ff5 0010e00001661ff6 0010e00001661ff7
MACs: 0010e0661ff5 0010e0661ff6
VSD:
PSID: ORC1090120019

The latest firmware revision for use in the Exalytics X4-4, X5-4 or X6-4 servers is 2.35.5530. If the previous command shows this revision reported on the "FW Version" line then there is no need for update. If the output shows an older revision (as in our example 2.11.1280) then proceed with the update steps.

Download the firmware update from MOS patch 24786646 and unzip the file. The firmware image file fw-ConnectX3-rel-2_35_5530-43-7046442_7092757.bin will be in the download, transfer it to the Exalytics server. In the following example the file is located in the /tmp directory, if your file is in a different location on the server change the command as appropriate.

The InfiniBand interfaces should be brought down before performing the update. This will interrupt any connection to the InfiniBand fabric and should be done when there is no active use of the interface required. Plan for downtime of this connection before proceeding.

If unsure of which interfaces the InfiniBand HCA are then use the following ip command to discover them.

[root@exalytics0 ~]# ip link show | grep -B 1 infiniband
15: ib0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast master bond0 state UP qlen 4096
link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:10:e0:00:01:66:1f:f5 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
16: ib1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast master bond0 state UP qlen 4096
link/infiniband 80:00:00:49:fe:80:00:00:00:00:00:00:00:10:e0:00:01:66:1f:f6 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
18: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 65520 qdisc noqueue state UP
link/infiniband 80:00:00:49:fe:80:00:00:00:00:00:00:00:10:e0:00:01:66:1f:f6 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

then bring down each of the interfaces

[root@exalytics0 ~]# ifconfig bond0 down
[root@exalytics0 ~]# ifconfig ib0 down
[root@exalytics0 ~]# ifconfig ib1 down

After the interfaces are brought down perform the firmware update using the following mstflint command

[root@exalytics0 ~]# /usr/bin/mstflint -d 20:00.0 -i /tmp/fw-ConnectX3-rel-2_35_5530-43-7046442_7092757.bin burn

Current FW version on flash: 2.11.1280
New FW version: 2.35.5530

Burning FS2 FW image without signatures - OK
Restoring signature - OK

when the firmware update is complete bring the interfaces back up

[root@exalytics0 ~]# ifconfig ib0 up
[root@exalytics0 ~]# ifconfig ib1 up
[root@exalytics0 ~]# ifconfig bond0 up

To make sure the firmware update was successful the HCA can be queried once again (note the following example now shows the FW Version as 2.35.5530 confirming that the firmware update succeeded)

[root@exalytics0 ~]# mstflint -d 20:00.0 q
Image type: FS2
FW Version: 2.35.5530
FW Release Date: 31.12.2015
Device ID: 4099
HW Access Key: Disabled
Description: Node Port1 Port2 Sys image
GUIDs: 0010e00001661ff4 0010e00001661ff5 0010e00001661ff6 0010e00001661ff7
MACs: 0010e0661ff5 0010e0661ff6
VSD:
PSID: ORC1090120019

Attachments

This solution has no attachment