Oracle ZFS Storage Appliance: General Procedure to Upgrade IB HCA Firmware in CLI

Asset ID:	1-79-2029161.1
Update Date:	2018-05-06
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2029161.1 : Oracle ZFS Storage Appliance: General Procedure to Upgrade IB HCA Firmware in CLI

Applies to:

Sun Storage 7210 Unified Storage System - Version All Versions and later
Oracle ZFS Storage ZS3-BA - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
7000 Appliance OS (Fishworks)

Purpose

This document helps you to upgrade the firmware version of infiniband HCA cards installed on the ZFS storage appliance.

Details

NOTE: As of September 2017, the M2 Falcon Infiniband HCA card 375-3696 / 7092767 and the CX3 Dual QDR InfiniBand HCA card 7046442 / 7092757 have necessary firmware upgrades to be done as per this document.

The original 375-3606 Infiniband HCA has only firmware version 2.6.000 and this has not changed. The 375-3606 Infiniband HCA currently does NOT have any firmware updates and is not covered by this document.

The appliance shell steps should be performed by an Oracle support representative. Please open a Support Request for assistance.

PLEASE NOTE: The information in this document pertains to ZFS Storage Appliances running the 2013.1.3.0/AK8.3.0 release (or later).

To perform an upgrade of the IB HCA Firmware, the appliance MUST be running (at least) 2013.1.3.0.

The OS 8.6.x release has known issues with fwflash utility, ideally you should upgrade the IB HCA firmware with older AK code and then to upgrade the AK code. Issue is Fixed in OS 8.7.0.

Alternate workaround, if the system has an older release version (2013.1.3.0 to 2013.1.5.6) available on the ZFS-SA:

1) Check if 'older' release is available on the ZFSSA, eg.:

nas# zfs list -o name -r system | grep root
system/ak-nas-2013.06.05.4.2_1-1.1/root

2) Mount the 'older' release on the temporary mountpoint ( /mnt ):

NAS# mount -f zfs system/ak-nas-2013.06.05.4.2_1-1.1/root /mnt

3) Copy the 'fwflash' utility to /var/tmp and run from there:

NAS# cp /mnt/usr/sbin/fwflash /var/tmp/

NAS# /var/tmp/fwflash ....

PLEASE NOTE: The ONLY currently supported M2 Falcon Infiniband HCA card (375-3696/7092767) firmware for the ZFS Storage Appliance is FW V2.11.2010.

( 2.11.2012 allows for sriov. This is a feature that the appliance will never use, if ZFSSA is already installed with 2.11.2012 either it can be downgraded or remain with same code. It will not cause any harm. )

If upgrading the firmware on an Infiniband CX-2 card, see Doc ID 2082722.1 - FMA 'pcie-fatal' event seen after upgrade of Infiniband CX-2 firmware on ZS3-ES

The latest IB HCA firmware patches can be downloaded using the reference document Doc ID 1345814.1 How to search for and download patches for my product within My Oracle Support?

Latest available firmware for M2 Falcon Infiniband HCA card (part numbers 375-3696 & 7092767) is FW V2.11.2010

Patch 16340059: FW25408 V2.11.2010 (375-3696-01)

Latest available firmware for CX3 Dual QDR InfiniBand HCA Card (part numbers 7046442 & 7092757) is FW 2.35.5532

Patch 26364954: FW 2.35.5532

PLEASE NOTE: As of code release OS 8.7.11, the HCA binaries are available in the /usr/lib/ak/fw directory

As per bug 25192027 (Add HCA firmware binaries to AK), /usr/lib/ak/fw now has two directories in it that have the HCA binaries.

Mellanox-CX2-2.11.2010
Mellanox-CX3-2.11.1280

NOTE: For the CX3 HCA, this is the 'old' code.

According to the latest bug 26676806 - Integrate HCA CX3 firmware 2.35.5532 to AK OS 8.7.18, /usr/lib/ak/fw/Mellanox-CX3-2.35.5532/ now has the new HCA binaries for CX3.

1. Copy the firmware patch from the client host system into a share exported from the appliance via NFS, HTTP, FTP, SFTP or SMB.

There are additional methods to get the file onto the appliance (as this seems to be the most common problem encountered).

The current methods most known are:
1) copy onto a share and from shell, copy off of share (this has drawback that it doesn't work for passive head)
2) scp -> can only scp out from appliance to an external ssh server. Cannot scp into the appliance.

Another method which may be valuable to add that is overlooked is using "uuencode" and they copy+paste through webex or shared shell.
Note: Shared Shell may timeout if copying large amounts of data and may need to break into parts

a) On local machine, uuencode the file:

uuencode -m fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin > fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin.uue

b) Open the file in notepad/leafpad or similar ASCII editor. Then select all and copy (CTRL+A; CTRL+C)

c) in PuTTY on webex or in shared shell, open a file to receive the data:

vi /var/tmp/fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin.uue

d) Enter append/edit mode using 'a' then paste the contents. In webex using PuTTY, right-click. In Shared Shell use "SHIFT+CTRL+V"

e) Let it finish then exit edit and save (ESCAPE twice, then :wq)

d) uudecode the file:

cd /var/tmp; uudecode fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin.uue

e) File is now at: /var/tmp/fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin

Verify with "digest -a md5 /var/tmp/fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin"

2. Verify the number of Infiniband cards installed in the head.

ZFS-head01:> maintenance hardware select chassis-000 select slot show

LABEL STATE MANUFACTURER MODEL SERIAL
slot-000 PCIe 0 ok Oracle 2x10Gb Optical Ethernet unknown
slot-001 PCIe 1 ok Sun Microsystems, Inc. Dual 4x6Gb External SAS-2 HBA 465769T+1246TC0187
slot-002 PCIe 2 ok Sun Microsystems, Inc. Dual 4x6Gb External SAS-2 HBA 465769T+1246TC019X
slot-003 PCIe 3 ok Sun Microsystems, Inc. Dual Port QDR IB HCA M2 unknown <<
slot-004 Cluster Card ok Oracle Fishworks CLUSTRON 200 unknown
slot-005 PCIe 9 ok Oracle 2x10Gb Optical Ethernet unknown
slot-006 PCIe 6 ok Sun Microsystems, Inc. Dual Port QDR IB HCA M2 unknown <<
slot-007 PCIe 7 ok Sun Microsystems, Inc. Dual 4x6Gb External SAS-2 HBA 465769T+1245TC013K
slot-008 PCIe 8 ok Sun Microsystems, Inc. Dual 4x6Gb External SAS-2 HBA 465769T+1245TC012M
slot-009 PCIe 4 absent - - -
slot-010 PCIe 5 absent - - -

3. Open a Service Request to engage a Oracle Storage-TSC NAS engineer for further assistance.

Instructions to TSC personnel:

4. Login to appliance shell

ZFS-head01:> confirm shell

5. Validate the HCA card(s) is/are running with the appropriate firmware and gather the device path of HCA(s).

ZFS-head01# fwflash -l -c IB

List of available devices:
Device[0] /devices/pci@0,0/pci8086,340e@7/pci15b3,673c@0:devctl
Driver hermon
Class [IB]
GUID: System Image - 0021280001fca5bf
Node Image - 0021280001fca5bc
Port 1 - 0021280001fca5bd
Port 2 - 0021280001fca5be
Mac 1 - 0000002128fca5bc
Mac 2 - 0000002128fca5bd
Firmware revision : 2.7.8130
Product : 375-3696-01 B0 (Sun Falcon QDR)
PSID : SUN0170000009

Device[1] /devices/pci@75,0/pci8086,340c@5/pci15b3,673c@0:devctl
Driver hermon
Class [IB]
GUID: System Image - 0021280001fca71f
Node Image - 0021280001fca71c
Port 1 - 0021280001fca71d
Port 2 - 0021280001fca71e
Mac 1 - 0000002128fca71c
Mac 2 - 0000002128fca71d
Firmware revision : 2.7.8130
Product : 375-3696-01 B0 (Sun Falcon QDR)
PSID : SUN0170000009

6a. Extract the file /var/tmp/p16340059_2112010_Solaris86-64.zip (for M2 Falcon Infiniband HCA card - 375-3696 & 7092767)

6b. Extract the file /var/tmp/p26364954_2355532_Solaris86-64.zip (for CX3 Dual QDR InfiniBand HCA Card - 7046442 & 7092757)

Example:

ZFS-head01# 7z x p16340059_2112010_Solaris86-64.zip 

7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov&nbsp; 2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

Processing archive: p16340059_2112010_Solaris86-64.zip
Extracting fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin
Extracting fwConnectX2-375-3696_metadata.xml
Extracting readme.txt

Everything is Ok

Files: 3
Size: 784141
Compressed: 331875

7a. Upgrade the IB HCA firmware. (FW bin file is 'fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin' for M2 Falcon Infiniband HCA card - 375-3696 & 7092767)

7b. Upgrade the IB HCA firmware. (FW bin file is 'fw-ConnectX3-rel-2_35_5532-15-7046442_7092757.bin' for CX3 Dual QDR InfiniBand HCA Card - 7046442 & 7092757)

Example:

ZFS-head01#fwflash -f ./fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin -d /devices/pci@0,0/pci8086,340e@7/pci15b3,673c@0:devctl
The current HCA firmware version is : 2.7.8130
Will be updated to HCA firmware ver of : 2.11.2010
About to update firmware on /devices/pci@0,0/pci8086,340e@7/pci15b3,673c@0:devctl
with file /var/tmp/fw-ConnectX2-rel-2_11_2010-be-375-3696-01.bin.
Do you want to continue? (Y/N): Y
. . . . . . . . . . . . . . . . . . . .
fwflash: New firmware will be activated after you reboot

NOTE: This method is safe to perform in a cluster or standalone environment. However a downtime is always recommended because the new firmware will be activated only after a reboot or by a takeover from the peer node.

8. Repeat the step 7 for the each available IB HCA card.

9. Reboot the system to activate the new firmware.

ZFS-head01# reboot     {or}     ZFS-head01 (CLI) maintenance system reboot

10. Verify the new version of the firmware by issuing the following command:

ZFS-head01# fwflash -l -c IB

Do you still have questions? You can use My Oracle Support Communities. Communities put you in touch with industry professionals like yourself. They are monitored by Oracle support engineers, so you can expect reliable and correct answers. Ask questions and see what others are asking about in the Disk Storage ZFS Storage Appliance Community.

References

<NOTE:2039263.1> - Exalogic: Intermittent Storage Connectivity Issues over InfiniBand due to Older HCA Card Firmware on ZFS Storage Appliance
<BUG:24473359> - BACKPORT 22595423 TO AK-2013-REL - CANNOT USE FWFLASH TO READ/WRITE FIRMWARE SIN
<NOTE:1345814.1> - How to search for and download patches for my product within My Oracle Support?
<BUG:26676806> - INTEGRATE HCA CX3 FIRMWARE 2.35.5532 TO AK
<NOTE:2082722.1> - Oracle ZFS Storage Appliance: FMA 'pcie-fatal' event seen after upgrade of Infiniband CX-2 firmware on ZS3-ES
<NOTE:2051841.1> - Infiniband (IB) HCA Firmware - Summary of Patches Available

Attachments

This solution has no attachment