SuperCluster : Procedure to manually upgrade (Out of Band) Infiniband gateway switch in SuperCluster

Asset ID:	1-71-2338348.1
Update Date:	2018-01-12
Keywords:

Solution Type Technical Instruction Sure

Solution 2338348.1 : SuperCluster : Procedure to manually upgrade (Out of Band) Infiniband gateway switch in SuperCluster

Applies to:

Sun Datacenter InfiniBand Switch 36 - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster Specific Software - Version 1.x to 2.x [Release 1.0 to 2.0]
Information in this document applies to any platform.

Goal

Manually upgrade Infiniband switch firmware in SuperCluster rack if ibsw-patchmgr is having failures or if upgrading them out of band. Perform the below steps only if you can afford production downtime as the below steps can result in DB node evictions when executed on a production (live) system. Production downtime needs to be accessed based on the number of DB domains in the rack.

Solution

Upgrading IB switches as part of QFSDP patching activity, use ibsw-patchmgr and follow the steps provided in README.ibswitch in QFSDP bundle.
Ensure that you are choosing an IB switch firmware version that is certified with your Exadata storage cell version per <Document 1567979.1>
In a SuperCluster rack, upgrade the firmware on the spine switch first and then on each leaf switches.
The below steps are ONLY as a last report when ibsw-patchmgr fails and can afford production downtime.

1. Login to the switch:

root@ssccn1# ssh 192.168.1.201
Password:
[root@sscnm1 ~]#

2. Disable Subnet Manager (SM) on the switch:

[root@sscnm1 ~]# disablesm
Stopping partitiond-daemon. [ OK ]
Stopping IB Subnet Manager..-.-.-.-.-.-+ [ OK ]

3. Connect to switch ILOM:

[root@sscnm1 ~]# spsh

Oracle(R) Integrated Lights Out Manager

Version 2.2.6-2 ILOM 3.2.6 r118629

Warning: HTTPS certificate is set to factory default.

Hostname: sscnm1

4. Load the FW onto the switch:

Download the FW (latest 2.2.7) from Patch 26575824
unzip the patch file and copy the resultant FW files on to the source machine.
sundcs_36p_repository_2.2.7_1.pkg
sundcs_36p_repository_upgrade_2.1_to_2.2.7_1.pkg

Ensure you enable the protocol (Source and Target) being used to transfer the firmware file from the source machine to the switch, the following transfer protocols FTP, TFTP, SFTP, SCP, HTTP, or HTTPS are supported.
In this case we are using scp which is enabled on both source and switch.
192.168.1.9 is the IP address of the source machine where the firmware file is located under /var/tmp/
If you are upgrading from 2.1.x to 2.2.7 then use the FW pkg sundcs_36p_repository_upgrade_2.1_to_2.2.7_1.pkg else use sundcs_36p_repository_2.2.7_1.pkg

-> load -source scp://root:<passwd>@source-host-ip/FW file path>

-> load -source scp://root:welcome1@192.168.1.9/var/tmp/sundcs_36p_repository_2.2.7_1.pkg
Downloading firmware image. This will take a few minutes.

SUN DCS 36p version: 2.2.6-2
Build time: Jul 4 2017 10:19:21
SP board info:
Manufacturing Date: 2009.12.08
Serial Number: "NCD4I0062"
Hardware Revision: 0x0006
Firmware Revision: 0x0102
BIOS version: NOW1R112
BIOS date: 04/24/2009
FROM_VERSION: 2.2.6-2
TO_VERSION: 2.2.7-1

NOTE: Firmware upgrade will upgrade the SUN DCS 36p firmware.
ILOM will enter a special mode to load new firmware. No
other tasks should be performed in ILOM until the firmware
upgrade is complete.

Are you sure you want to load the specified file (y/n)? y
Setting up environment for firmware upgrade. This will take a few minutes.
Starting SUN DCS 36p FW update

==========================
Performing operation: I4 A
==========================
I4 A: I4 is already at the given version.

=========================================
Performing operation: SUN DCS 36p firmware update
=========================================
SUN DCS 36p Kontron module fw upgrade from 2.2.6-2 to 2.2.7-1:
Please reboot the system to enable firmware update of Kontron module. The download of the Kontron firmware image happens during reboot.

After system reboot, Kontron FW update progress can be monitored in browser using URL [http://system] OR at OS command line prompt by using command [telnet system 1234] where system is the hostname or IP address of SUN DCS 36P or GW.

Firmware update is complete.

5. Reset the switch for the new firmware to take effect:

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

->
Broadcast message from root@sscnm1
(unknown) at 22:32 ...

The system is going down for reboot NOW!
Connection to 192.168.1.201 closed by remote host.
Connection to 192.168.1.201 closed.
root@ssccn1:/var/tmp# telnet 192.168.1.201 1234
Trying 192.168.1.201...
Connected to 192.168.1.201.
Escape character is '^]'.
Fri Dec 8 06:33:25 UTC 2017 : Created magnumfw_repository.
Downloading Package: 165MB 0:00:40 [4.02MB/s] [ <=> ]
Fri Dec 8 06:34:06 UTC 2017 : Download done.
Installing Part 1: 431MB 0:00:53 [8.02MB/s] [ <=> ]
Fri Dec 8 06:35:06 UTC 2017 : Disk image written.
Fri Dec 8 06:35:06 UTC 2017 : Doing filesystem check.
Fri Dec 8 06:35:06 UTC 2017 : Filesystem check done.
Installing Part 2: 23.5MB 0:00:01 [14.2MB/s] [ <=> ]
Fri Dec 8 06:35:11 UTC 2017 : Disk image written.
Fri Dec 8 06:35:11 UTC 2017 : Doing filesystem check.
Fri Dec 8 06:35:11 UTC 2017 : Filesystem check done.
Fri Dec 8 06:35:11 UTC 2017 : Rebooting.

6. Wait for the switch to reboot (might take about ~3 mins).

Check: MOS Note: Infiniband Gateway Switch Stays In Pre-boot Environment During Upgrade/Reboot <Document 2202721.1>

root@ssccn1:/# ssh 192.168.1.201

Sun Data center switch pre-boot environment.

root@192.168.1.201's password:

Sun Data center switch pre-boot environment.

======================================================================
= =
= WARNING: This is pre-boot environment used for system maintenance. =
= Application image is not active!!! =
= =
======================================================================

Do you wish to remain in pre-boot environment?
If you do, please answer 'y' (timeout 10 seconds) [N/y]:n

Trying to start application image ...
Previous application starts failed (3 times). Please run check_app_partition.
Will not start application image.

init> check_app_partition
Doing filesystem check ...
e2fsck 1.39 (29-May-2006)
/dev/sda5: clean, 15728/110592 files, 401487/441116 blocks
Everything looks OK.
init> boot
init> Connection to 192.168.1.201 closed by remote host.
Connection to 192.168.1.201 closed.
root@ssccn1:/#

7. Login to the switch and verify the firmware:

root@ssccn1:/var/tmp# ssh 192.168.1.201
Password:
FW upgrade completed successfully on Thu Dec 7 22:36:18 PST 2017.
Please run the "fwverify" CLI command to verify the new image.
This message will be cleared on next reboot.

You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.

[root@sscnm1 ~]# fwverify

Checking all present packages:
.............................................................................................................................................................................................................................................. OK

Checking if any packages are missing:
............................................................................................................................................................................................................................................. OK

Verifying installed files:
............................................................................................................................................................................................................................................. OK

Checking FW Coreswitch:
FW Version: 7.4.3002 OK
PSID: SUN_NM2-36p_006 OK
Verifying image integrity OK

[root@sscnm1 ~]# version
SUN DCS 36p version: 2.2.7-1
Build time: Aug 4 2017 12:20:53
SP board info:
Manufacturing Date: 2009.12.08
Serial Number: "NCD4I0062"
Hardware Revision: 0x0006
Firmware Revision: 0x0102
BIOS version: NOW1R112
BIOS date: 04/24/2009
[root@sscnm1 ~]#

8. Enable Subnet Manager (SM) on the switch:

[root@sscnm1 ~]# enablesm
Stopping partitiond-daemon. [ OK ]
Stopping IB Subnet Manager..-.-.-.-.-.-+ [ OK ]

9. Perform the above steps on all the other switches in the rack.

References

<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types

Attachments

This solution has no attachment