FS System: How to Manually Downgrade SPBIOS Firmware on an FS1-2 Controller

Asset ID:	1-72-2173777.1
Update Date:	2017-03-08
Keywords:

Solution Type Problem Resolution Sure

Solution 2173777.1 : FS System: How to Manually Downgrade SPBIOS Firmware on an FS1-2 Controller

Applies to:

Oracle FS1-2 Flash Storage System - Version 6.1 to 6.2 [Release 6.1 to 6.2]
Information in this document applies to any platform.

Symptoms

Upon replacement, the FS1 will attempt to automatically synchronize its version of SPBIOS from the surviving Controller onto the replacement Controller motherboard. If that synchronization fails and replacement motherboard does not come online, check the event log for Controller firmware upgrade failures.

Attempts to manually synchronize the SPBIOS will fail with an authentication error:

WN508002000XXXXXX0 # echo "changeme" | fwupdate update sp-bios-firmware -n sp_bios -f /var/tftp/spbios.rom -q -H=169.254.2.9 -U=root
Enter password (8 to 16 characters):
********
Authentication type NONE not supported
Error: Unable to establish LAN session
Authentication type NONE not supported
Error: Unable to establish LAN session
WN508002000XXXXXX0 #

Changes

Early in 2016 Netra Server X3-2 motherboards spares were updated with a new version of ILOM SPBIOS, 3.2.4.58. An even newer version 3.2.4.58.a has also been observed.

Cause

In systems running R6.2.7 and below, the fwupdate utility will not be able to synchronize (downgrade) the ILOM SPBIOS if the version of the surviving Controller is 3.1.2.40 and the replacement Controller has 3.2.4.58 or higher.

Solution

Steps to setup ILOM Web Interface on an FS1-2 Controller and Download the SPBIOS

Verify if the version of SPBIOS on the replacement Controller motherboards is different:
1. ssh as root to the active Pilot.
  1. If direct Access to the Pilot is available from customer network, reference Document 2029847.1 FS System: How to Enable SSH to a Pilot & Document 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System.
  2. If ssh cannot be enabled to the Pilots, access can be achieved by using the Controller SER_MGT port, reference Document 2070735.1 FS System: How to Access FS1-2 ILOMs Using a Serial Connection.
2. ssh to the working Controller, Controller 1 = 172.30.80.128, Controller 2 = 172.30.80.129.
3. Check the SPBIOS version on the working Controller:
  
  WN508002000XXXXXX0 # ipmitool -H 169.254.2.5 -U root -P changeme sunoem cli "version"
  Connected. Use ^D to exit.
  -> version
  SP firmware 3.1.2.40
  SP firmware build number: 93718
  SP firmware date: Fri Oct 17 18:23:31 EDT 2014
  SP filesystem version: 0.1.23
  ->
  Session closed
  Disconnected
  WN508002000XXXXXX0 #
4. Check the SPBIOS version on the replaced Controller motherboard:
  
  WN508002000XXXXXX0 # ipmitool -H 169.254.2.9 -U root -P changeme sunoem cli "version"
  Connected. Use ^D to exit.
  -> version
  SP firmware 3.2.4.58
  SP firmware build number: 101476
  SP firmware date: Thu Jul 30 13:05:23 EDT 2015
  SP filesystem version: 0.2.10
  ->
  Session closed
  Disconnected
  WN508002000XXXXXX0 #
5. If the version of the replacement motherboard is 3.2.4.58 or similar, proceed to step 2. If not, reengage Oracle support for further assistance.
Obtain the proper SPBIOS file from the surviving Controller:
1. Remaining on the working Controller, cd /var/tftp
2. Copy the spbios.rom file to /var/tmp on the active Pilot:
  
  WN508002000XXXXXXX # scp spbios.rom 172.30.80.X:/var/tmp
  spbios.rom 100% 17MB 16.9MB/s 00:01
  WN508002000XXXXXXX #
  
  NOTE: the X in the IP address can be either 2 or 3 depending on which Pilot is active. 172.30.80.2 is Pilot 1, 172.30.80.3 is Pilot 2.
  1. If direct access to the Pilot was available from the customer network, exit back to the active Pilot use SCP to copy the spbios file to a laptop.
  2. If access to the Pilot was achieved via the Controller serial port, copy the file to a USB drive:
    1. ssh to one of the Pilots
      
      WN508002000XXXXXX0 # ssh pilot1
          You are entering the operating system shell. By proceeding you have agreed that THIS ACTION MAY VOID ANY SUPPORT AGREEMENT. If you do not agree to this --- or don`t otherwise understand what you are doing --- you should type "exit" at the shell prompt. EVERY COMMAND THAT YOU ENTER HERE IS AUDITED, and support personnel may use this audit trail to substantiate invalidating your support contract. The operating system shell is NOT a supported mechanism for managing this appliance, and COMMANDS EXECUTED HERE MAY DO IRREPARABLE HARM.
      
          NOTHING SHOULD BE ATTEMPTED HERE BY UNTRAINED SUPPORT PERSONNEL UNDER ANY CIRCUMSTANCES. This appliance is a non-traditional operating system environment, and expertise in a traditional operating system environment in NO WAY constitutes training for supporting this appliance. THOSE WITH EXPERTISE IN OTHER SYSTEMS - HOWEVER SUPERFICIALLY SIMILAR - ARE MORE LIKELY TO MISTAKENLY EXECUTE OPERATIONS HERE THAT WILL DO IRREPARABLE HARM. Unless you have been explicitly trained on supporting this appliance via the operating system shell, you should immediately return to the appliance shell.
      
          Type "exit" now to logout.
      
      [root@pilot1 ~]#
    2. Insert a USB drive into one of the two USB ports in the front of that Pilot
    3. Mount the USB drive
      
      [root@pilot1 ~]# mkdir /mnt/USB
      [root@pilot1 ~]# mount /dev/sdb1 /mnt/USB
    4. Copy the spbios.rom file to the USB drive, then unmount the USB drive to remove it
      
      [root@pilot1 ~]# cp /var/tmp/spbios.rom /mnt/USB
      [root@pilot1 ~]# umount /mnt/USB
3. On the laptop, rename the spbios.rom file to spbios.pkg.
Setup the ILOM network on the replaced Controller motherboard:
1. Isolate the Controller by removing all Private Management Interface (PMI) and ILOM network cables (NET2, NET3, NET MGT & <...>).
2. Attach a network cable between the laptop and the Controller NET MGR port.
3. Use PuTTY to establish a serial connection with SER MGT port of the problem Controller. Reference Document 2070735.1 FS System: How to Access FS1-2 ILOMs Using a Serial Connection.
4. Log into the ILOM as root, reference Document 2046703.1 FS System: Passwords Associated with the Oracle FS1-2 Flash Storage System, and run the following commands:
  
  -> cd /SP/network
  /SP/network
  
  -> set pendingipdiscovery=static
  Set 'pendingipdiscovery' to 'static'
  
  -> set pendingipaddress=169.254.2.9
  Set 'pendingipaddress' to '169.254.2.9'
  
  -> set pendingipnetmask=255.255.255.252
  Set 'pendingipnetmask' to '255.255.255.252'
  
  -> set commitpending=true
  Set 'commitpending' to 'true'
  
  ->show (to confirm updates)
  
  /SP/network
      Targets:
          interconnect
          ipv6
          test
  
      Properties:
          commitpending = (Cannot show property)
          dhcp_clientid = none
          dhcp_server_ip = none
          ipaddress = 169.254.2.9
          ipdiscovery = static
          ipgateway = 0.0.0.0
          ipnetmask = 255.255.255.252
          macaddress = 00:21:28:A1:C6:44
          managementport = MGMT
          outofbandmacaddress = 00:21:28:A1:C6:44
          pendingipaddress = 169.254.2.9
          pendingipdiscovery = static
          pendingipgateway = 0.0.0.0
          pendingipnetmask = 255.255.255.252
          pendingmanagementport = MGMT
          sidebandmacaddress = 00:21:28:A1:C6:45
          state = enabled
  
      Commands:
          cd
          set
          show
Setup the laptop network interface:
1. Control Panel -> Network and Sharing -> Change adapter settings
2. Right click on the Local Area Connection to be used and select Properties.
3. Select the Internet Protocol Version 4 (TCP/IPv4) and then Properties.
4. Update the settings as follows:
  1. Select the radio button next to Use the following IP address
  2. IP address: 169.254.2.10
  3. Subnet mask: 255.255.255.252
  4. Click on OK
5. At this point you should be able to ping the Controller ILOM (169.254.2.9)
Use a browser to download the spbios.pkg file:
1. Point the browser to the Controller ILOM's IP (169.254.2.9).
2. Log into the ILOM as root.
3. In the navigation tree on the left, expand ILOM Administration and the select Maintenance.
4. In the main window, select Firmware Update and then Enter Update Mode.
5. Select Upload Local File and use the Browse option to select the spbios.pkg file from step 2-d.
6. Select Upload to start the process.
7. Once update completes, confirm the version of ILOM is correct (System Information, Summary page).
Post procedure clean up:
1. Use PuTTY to reestablish a serial connection with the SER MGT port as in step 2.
2. After logging into the ILOM as root, run the following commands:
  
  -> cd /SP/network
  -> set pendingipdiscovery=dhcp
  Set 'pendingipdiscovery' 'dhcp'
  
  -> set pendingipnetmask=255.255.255.252
  Set 'pendingipnetmas' '255.255.255.252'
  
  -> set commitpending=true
  Set 'commitpending' 'true'
  
  -> show (confirm updates)
  
  /SP/network
      Targets:
          interconnect
          ipv6
          test
  
      Properties:
          commitpending = (Cannot show property)
          dhcp_clientid = none
          dhcp_server_ip = 169.254.2.10
          ipaddress = 169.254.2.9
          ipdiscovery = dhcp
          ipgateway = 0.0.0.0
          ipnetmask = 255.255.255.252
          macaddress = 00:21:28:A1:C6:44
          managementport = MGMT
          outofbandmacaddress = 00:21:28:A1:C6:44
          pendingipaddress = 169.254.2.9
          pendingipdiscovery = dhcp
          pendingipgateway = 0.0.0.0
          pendingipnetmask = 255.255.255.252
          pendingmanagementport = MGMT
          sidebandmacaddress = 00:21:28:A1:C6:45
          state = enabled
  
      Commands:
          cd
          set
          show
Power off the Controller by removing both power cords.
Reattach the PMI and ILOM network cables on the replacement Controller that were disconnected in step 3-a.
Power on the Controller by inserting both power cords.

References

<BUG:24407901> - FS1 FAILING QUORUM AFTER MOBO REPLACEMENT IN CONTROLLER-01.

Attachments

This solution has no attachment