ODA: Infra Patching Hangs Updating Controllers with "Updating the Controller : c1 with the firmware : 0x0072 11.05.03.00"

Asset ID:	1-72-2018659.1
Update Date:	2015-08-21
Keywords:

Solution Type Problem Resolution Sure

Solution 2018659.1 : ODA: Infra Patching Hangs Updating Controllers with "Updating the Controller : c1 with the firmware : 0x0072 11.05.03.00"

Applies to:

Oracle Database Appliance - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Upgrade of infra on ODA V1 system hangs at following step :

[root@ODAUATCGNS01 bin]# ./oakcli update -patch 12.1.2.2.0 --infra

INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Both nodes may get rebooted automatically during the patch if required
Do you want to continue: [Y/N]?: y
INFO: User has confirmed the reboot
INFO: Patch bundle must be unpacked on the second node also before applying this patch
Did you unpack the patch bundle on the second node?: [Y/N]?: y

Please enter the 'root' password :
Please re-enter the 'root' password:
INFO: Setting up the SSH
..
========truncated =========================

...
...
INFO: 2015-05-18 03:53:32: ....................Patching shared HDDs...............
INFO: 2015-05-18 03:53:32: Disk : d0 is already running with : ST360057SSUN600G 0B25
INFO: 2015-05-18 03:53:33: Disk : d1 is already running with : ST360057SSUN600G 0B25
INFO: 2015-05-18 03:53:33: Disk : d2 is already running with : ST360057SSUN600G 0B25
INFO: 2015-05-18 03:53:42: ....................Patching Expanders...............
INFO: 2015-05-18 03:53:42: Expander : x0 is already running with : T4 Storage 0342
INFO: 2015-05-18 03:53:43: Expander : x1 is already running with : T4 Storage 0342
INFO: 2015-05-18 03:53:43: ....................Patching Controllers...............
INFO: 2015-05-18 03:53:43: No-update for the Controller: c0

INFO: 2015-05-18 03:53:43: Updating the Controller : c1 with the firmware : 0x0072 11.05.03.00 ------>>>Update infra got stuck at this point
..
command stucks after this for hours ============ and it failed

Later when stop with "CTL+c" it failed with following error :

...
INFO: 2015-05-18 08:25:30: Expander : x0 is already running with : T4
Storage 0342
INFO: 2015-05-18 08:25:31: Expander : x1 is already running with : T4
Storage 0342
INFO: 2015-05-18 08:25:31: ....................Patching
Controllers...............
INFO: 2015-05-18 08:25:31: No-update for the Controller: c0
INFO: 2015-05-18 08:25:31: Updating the Controller : c1 with the firmware : 0x0072 11.05.03.00
ERROR: 2015-05-18 08:43:27: Failed to update the Controller : c1 with the firmware 0x0072 11.05.03.00

Changes

Updating the infra on the ODA from 2.8 and lower
to ODA versions 12.1.x and higher.

Cause

This issue is being worked in following bug :

Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05.03.00

Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05 Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05 Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05 Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05 Bug 21128684 - INFRA PATCH FROM 2.5 TO 12.1.2.2 HANGS UPDATING CONTROLLER FW C1 TO 0X0072 11.05

Solution

Until bug is resolved follow workaround as given below :

1 : (i) If you see the hang during the controller upgrade then break the existing running upgrade with Control^C : (after waiting at least 15min).

Note that there are two Controllers per node.

After the Control^C the update should be continued and other components will be upgraded.

Repeat this same procedure if the Controller hangs while updating the second Node.

2: Once you get command prompt again you can manually update the controller :

cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/SGX-SAS6-INT-Z/Base/

# fwupdate update controller -n c2 -x metadata.xml --silent-no-reboot -q -o /tmp/c2.xml

3. Reboot the system.

4. Now again run the update infra command . So incase if remaining components are not updated (as mentioned in 1(i) ) , those components will be updated.

5. Now check the following output to verify if infra upgrade is successful or not :

#oakcli show version -detail

Attachments

This solution has no attachment