Exadata X2-8 - Patch 13536739 : FW upgrade fails due to NEM sas-expander FW being too low. When manually upgrading NEM FW to match the requirements, it fails with "Error: Component FW update failed"

Asset ID:	1-72-1473363.1
Update Date:	2018-03-26
Keywords:

Solution Type Problem Resolution Sure

Solution 1473363.1 : Exadata X2-8 - Patch 13536739 : FW upgrade fails due to NEM sas-expander FW being too low. When manually upgrading NEM FW to match the requirements, it fails with "Error: Component FW update failed"

Applies to:

Exadata Database Machine X2-8 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

When applying Exadata X2-8 <Patch: 13536739> the FW (firmware) upgrade exits with below error:

[NEMFirmwareCount ]
Requires:
NEM0-5.7.0.0 NEM1-5.7.0.0 2
Found:
NEM0-5.3.3.4 NEM1-5.3.3.4 2
[Warning] The hardware and firmware are not supported

When manually upgrading the NEMs to FW version 5.7.0, both the ILOM (Integrated Lights Out Manager) GUI (Graphical User Interface) and ILOM CLI (Command Line Interface) procedures fails throwing the same error:

"Error: Component FW update failed"

At the same time, on the SP (Service Processor) OS (Operating System), the following messages may appear:

Jul 2 12:02:02 ed8ncomp01-ilom dynafrud: fct_get_node:2921 ERROR:frucapi:frucache_read_container /SYS,Unknown error
Jul 2 12:02:03 ed8ncomp01-ilom dynafrud: fct_get_node:2931 ERROR:frucapi:frucache_read_container /SYS/NEM0/MB,Unknown error
Jul 2 12:02:04 ed8ncomp01-ilom dynafrud: fct_get_node:2931 ERROR:frucapi:frucache_read_container /SYS/NEM1/MB,Unknown error

Cause

Before applying <Patch: 13536739> (image version 11.2.3.1.0), NEM FW should stand at 5.7.0.0 , as explained on the README files on previous patches:

Only for Sun Fire X4800 Oracle Database Servers and Sun Fire X4800 M2 Oracle Database Servers: Verify the NEM (Network Express Module) firmware version is 5.7 or later using the following command.

for a_nem in 0 1; do ipmitool sunoem cli force "show /SYS/NEM${a_nem}" \
| sed -e '/fru_version/!d;s/.*FW//g;s/[\t ]\+//g;s/^/NEM'$a_nem'-/g;'; done

If the firmware is an earlier version than 5.7.0.0, then update it as follows:

1. Make db_patch_11.2.2.4.1.111203/x4800-sas2exp-5.7.0.0-rom.pkg file available from a URL.
For example http://myhost.mycompany.internal.com/firmware/x4800-sas2exp-5.7.0.0-rom.pkg
2. Ensure that the grid infrastructure is shut down.
3. Use SSH to connect to the ILOM as the root user.
4. For each NEMx where x is 0 and 1, do the following:
cd /SYS/NEMx
load -source http://myhost.mycompany.internal.com/firmware/x4800-sas2exp-5.7.0.0-rom.pkg

5 .Power cycle the database server.

If customers fail to upgrade the NEM FW before applying <Patch: 13536739>, the NEM sas-expander FW check won't pass :

[NEMFirmwareCount ]
Requires:
NEM0-5.7.0.0 NEM1-5.7.0.0 2
Found:
NEM0-5.3.3.4 NEM1-5.3.3.4 2
[Warning] The hardware and firmware are not supported

In this particular case, as part of the troubleshooting, the customer manually updated the ILOM FW to the version contained under <Patch: 13536739> :

ILOM version
SP firmware 3.0.16.16
SP firmware build number: 69500
SP firmware date: Mon Nov 14 10:48:38 CST 2011
SP filesystem version: 0.1.23

Therefore, when manually upgrading the NEMs to match the required version, it will fail with below error:

"Error: Component FW update failed"

Solution

Reload previous patch - roll back to previous patch (see instructions on the Patch README file) so that the db node ILOM FW is downgraded, or manually downgrade ILOM FW to the previous version.

Update NEM FW as explained on the README file so that the requirements are met (NEM FW 5.7.0)

Once the requirements are met, then apply Exadata <Patch: 13536739>

Attachments

This solution has no attachment