Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Linux

Asset ID:	1-79-2057632.1
Update Date:	2016-03-10
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2057632.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Linux - Physical) for October 2015

Applies to:

Exalogic Elastic Cloud X4-2 Hardware - Version X4 to X4 [Release X4]
Oracle Exalogic Elastic Cloud Software - Version 2.0.0.0.0 to 2.0.6.2.3
Exalogic Elastic Cloud X5-2 Hardware
Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 to X2 [Release X2]
Exalogic Elastic Cloud X3-2 Hardware
Linux x86-64

Purpose

Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around October 2015 Patch Set Update (PSU) for that system.

Scope

The target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU.

This document provides the following:

README files that are included in the October 2015 PSU for convenience. These can be used for reference without having to download the PSU first.
Known issues with the PSU - these provide a list of known issues along with workarounds/solutions, if available. Engineers and administrators planning to apply the PSU should ensure to review these before applying the PSU.

This document will be kept up-to-date with updates to errata and known issues.

Details

NOTE:

This PSU contains upgrades to both OL5, OL6 releases.
This PSU includes updates to ELLC 14.2.4

Patch Download

Released: October 2015

Product Version: 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) for Oracle Linux x86-64

PATCH 21445814 - EXALOGIC LINUX 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR OCTOBER 2015

Patch Readme Documentation

Refer to the readme documentation attached 21445814-Linux.zip to this Document on how to upgrade the Exalogic infrastructure:

Exalogic X2-2,X3-2,X4-2 and X5-2 Systems: Upgrading to 2.0.6.2.3

The readme content structure layout of the 21445814-Linux.zip:

21445814-Linux
|
| - README.html
| - exalogic-lctools-14.2.4-release-notes.txt
| - docs/README.html
| - docs/components/Enable_http_zfs.html
| - docs/components/ILOM_ComputeNode_Upgrade.html
| - docs/components/Index.html
| - docs/components/Menu.html
| - docs/components/NM2_Upgrade.html
| - docs/components/ZFS_Upgrade.html

Pre-requisites

PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racks

Refer to following Note which has information on this important pre-requisite check.

<Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Appendices

Appendix A: Fixed Bugs List

Please review Document ID : 2057633.1 - Exalogic Infrastructure October 2015 PSU - Fixed Bugs List.

Appendix B: Patching Known Issues

RPM dependencies/conflicts - oracle-hmp-tools / oracle-hmp-hwmgmt / oracle-hmp-libs

Symptoms

Upgrading from very old version EECS to latest PSUs fails with errors similar to the following:

INFO: Tue Jun 16 09:47:53 EST 2015: >>>> Backup /usr/lib/init-exalogic-node/exalogic_machine.conf
ERROR: Tue Jun 16 09:47:53 EST 2015: Unable to locate /usr/lib/init-exalogic-node/exalogic_machine.conf

INFO: Tue Jun 16 10:20:22 EST 2015: Uninstalling OpenIPMI RPM
error: Failed dependencies:
OpenIPMI is needed by (installed) oracle-hmp-tools-2.2.1-1.el5.x86_64
ERROR: Tue Jun 16 10:20:22 EST 2015: FAILED - Uninstalling OpenIPMI RPM
INFO: Tue Jun 16 10:20:22 EST 2015: >>>> remove services that launch post patching procedures
[/opt/ebi_upgrade/BaseImageOEL5/scripts/ebi_patch.sh: Tue Jun 16 10:25:26 EST 2015: ]: BEGIN on Platform Type: 2 ...

Cause

RPM dependencies/conflicts with oracle-hmp-hwmgmt, oracle-hmp-tools, oracle-hmp-libs

Solution/Workaround

1)Remove following packages prior to Compute Node patching. As latest PSU bundle will have higher version of these packages.

$ rpm -ev oracle-hmp-hwmgmt-2.2.1-1.el5

$ rpm -ev oracle-hmp-tools-2.2.1-1.el5

$ rpm -ev oracle-hmp-libs-2.2.1-1.el5

2)Validate if exalogic_machine.conf file present or not on each Compute Nodes.

$ /usr/lib/init-exalogic-node/exalogic_machine.conf

3)Validate if CheckSWProfile and CheckHWnFWProfile have any outstanding issues prior patching.

PCIE fault HW alert in ZFS post IB firmware update

Symptoms:

PCIE related HW fault is reported in ZFS, after applying IB card firmware.

Error message is Similar to:-

SUNW-MSG-ID: SPX86-8003-QH, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Aug 17 06:34:36 2015
PLATFORM: i86pc, CSN: 1441NML0FE, HOSTNAME: denp04sn02
SOURCE: appliance/kit/akd:default, REV: 1.0
EVENT-ID: d63362ca-89ea-e1e9-90cb-ee2bc152e335
DESC: An Integrated I/O (II0) non-fatal error in downstream PCIE device has
occurred.
AUTO-RESPONSE: The service-required LED on the chassis will be illuminated.
IMPACT: System continues to run with degraded resources.
REC-ACTION: Contact your service provider for proper repair procedures.

Cause:

A Hermon IB card firmware upgrade was active on the card.

Solution/Workaround:

Workaround is mentioned in Mos Note 2040757.1

ZFS Upgrade Failed. Storage node ilom console indicates error in joining cluster.

Symptoms

While applying the Oct 2015 PSU on X4-2 or X5-2 rack, the upgrade of ZS3-ES may fail.

The exapatch fails with:-

"ERROR: Could not ssh to ZFS host xx.xx.xx.xx after upgrade. "

Further in the storage node ilom of the failed head, it is stuck at this stage :

Oracle ZFS Storage ZS3-ES Configuration
    Copyright (c) 2008, 2015, Oracle and/or its affiliates. All rights
reserved.

    NET-0 <=>  NET-1 <=>  NET-2 <=>  NET-3 <X>

    SUNW-MSG-ID: AK-8000-2U, TYPE: Defect, VER: 1, SEVERITY: Major
    EVENT-TIME: Tue Jun  2 17:01:26 UTC 2015
    PLATFORM: i86pc, CSN: 1427NML0E5, HOSTNAME: scat01sn02
    SOURCE: aktty, REV: 1.0
    EVENT-ID: 1fc05eee-db80-e94d-d6c2-c06714389111
    DESC: The appliance experienced an unrecoverable protocol error while
    attempting to join a cluster of appliances.  Refer to
    http://sun.com/msg/AK-8000-2U for more information.
    AUTO-RESPONSE: No automated response is possible for this defect.
    IMPACT: The appliance cannot be configured into the cluster.
    REC-ACTION: Reboot the system and attempt to configure the cluster again.
    Contact your vendor for support.

    ESC-3: Halt   ESC-4: Reboot   ESC-5: Info

    For help, see http://www.oracle.com/goto/ZS3-ES/docs/

Cause

@ Details of this issue are in this MOS note:
@ .
@ Oracle ZFS Storage Appliance: ZS3-X Clustron Card in PCIE Slot 2 Does Not
@ Initialize During boot. (Doc ID 1910623.1)
@ .
@ The following bug is referenced from the MOS note:
@ .
@ BUG 18672251 - AK-8000-2U: unrecoverable protocol error on the ZS3-ES
@ (X4170M3) appliance

Solution/Workaround

Manually reboot the storage head with ESC-4. Refer to <Note 1910623.1>.

PSU Upgrade fails while upgrading ZFSSA.

Symptoms: The ZFSSA upgrade will fail when applying the appliance from 2011 version of the software to the 2013 version of the software. Exapatch terminated as follows .....

INFO: ZFS-Storage-Head xx.xx.xx.x2 is active ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 is ready ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 running pre-patch checks for ready head

INFO: ZFS-Storage-Head xx.xx.xx.x1 Ready head checks completed.

INFO: ZFS-Storage-Head xx.xx.xx.x1 successfully completed all pre-patch checks

ERROR: ZFS-Storage-Head xx.xx.xx.x2 STOP!! There are 4 active problem(s) on this head, Please resolve them first, Exiting

Unable to proceed as pre-patch checks failed for active head xx.xx.xx.x2

Additional information may be found in the console output and the log file: /var/log/exapatch_20150818205629.log

Errors are similar to the following:

problem-000 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-001 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-002 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-003 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

Cause: The ZFSSA upgrade will fail when applying the Oct 2015 PSU due to bug
19024367 with the ZFSSA. This is a known issue that can happen when upgrading
from the 2011 version of the software to the 2013 version of the software.

After the head that has been upgraded becomes active, it tries to access the
readzilla on the passive head. The readzillas are local to head each head and
one head cannot access the readzilla of its peer. Typically the active head
is supposed to ignore the readzilla disks on its peer, but due to bug
19024367, these disks are marked as failed on the active head. The passive
head will not show any failed disks.

To validate this compare the serial numbers of the disks marked as failed
with the serial numbers of the readzilla disks on its peer.

This is a false positive and can be safely ignored.

Solution/Workaround:

Since Exapatch does not allow you to proceed, the software on the of the
remaining head must be upgraded manually.

Note: After the manual update if still the issue exist, reboot the storage head.

Exapatch console output message has ' WARNING: unable to update patch history: '.

Symptoms

While running the Exapatch command, the console output has following warning message:-

WARNING: unable to update patch history: Unable to find unique identifier "1352NML06M" for Compute-Node xx.xx.xx.xx in the rack history file.

Cause

This is an expected behavior when the rack history file is not created beforehand and does not affect the node functionality.

Solution/Workaround

To avoid this, prior to patching a rack history should be created:

[root@compute-node]# /exalgic-lctools/bin/exapatch -a refreshHistory

______________________________________________________________________________________________________________________________________________________________________

Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"

Refer to <Note 2092033.1> for details on this known issue.

_____________________________________________________________________________________________________________________________________________________________________________________

Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Refer to <Note 2114516.1> for details on this known issue.

Appendix: C

Exapatch with '-f' or '--force' option

Refer to below Note

<Note 1997466.1>: Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

References

<NOTE:2087741.1> - Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks
<NOTE:2092033.1> - Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"
<NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic
<NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note
<NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic
<NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Attachments

This solution has no attachment