Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Solaris

Asset ID:	1-79-2057614.1
Update Date:	2016-06-23
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2057614.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Solaris - Physical) for October 2015

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.4.0.0 to 2.0.6.2.3
Oracle Exalogic Elastic Cloud X2-2 Hardware
Exalogic Elastic Cloud X3-2 Hardware
Exalogic Elastic Cloud X4-2 Hardware
Exalogic Elastic Cloud X5-2 Hardware
Oracle Solaris on x86-64 (64-bit)

Purpose

Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around October 2015 Patch Set Update (PSU) for that system.

Scope

The target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU.

This document provides the following:

README files that are included in the October 2015 PSU for convenience. These can be used for reference without having to download the PSU first.
Known issues with the PSU - these provide a list of known issues along with workarounds/solutions, if available. Engineers and administrators planning to apply the PSU should ensure to review these before applying the PSU.

This document will be kept up-to-date with updates to errata and known issues.

Details

Note: This PSU includes updates to ELLC 14.2.4

Patch Download

Released: October 2015

Product Version: 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) for Oracle Solaris x86-64

PATCH 21445814 - EXALOGIC SOLARIS 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR OCTOBER 2015

Patch Readme Documentation

Refer to the readme documentation attached 21445814-Solaris.zip to this Document on how to upgrade the Exalogic infrastructure:

Exalogic X2-2,X3-2,X4-2 and X5-2 Systems: Upgrading to 2.0.6.2.3

The readme content structure layout of the 21445814-Solaris.zip:

21445814-Solaris
|
| - README.html
| - exalogic-lctools-14.2.4-release-notes.txt
| - docs/README.html
| - docs/components/Enable_http_zfs.html
| - docs/components/ILOM_ComputeNode_Upgrade.html
| - docs/components/Index.html
| - docs/components/Menu.html
| - docs/components/NM2_Upgrade.html
| - docs/components/ZFS_Upgrade.html

INTERNAL

Pre-requisites

PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racks

Refer to following Note which has information on this important pre-requisite check.

<Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Appendices

Appendix A: Fixed Bugs List

Please review Document ID : 2057633.1 - Exalogic Infrastructure October 2015 PSU - Fixed Bugs List.

Appendix B: Patching Known Issues

CheckHWnFWProfile fails on X5-2 Solaris.

Symptoms

On Physical Solaris X5-2 systems /opt/exalogic.tools/tools/CheckHWnFWProfile fails with:-

*******************************************************************************************************************

Does not contain supported disk controller. Required LSI MR9261-8i. But has LSI
SUN
SUN
SUN
SUN
SUN
SUN
SUN MR9261-8i
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320

*******************************************************************************************************************

Cause

This is due to a known issue that is tracked through the following bug:

BUG 20745068 - CHECKHWNFWPROFILE FAILS POST COMPUTE NODE UPGRADE TO APR15 PSU ON X5-2

There are two underlying issues that cause CheckHWnFWProfile to fail on Solaris X5-2 systems:

1. The supported configuration metadata is incorrect and contains an incorrect reference to the LSI MR9261-8i disk controller. The profile check against the actual version installed on the node (LSI MR9361-8i) fails. The supported version is LSI MR9361-8i; this error may be safely ignored

2. The MegaCli utility that is installed on X5-2 nodes is unable to detect the disk controller to obtain/upgrade/downgrade the installed firmware version. You may safely continue to use the node with the version of the disk controller firmware shipped from the factory.

Solution/Workaround

This error may be safely ignored and you may safely continue to use the node with the version of the disk controller firmware shipped from the factory.

Upgrade from EECS 2.0.4.0.4 and earlier releases (containing NM2 FW v2.0.x) to EECS 2.0.4.0.5 and newer releases (containing NM2 FW v2.1.x) requires downtime to perform the switch FW upgrade.

Symptoms

Following the NM2 gateway switch firmware upgrade, all Exalogic compute nodes running Solaris experience an outage of any EoIB networks configured on the nodes.

Cause

This is a known issue for NM2 switch firmware upgrades from v2.0.x to v2.1.x, which results in a change of the EoIB network interface names on the compute nodes.

Solution/Workaround

The solution is already provided in the README for the Exalogic Solaris PSUs, and will be added to https://support.oracle.com/epmos/faces/DocumentDisplay?id=1571367.1 for quick reference.

Post Patching Switches, NFS mount point is in hung state

Symptoms:

During the NM2-GW upgrade, if the NFS servers stops responding, the patch upgrade process launched from the compute node may hang.

Cause:

Observed to happen while upgrade from PSU 2.0.4.0.6 or before (NM2 v2.1.4-1 or earlier). See bug 18465204

Solution/Workaround:

In such situation, force kill the upgrade process, restart the network services and retry the patching command. To restart network services, run:

# svcadm restart network/physical:default

ZFS Upgrade Failed. Storage node ilom console indicates error in joining cluster.

Symptoms:

While applying the Oct 2015 PSU on X4-2 or X5-2 rack, the upgrade of ZS3-ES may fail.

The exapatch fails with:-

"ERROR: Could not ssh to ZFS host xx.xx.xx.xx after upgrade. "

Further in the storage node ilom of the failed head, it is stuck at this stage :

Oracle ZFS Storage ZS3-ES Configuration
    Copyright (c) 2008, 2015, Oracle and/or its affiliates. All rights
reserved.

    NET-0 <=>  NET-1 <=>  NET-2 <=>  NET-3 <X>

    SUNW-MSG-ID: AK-8000-2U, TYPE: Defect, VER: 1, SEVERITY: Major
    EVENT-TIME: Tue Jun  2 17:01:26 UTC 2015
    PLATFORM: i86pc, CSN: 1427NML0E5, HOSTNAME: scat01sn02
    SOURCE: aktty, REV: 1.0
    EVENT-ID: 1fc05eee-db80-e94d-d6c2-c06714389111
    DESC: The appliance experienced an unrecoverable protocol error while
    attempting to join a cluster of appliances.  Refer to
    http://sun.com/msg/AK-8000-2U for more information.
    AUTO-RESPONSE: No automated response is possible for this defect.
    IMPACT: The appliance cannot be configured into the cluster.
    REC-ACTION: Reboot the system and attempt to configure the cluster again.
    Contact your vendor for support.

    ESC-3: Halt   ESC-4: Reboot   ESC-5: Info

    For help, see http://www.oracle.com/goto/ZS3-ES/docs/

Cause:

@ Details of this issue are in this MOS note:
@ .
@ Oracle ZFS Storage Appliance: ZS3-X Clustron Card in PCIE Slot 2 Does Not
@ Initialize During boot. (Doc ID 1910623.1)
@ .
@ The following bug is referenced from the MOS note:
@ .
@ BUG 18672251 - AK-8000-2U: unrecoverable protocol error on the ZS3-ES
@ (X4170M3) appliance

Solution/Workaround:

Manually reboot the storage head with ESC-4. Refer to <Note 1910623.1>.

PCIE fault HW alert in ZFS post IB firmware update

Symptoms:

PCIE related HW fault is reported in ZFS, after applying IB card firmware.

Error message is Similar to:-

SUNW-MSG-ID: SPX86-8003-QH, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Aug 17 06:34:36 2015
PLATFORM: i86pc, CSN: 1441NML0FE, HOSTNAME: denp04sn02
SOURCE: appliance/kit/akd:default, REV: 1.0
EVENT-ID: d63362ca-89ea-e1e9-90cb-ee2bc152e335
DESC: An Integrated I/O (II0) non-fatal error in downstream PCIE device has
occurred.
AUTO-RESPONSE: The service-required LED on the chassis will be illuminated.
IMPACT: System continues to run with degraded resources.
REC-ACTION: Contact your service provider for proper repair procedures.

Cause:

A Hermon IB card firmware upgrade was active on the card.

Solution/Workaround:

Workaround is mentioned in MOS Note 2040757.1

PSU Upgrade fails while upgrading ZFSSA.

Symptoms: The ZFSSA upgrade will fail when applying the appliance from 2011 version of the software to the 2013 version of the software. Exapatch terminated as follows .....

INFO: ZFS-Storage-Head xx.xx.xx.x2 is active ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 is ready ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 running pre-patch checks for ready head

INFO: ZFS-Storage-Head xx.xx.xx.x1 Ready head checks completed.

INFO: ZFS-Storage-Head xx.xx.xx.x1 successfully completed all pre-patch checks

ERROR: ZFS-Storage-Head xx.xx.xx.x2 STOP!! There are 4 active problem(s) on this head, Please resolve them first, Exiting

Unable to proceed as pre-patch checks failed for active head xx.xx.xx.x2

Additional information may be found in the console output and the log file: /var/log/exapatch_20150818205629.log

Errors are similar to the following:

problem-000 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-001 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-002 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-003 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

Cause: The ZFSSA upgrade will fail when applying the Oct 2015 PSU due to bug
19024367 with the ZFSSA. This is a known issue that can happen when upgrading
from the 2011 version of the software to the 2013 version of the software.

After the head that has been upgraded becomes active, it tries to access the
readzilla on the passive head. The readzillas are local to head each head and
one head cannot access the readzilla of its peer. Typically the active head
is supposed to ignore the readzilla disks on its peer, but due to bug
19024367, these disks are marked as failed on the active head. The passive
head will not show any failed disks.

To validate this compare the serial numbers of the disks marked as failed
with the serial numbers of the readzilla disks on its peer.

This is a false positive and can be safely ignored.

Solution/Workaround:

Since Exapatch does not allow you to proceed, the software on the of the
remaining head must be upgraded manually.

Note: After the manual update if still the issue exist, reboot the storage head.

Exapatch console output message has ' WARNING: unable to update patch history: '.

Symptoms

While running the Exapatch command, the console output has following warning message:-

WARNING: unable to update patch history: Unable to find unique identifier "1352NML06M" for Compute-Node xx.xx.xx.xx in the rack history file.

Cause

This is an expected behavior when the rack history file is not created beforehand and does not affect the node functionality.

Solution/Workaround

To avoid this, prior to patching a rack history should be created:

[root@compute-node]# /exalgic-lctools/bin/exapatch -a refreshHistory

_____________________________________________________________________________________________________________________________________________________________________________________

Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Refer to <Note 2114516.1> for details on this known issue.

_____________________________________________________________________________________________________________________________________________

Appendix C: FAQs

Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

Refer to <Note 1997466.1> for details

References

<NOTE:1997466.1> - Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching
<NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic
<NOTE:2087741.1> - Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks
<NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic
<NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note
<NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Attachments

This solution has no attachment