Exalogic Patch Set Update (PSU) Release 2.0.6.2.4 (Solaris

Asset ID:	1-79-2093675.1
Update Date:	2016-06-23
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2093675.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.4 (Solaris - Physical) for January 2016

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.4.0.0 to 2.0.6.2.4
Oracle Exalogic Elastic Cloud X2-2 Hardware
Exalogic Elastic Cloud X3-2 Hardware
Exalogic Elastic Cloud X4-2 Hardware
Exalogic Elastic Cloud X5-2 Hardware
Oracle Solaris on x86-64 (64-bit)

Purpose

Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around January 2016 Patch Set Update (PSU) for that system.

Scope

The target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU.

This document provides the following:

README files that are included in the January 2016 PSU for convenience. These can be used for reference without having to download the PSU first.
Known issues with the PSU - these provide a list of known issues along with workarounds/solutions, if available. Engineers and administrators planning to apply the PSU should ensure to review these before applying the PSU.

This document will be kept up-to-date with updates to errata and known issues.

Details

Note: This PSU includes updates to ELLC 14.2.5

Patch Download

Released: January 2016

Product Version: 2.0.6.2.4 (on X2-2/X3-2/X4-2/X5-2) for Oracle Solaris x86-64

PATCH 22136605 - EXALOGIC SOLARIS 2.0.6.2.4 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR JANUARY 2016

Patch Readme Documentation

Refer to the readme documentation attached 22136605-Solaris.zip to this Document on how to upgrade the Exalogic infrastructure:

Exalogic X2-2,X3-2,X4-2 and X5-2 Systems: Upgrading to 2.0.6.2.4

The readme content structure layout of the 22136605-Solaris.zip:

22136605-Solaris
|
| - README.html
| - exalogic-lctools-14.2.5-release-notes.txt
| - docs/README.html
| - docs/components/Enable_http_zfs.html
| - docs/components/ILOM_ComputeNode_Upgrade.html
| - docs/components/Index.html
| - docs/components/Menu.html
| - docs/components/NM2_Upgrade.html
| - docs/components/ZFS_Upgrade.html

INTERNAL

Pre-requisites

PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racks

Refer to following Note which has information on this important pre-requisite check.

<Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Appendices

Appendix A: Fixed Bugs List

Please review Document ID : 2093677.1 - Exalogic Infrastructure January 2016 PSU - Fixed Bugs List.

Appendix B: Patching Known Issues

CheckHWnFWProfile fails on X5-2 Solaris.

Symptoms

On Physical Solaris X5-2 systems /opt/exalogic.tools/tools/CheckHWnFWProfile fails with:-

*******************************************************************************************************************

Does not contain supported disk controller. Required LSI MR9261-8i. But has LSI
SUN
SUN
SUN
SUN
SUN
SUN
SUN MR9261-8i
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320
ZFS Storage 7320

*******************************************************************************************************************

Cause

This is due to a known issue that is tracked through the following bug:

BUG 20745068 - CHECKHWNFWPROFILE FAILS POST COMPUTE NODE UPGRADE TO APR15 PSU ON X5-2

There are two underlying issues that cause CheckHWnFWProfile to fail on Solaris X5-2 systems:

1. The supported configuration metadata is incorrect and contains an incorrect reference to the LSI MR9261-8i disk controller. The profile check against the actual version installed on the node (LSI MR9361-8i) fails. The supported version is LSI MR9361-8i; this error may be safely ignored

2. The MegaCli utility that is installed on X5-2 nodes is unable to detect the disk controller to obtain/upgrade/downgrade the installed firmware version. You may safely continue to use the node with the version of the disk controller firmware shipped from the factory.

Solution/Workaround

This error may be safely ignored and you may safely continue to use the node with the version of the disk controller firmware shipped from the factory.

Upgrade from EECS 2.0.4.0.4 and earlier releases (containing NM2 FW v2.0.x) to EECS 2.0.4.0.5 and newer releases (containing NM2 FW v2.1.x) requires downtime to perform the switch FW upgrade.

Symptoms

Following the NM2 gateway switch firmware upgrade, all Exalogic compute nodes running Solaris experience an outage of any EoIB networks configured on the nodes.

Cause

This is a known issue for NM2 switch firmware upgrades from v2.0.x to v2.1.x, which results in a change of the EoIB network interface names on the compute nodes.

Solution/Workaround

The solution is already provided in the README for the Exalogic Solaris PSUs. For further details please review <Note 1571367.1>

Post Patching Switches, NFS mount point is in hung state

Symptoms:

During the NM2-GW upgrade, if the NFS servers stops responding, the patch upgrade process launched from the compute node may hang.

Cause:

Observed to happen while upgrade from PSU 2.0.4.0.6 or before (NM2 v2.1.4-1 or earlier). Known bug 18465204 [unpublished]

Solution/Workaround:

In such situation, force kill the upgrade process, restart the network services and retry the patching command. To restart network services, run:

# svcadm restart network/physical:default

ZFS patching fails due to losing IO access to the pool.

Symptoms:

ZFS patching fails and Exapatch terminated as follows ...

INFO: ZFS-Storage-Head xx.xx.xx.xx successfully completed all pre-patch
checks
ERROR: ZFS-Storage-Head xx.xx.xx.xx STOP!! This head in pool exalogic is not
online.
Unable to proceed as pre-patch checks failed for active head xx.xx.xx.xx
Additional information may be found in the console output and the log file:
/var/log/exapatch_20151119112342.log

Observed in the exapatch log, errors are similar to the following:-

Head in Active state, continuing...
STOP!! This head in pool exalogic is not online.
STATUS : degraded
]]
[2015-11-19T11:53:27.441-05:00] [exapatch] [TRACE:1] [] [utils] [pid: 14494] [tid: MainThread] [ecid: ] [lineno: 219] [[
STDERR:
1
STOP!! This head in pool exalogic is not online.
STATUS : degraded
]]
[2015-11-19T11:53:27.441-05:00] [exapatch] [NOTIFICATION:1] [] [utils] [pid: 14494] [tid: MainThread] [ecid: ] [lineno: 340] ERROR: ZFS-Storage-Head xx.xxx.xx.xx STOP!! This head in pool exalogic is not online.
[2015-11-19T11:53:27.441-05:00] [exapatch] [NOTIFICATION:1] [] [utils] [pid: 14494] [tid: MainThread] [ecid: ] [lineno: 340] Unable to proceed as pre-patch checks failed for active head xx.xxx.xx.xx

Cause:

During the patching, we found the system in a strange situation where the storage pool was owned by the passive(STRIPPED) head:

el01sn01:configuration storage> get

pool = exalogic

status = exported

owner = el01sn01

el01sn01:> configuration cluster show

Properties:

state = AKCS_OWNER << Active head doesn't own the pool

description = Active (takeover completed)

peer_asn = 2058f7ae-0247-c86e-c2f1-94f8cd68faa7

peer_hostname = el01sn02

peer_state = AKCS_STRIPPED

peer_description = Ready (waiting for failback)

Children:

resources => Configure resources

Node el01sn02 was READY (PASSIVE) but the storage was online on this.

el01sn02:configuration storage> get << this is the passive head

pool = exalogic

status = online

errors = 0

owner = el01sn01

profile = mirror

log_profile = log_stripe

cache_profile = cache_stripe

scrub = resilver completed after 0h0m with 0 errors

As the result, all clients are losing IO access to the pool.

Solution/Workaround:

Reboot the active head and retry the patching

Refer <Note 1910623.1>

psuSetup.sh failing due to lack of free space.

Symptoms:

psusetup.sh fails due to lack of free space. It fails while reserving share quota to 200M for exalogic-lctools.

[root@compute-node]# ./psuSetup.sh <zfs_ip>

INFO: xx.xx.xx.xxx is a ZFS appliance
INFO: The IPoIB of the ZFS is 192.168.6.131
INFO: /exalogic-lctools does not appear to be mounted from xx.xx.xx.xxx

@ INFO: using existing "common" on the ZFS Appliance with IP address
xx.xx.xx.xxx
@ INFO: using existing "exalogic-lctools" on the ZFS Appliance with IP address
xx.xx.xx.xxx
ERROR: (failed to commit changes (encountered while attempting to run command "commit"))

Cause:

When it is trying to set the limit, it was already over the current available limit.

Solution/Workaround:

Free up space in ZFSA. Refer <Note 2039102.1>

_______________________________________________________________________________________________________________________________________________________________________________________

PCIE fault HW alert in ZFS post IB firmware update

Symptoms:

PCIE related HW fault is reported in ZFS, after applying IB card firmware.

Error message is Similar to:-

SUNW-MSG-ID: SPX86-8003-QH, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Aug 17 06:34:36 2015
PLATFORM: i86pc, CSN: 1441NML0FE, HOSTNAME: denp04sn02
SOURCE: appliance/kit/akd:default, REV: 1.0
EVENT-ID: d63362ca-89ea-e1e9-90cb-ee2bc152e335
DESC: An Integrated I/O (II0) non-fatal error in downstream PCIE device has
occurred.
AUTO-RESPONSE: The service-required LED on the chassis will be illuminated.
IMPACT: System continues to run with degraded resources.
REC-ACTION: Contact your service provider for proper repair procedures.

Cause:

A Hermon IB card firmware upgrade was active on the card.

Solution/Workaround:

Implement the workaround mentioned in <Note 2040757.1>

_____________________________________________________________________________________________________________________________________________________________________________________

Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Refer to <Note 2114516.1> for details on this known issue.

_____________________________________________________________________________________________________________________________________________________________________________________

Exalogic: ZFS-7320 ILOM Network Settings Lost on X2-2 and X3-2 Racks After PSU Upgrade

Refer to <Note 2115679.1> for details on this known issue.

Appendix C: FAQs

Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

Refer to below Note

<Note 1997466.1>: Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

References

<NOTE:1997466.1> - Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching
<NOTE:2040757.1> - Oracle ZFS Storage Appliance: How To Resolve - An Integrated I/O non-fatal error in downstream PCIE device has occurred.
<NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note
<NOTE:2087741.1> - Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks
<NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic
<NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade
<NOTE:2115679.1> - Exalogic: ZFS-7320 ILOM Network Settings Lost on X2-2 and X3-2 Racks After PSU Upgrade
<NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic

Attachments

This solution has no attachment