Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Linux

Asset ID:	1-79-2057607.1
Update Date:	2016-10-26
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2057607.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.3 (Linux - Virtual) for October 2015

Applies to:

Exalogic Elastic Cloud X5-2 Hardware
Oracle Exalogic Elastic Cloud Software - Version 2.0.6.0.0 to 2.0.6.2.3
Oracle Exalogic Elastic Cloud X2-2 Hardware
Exalogic Elastic Cloud X3-2 Hardware
Exalogic Elastic Cloud X4-2 Hardware
Linux x86-64
Oracle Virtual Server(x86-64)

Purpose

Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around October 2015 Patch Set Update (PSU) for that system.

Scope

The target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU.

This document provides the following:

README files that are included in the October 2015 PSU for convenience. These can be used for reference without having to download the PSU first
Known issues with the PSU - these provide a list of known issues along with workarounds/solutions, if available. Engineers and administrators planning to apply the PSU should ensure to review these before applying the PSU

This document will be kept up-to-date with updates to errata and known issues.

Details

Note:

This PSU it includes updates to ELLC 14.2.4
This PSU includes updates for vservers based on OL5 and OL6 templates

Patch Download

Released: October 2015

Product Version: 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) for Oracle Exalogic Elastic Cloud infrastructure

PATCH 21445837 - EXALOGIC VIRTUAL 2.0.6.2.3 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR October 2015

Patch Readme Documentation

Refer to the readme documentation attached 21445837-Virtual.zip to this Document on how to upgrade the Exalogic infrastructure:

Exalogic X2-2, X3-2,X4-2 and X5-2 Systems: Upgrading to 2.0.6.2.2

The readme content structure layout of the 21445837-Virtual.zip:

21445837-Virtual
|
| - README.html
| - exalogic-lctools-14.2.4-release-notes.txt
| - docs/README.html
| - docs/components/Enable_http_zfs.html
| - docs/components/ECServices_Upgrade.html
| - docs/components/ILOM_ComputeNode_Upgrade.html
| - docs/components/Index.html
| - docs/components/Menu.html
| - docs/components/NM2_Upgrade.html
| - docs/components/VServers_Upgrade.html
| - docs/components/ZFS_Upgrade.html

Pre-requisites

O2CB Cluster Timeout

Before patching, verify that the cluster heartbeat timeout is set correctly, as documented in the Document ID : 1995593.1 - Increase O2CB Cluster Heartbeat Timeout on Exalogic Virtual

PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racks

Refer to following Note which has information on this important pre-requisite check.

<Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Appendices

Appendix A: Fixed Bugs List

Please review Doc ID 2057633.1: Exalogic Infrastructure October 2015 PSU – Fixed Bugs List

Appendix B: Patching Known Issues

The pre-requisite step to update Weblogic Server configuration properties for non-default passwords prior to upgrading Exalogic Control Services fails

Symptoms

A pre-requisite step is included in section entitled "1.1 Prerequisites" to configure Weblogic Server prior to upgrading Exalogic Control Services to handle non-default passwords configured for the weblogic user. Running the command as documented results in an error as shown below:

[oracle@elctrl ~]$ /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh weblogic <non-default-password>

-bash: /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh: Permission denied

Invoking it using sh as follows runs into ClassNotFoundException, as shown in the sample below:

[oracle@elctrl ~]$ sh /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh weblogic welcome1
Exception in thread "main" java.lang.NoClassDefFoundError: weblogic.WLST
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
Caused by: java.lang.ClassNotFoundException: weblogic.WLST not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
[oracle@elctrl ~]

Cause

A step to set the WLS environment variables is missing in the documentation. Also, the wlconfig.sh script is not executable; it needs to be invoked using "sh".

Solution/Workaround

Login as root to the Exalogic Control VServer
Switch to user oracle

[root@elctrl ~]# su - oracle
[oracle@elctrl ~]$
Run the following command to set the WLS environment variables:

[oracle@elctrl ~]$ cd /u01/app/oracle/Middleware/wlserver_10.3/server/bin
[oracle@elctrl bin]$ . ./setWLSEnv.sh

CLASSPATH=/u01/app/oracle/Middleware/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/Middleware/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/java/lib/tools.jar:/u01/app/oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/Middleware/modules/features/weblogic.server.modules_10.3.5.0.jar:/u01/app/oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/u01/app/oracle/Middleware/wlserver_10.3/server/bin:/u01/app/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/u01/app/oracle/java/jre/bin:/u01/app/oracle/java/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin

Your environment has been set.

NOTE: Ensure that the above command is run with a dot and a space preceding ./setELSEnv.sh as shown.
Run the wlsconfig.sh command to configure the non-default password for the weblogic user

[oracle@elctrl bin]$ cd /u01/app/oracle/ovm-manager-3/weblogic
[oracle@elctrl weblogic]$ sh ./wlconfig.sh weblogic <non-default-password>

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_adf_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Using existing user key file...
The username and password that were used for this WebLogic Server connection are stored in wlconfig.properties and wlkey.properties.
[oracle@elctrl weblogic]$

After above steps you may proceed with updating Exalogic Control Services.

Exapasswd: EMOC sync fails for Exapasswd with special characters.

Symptoms

If exapasswd is used with certain special characters synchronization with EMOC would fail. Observed following error messages:

Error while synchronizing passwords with Ops Center

Cause

The issue is with following special characters:

1) ` (acute or back quote)
2) : (colon)
3) ; (semi-colon)

Solution/Workaround

It is known limitation that exapasswd does not support quotes (double/single) also. And it has already been documented.

These new findings will also be updated in the Expasswd user's guide accordingly.

Many oclib (/var/mnt/virtlibs/*) Mounts Seen On EMOC Control VServer.

Refer to <Note 2053733.1> for information on this known issue.

ZFS Upgrade Failed. Storage node ilom console indicates error in joining cluster.

Symptoms

While applying the July 2015 PSU on X5-2 rack, the upgrade of ZS3-ES may fail. The exapatch fails with:-

"ERROR: Could not ssh to ZFS host xx.xx.xx.xx after upgrade. "

Further,

When I see the storage node ilom of the failed head, it is stuck at this stage :

Oracle ZFS Storage ZS3-ES Configuration
    Copyright (c) 2008, 2015, Oracle and/or its affiliates. All rights
reserved.

    NET-0 <=>  NET-1 <=>  NET-2 <=>  NET-3 <X>

    SUNW-MSG-ID: AK-8000-2U, TYPE: Defect, VER: 1, SEVERITY: Major
    EVENT-TIME: Tue Jun  2 17:01:26 UTC 2015
    PLATFORM: i86pc, CSN: 1427NML0E5, HOSTNAME: scat01sn02
    SOURCE: aktty, REV: 1.0
    EVENT-ID: 1fc05eee-db80-e94d-d6c2-c06714389111
    DESC: The appliance experienced an unrecoverable protocol error while
    attempting to join a cluster of appliances.  Refer to
    http://sun.com/msg/AK-8000-2U for more information.
    AUTO-RESPONSE: No automated response is possible for this defect.
    IMPACT: The appliance cannot be configured into the cluster.
    REC-ACTION: Reboot the system and attempt to configure the cluster again.
    Contact your vendor for support.

    ESC-3: Halt   ESC-4: Reboot   ESC-5: Info

    For help, see http://www.oracle.com/goto/ZS3-ES/docs/

Cause

@ Details of this issue are in this MOS note:
@ .
@ Oracle ZFS Storage Appliance: ZS3-X Clustron Card in PCIE Slot 2 Does Not
@ Initialize During boot. (Doc ID 1910623.1)
@ .
@ The following bug is referenced from the MOS note:
@ .
@ BUG 18672251 - AK-8000-2U: unrecoverable protocol error on the ZS3-ES
@ (X4170M3) appliance

Solution/Workaround

Manually reboot the storage head with ESC-4. Refer to <Note 1910623.1>

___________________________________________________________________________________________________________________________________________________________________________________

PCIE fault HW alert in ZFS post IB firmware update

Symptoms:

PCIE related HW fault is reported in ZFS, after applying IB card firmware.

Error message is Similar to:-

SUNW-MSG-ID: SPX86-8003-QH, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Aug 17 06:34:36 2015
PLATFORM: i86pc, CSN: 1441NML0FE, HOSTNAME: denp04sn02
SOURCE: appliance/kit/akd:default, REV: 1.0
EVENT-ID: d63362ca-89ea-e1e9-90cb-ee2bc152e335
DESC: An Integrated I/O (II0) non-fatal error in downstream PCIE device has
occurred.
AUTO-RESPONSE: The service-required LED on the chassis will be illuminated.
IMPACT: System continues to run with degraded resources.
REC-ACTION: Contact your service provider for proper repair procedures.

Cause:

A Hermon IB card firmware upgrade was active on the card.

Solution/Workaround:

Workaround is mentioned in Mos <Note 2040757.1>

PSU Upgrade fails while upgrading ZFSSA.

Symptoms: The ZFSSA upgrade will fail when applying the appliance from 2011 version of the software to the 2013 version of the software. Exapatch terminated as follows .....

INFO: ZFS-Storage-Head xx.xx.xx.x2 is active ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 is ready ZFS head.

INFO: ZFS-Storage-Head xx.xx.xx.x1 running pre-patch checks for ready head

INFO: ZFS-Storage-Head xx.xx.xx.x1 Ready head checks completed.

INFO: ZFS-Storage-Head xx.xx.xx.x1 successfully completed all pre-patch checks

ERROR: ZFS-Storage-Head xx.xx.xx.x2 STOP!! There are 4 active problem(s) on this head, Please resolve them first, Exiting

Unable to proceed as pre-patch checks failed for active head xx.xx.xx.x2

Additional information may be found in the console output and the log file: /var/log/exapatch_20150818205629.log

Errors are similar to the following:

problem-000 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-001 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-002 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

problem-003 2015-4-5 02:51:11 Major Fault ZFS device 'id1,sd@SATA_____TOSHIBA_THNSNC51________XXXXXXXXXXXX/a' in pool 'exalogic' failed. Major Fault Never

Cause: The ZFSSA upgrade will fail when applying the Oct 2015 PSU due to bug
19024367 with the ZFSSA. This is a known issue that can happen when upgrading
from the 2011 version of the software to the 2013 version of the software.

After the head that has been upgraded becomes active, it tries to access the
readzilla on the passive head. The readzillas are local to head each head and
one head cannot access the readzilla of its peer. Typically the active head
is supposed to ignore the readzilla disks on its peer, but due to bug
19024367, these disks are marked as failed on the active head. The passive
head will not show any failed disks.

To validate this compare the serial numbers of the disks marked as failed
with the serial numbers of the readzilla disks on its peer.

This is a false positive and can be safely ignored.

Solution/Workaround:

Since Exapatch does not allow you to proceed, the software on the of the
remaining head must be upgraded manually.

Note: After the manual update if still the issue exist, reboot the storage head.

Guest vServer update fails during yum update of rpms with 'up2dateErrors'

Refer to Exalogic PSU Known Issues <Note 1571367.1>, section "Guest vServer update fails during yum update of rpms with 'up2dateErrors'" for information on this known issue.

Exapatch console output message has ' WARNING: unable to update patch history: '.

Symptoms

While running the Exapatch command, the console output has following warning message:-

WARNING: unable to update patch history: Unable to find unique identifier "1352NML06M" for Compute-Node xx.xx.xx.xx in the rack history file.

Cause

This is an expected behavior when the rack history file is not created beforehand and does not affect the node functionality.

Solution

To avoid this, prior to patching a rack history should be created:

[root@compute-node]# /exalgic-lctools/bin/exapatch -a refreshHistory

______________________________________________________________________________________________________________________________________________________________________

Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"

Refer to <Note 2092033.1> for details on this known issue.

_____________________________________________________________________________________________________________________________________________________________________________________

Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Refer to <Note 2114516.1> for details on this known issue.

Appendix C: Errata

1. Migration of assets managed by Proxy Controllers

In the troubleshooting MOS note section "Problem: components are listed under both the ProxyControllers(PC)"

Original Text

In the Exalogic Control BUI, if it is observed that any of the component asset is listed under both the Proxy Controllers (PC), they need to be migrated to a single PC.

Updated Text

For a given switch (NM2-GW switches, NM2-36p switch), only one proxy controller needs to be managing it. If a particular switch appears to be managed by both PC1 and PC2, it must be migrated to one of the proxy controllers. It does not matter whether it is migrated to PC1 or PC2.

One or more compute nodes may appear in the Managed Assets list of both proxy controllers. For instance, el01cn01.example.com may appear as being managed by both PC1 and PC2. This is expected behavior; no migration is required for the compute nodes.

Other non-switch assets, such as ZFS storage heads and PDU may also appear in the Managed Assets list of both proxy controllers. Migration is not required for these assets prior to applying the July 2015 PSU; migration is required only for switches.

Login to EMOC BUI and navigate to "Administration" item in the left panel and find the entries for 'PC1' and 'PC2' vServers.

Select a Proxy Controller say, "PC1" in the left panel. In the center panel click on the "Managed Assets" tab and set the Asset Type Filter to "Network Switches" to get list of switches managed by the 'PC1' Proxy Controller.

Select the switch that you wish to migrate to the other ProxyController "PC2". Click on the icon that provides the option to "Migrate Assets". A confirmation dialog shows up, select 'Migrate' button to proceed.

Once the migration completes finishes, a notification pop-up appears at the bottom right corner of the EMOC BUI, confirming the successful migration.

______________________________________________________________________________________________________________________________________________________________

Appendix D: FAQs

Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

Refer to <Note 1997466.1> for details

Guest vServer patching FAQ's

Refer to <Note 2031749.1> for details.

References

<NOTE:2092033.1> - Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"
<NOTE:1571367.1> - Exalogic Infrastructure PSU Upgrade - Known Issues
<NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade
<NOTE:1530781.1> - Exalogic Infrastructure Physical and Virtual Releases/PSUs – Software and Firmware Version Information
<NOTE:2053733.1> - Many oclib (/var/mnt/virtlibs/*) Mounts Seen On EMOC Control VServer After Upgrading To Exalogic April 2015 PSU 2.0.6.2.1 Or Later Versions
<NOTE:2016700.1> - Exalogic: vServer Creation Takes a Long Time or Fails After a Few Days of Uptime
<NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note
<NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic
<NOTE:1995236.1> - Exalogic Patch Set Update (PSU) Release 2.0.6.2.1 (Linux - Virtual) for April 2015
<NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic
<NOTE:2087741.1> - Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Attachments

This solution has no attachment