Exalogic Patch Set Update (PSU) Release 2.0.6.2.1 (Linux

Asset ID:	1-79-1995223.1
Update Date:	2016-06-23
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1995223.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.1 (Linux - Physical) for April 2015

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.0.0.0 to 2.0.6.2.0
Exalogic Elastic Cloud X3-2 Hardware - Version X3 to X3 [Release X3]
Exalogic Elastic Cloud X5-2 Hardware - Version X5 to X5 [Release X5]
Exalogic Elastic Cloud X4-2 Hardware - Version X4 to X4 [Release X4]
Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 to X2 [Release X2]
Linux x86-64

Purpose

Oracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around April 2015 Patch Set Update (PSU) for that system.

Scope

The target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU.

This document provides the following:

README files that are included in the April 2015 PSU for convenience. These can be used for reference without having to download the PSU first.
Known issues with the PSU - these provide a list of known issues along with workarounds/solutions, if available. Engineers and administrators planning to apply the PSU should ensure to review these before applying the PSU.

This document will be kept up-to-date with updates to errata and known issues.

Details

NOTE:

This PSU contains upgrades to both OL5, OL6 releases.
This PSU includes updates to ELLC 14.2.2

Patch Download

Released: April 2015

Product Version: 2.0.6.2.1 (on X2-2/X3-2/X4-2/X5-2) for Oracle Linux x86-64

PATCH 20448180 - EXALOGIC LINUX 2.0.6.2.1 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR APRIL 2015

Patch Readme Documentation

Refer to the readme documentation attached 20448180-Linux.zip to this Document on how to upgrade the Exalogic infrastructure:

Exalogic X2-2,X3-2,X4-2 and X5-2 Systems: Upgrading to 2.0.6.2.1

The readme content structure layout of the 20448180-Linux.zip:

20448180-Linux
|
| - README.html
| - exalogic-lctools-14.2.2-release-notes.txt
| - docs/README.html
| - docs/components/ILOM_ComputeNode_Upgrade.html
| - docs/components/Index.html
| - docs/components/Menu.html
| - docs/components/NM2_Upgrade.html
| - docs/components/ZFS_Upgrade.html

Known Issue with data corruption in ZFS Storage Appliance Software (2013.1.3.5) bundled in April 2015 PSU

There is a known issue with data corruption in the ZFS version (2013.1.3.5) bundled in the April 2015 PSU. Please refer to the following MOS Note for details:

<Note 2006121.1>: Exalogic: Risk of Data Corruption in ZFS Storage Appliance Software (2013.1.3.5) Bundled in April 2015 PSU

All applications of April 2015 PSU MUST apply the updated ZFS version provided through patch 20983458 as a replacement to the version included in the PSU.

INTERNAL

Pre-requisites

PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racks

Refer to following Note which has information on this important pre-requisite check.

<Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks

Appendices

Appendix A: Fixed Bugs List

Please review Document ID : 1995246.1 - Exalogic Infrastructure April 2015 PSU - Fixed Bugs List.

Appendix B: Patching Known Issues

During the April 2015 PSU patching, compute nodes will crash if the IB master subnet manager is not running on an NM2-GW switch on the Exalogic rack

Symptoms

When patching IB switches on an Exalogic rack, some or all compute nodes lose IB communication capabilities. This causes them to reboot.

Cause

The problem is that when an IB master subnet manager's host IB switch is being upgraded it needs to reboot. Therefore, before the upgrade, the subnet manager daemon on the NM2-GW switch is disabled. The other IB components on the rack running the subnet manager daemon negotiate a new master subnet manager. If the new master subnet manager ends up on a device other than an NM2-GW switch on the Exalogic rack, compute nodes will lose their IB connectivity and then crash due to the fact that they cannot communicate with the cluster.

Solution/Workaround

The workaround is to follow the solution documented in following Note.

<Note 1682501.1>: Setting up the subnet manager in a multirack configuration containing Exalogic/BDA and Exadata/SSC/Expansion Rack.

Note that the configuration must be correct on ALL IB subnet managers on the IB fabric, and is not limited to the IB switches on the Exalogic rack.

_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Upgrading NM2 Switch Firmware on full rack Exalogic systems or multi-rack Exa systems causing Network outage

Note:

The issue does not occur when Exalogic racks (one eighth, quarter, half ) are connected to Exadata racks.

Symptoms

Network outages on the InfiniBand fiber.

Cause

The NM2 Port ID space has increased four times from firmware version 2.0 to 2.1. Therefore, port ID collisions may occur if NM2 switches with firmware version 2.0 and 2.1 are run on the same fabric. This issue can cause network outages on the InfiniBand fiber.

Solution/Workaround

In order to prevent this issue, manipulation of the switch 'GWInstance' value may be required to ensure unique port IDs are generated by performing the following steps:

Log in to each NM2-GW switch as root and run:

# showgwconfig

Note the GWInstance "Running Value" on each of the switches.

1. Ensure that all the GWInstance values are even numbers
2. Ensure that GWInstance values when multiplied by 4, are greater than the largest GWInstance value.

To change a GWInstance value to 16, run:

[root@nm2gw-ib01 ~]# setgwinstance 16
Stopping Bridge Manager.. [ OK ]
Starting Bridge Manager. [ OK ]
root@nm2gw-ib01 ~]#

For example, if the GWInstance values of four NM2 switches are 10, 20, 30 & 40, it will be necessary to change the GWInstance value of 10 to 16.

ExaPatch 1.2.4 throws unwanted ifconfig warning messages while running on OEL6

Symptoms

ExaPatch 1.2.4 throws unwanted ifconfig warning messages while running on OEL6:-

***************************************************************************
Ifconfig uses the ioctl access method to get the full address information,
which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are displayed
correctly.
Ifconfig is obsolete! For replacement check ip.
**************************************************************************

Cause

'/sbin/ifconfig' is obsolete in OL6 (yet it is fully functional). Users are expected to use the 'ip' command instead.

Solution/Workaround

Ignore this warning message. The upgrade procedure is not affected by this though.

ExaPatch fix targeted in upcoming PSU . (see bug: 20669133)

CheckHWnFWProfile failed with syntax error: invalid arithmetic operator

Symptoms

CheckHWnFWProfile failed running on CN with error message

"./CheckHWnFWProfile: line 147: local: xxx.xx: syntax error: invalid arithmetic operator (error token"

Cause

The CheckHWnFWProfile is not handling floating point number comparison correctly while checking the registered memory:

line 147, it is like below:

local -i actualMemGB=`dmidecode --type 17 | grep Size | grep MB | awk '{ print $2; }' | awk '{total = total + $1}END{print total/1024}'`

In some cases, the above computation can return a float value for e.g. 511.984, while actualMemGB is defined as a integer and the subsequent integer comparison in the script with expectedMemGB fails.

Solution/Workaround

Keep a backup of /opt/exalogic.tools/tools/CheckHWnFWProfile and edit /opt/exalogic.tools/tools/CheckHWnFWProfile, change the line 147 to:-

local -i actualMemGB=$(printf '%.0f\n' `dmidecode --type 17 | grep Size | grep MB | awk '{ print $2; }' | awk '{total = total + $1}END{print total/1024}'`)

It will round off to integer value and comparison will succeed.

Failure to stop OSWatcher (oswbb) service while applying April 2015 PSU

Refer to <Note 2006110.1> for information on this known issue.

Exapatch fails with Error "FAILED - yum repo for EECS 2.0.6.2.1 (eecs_20621_updates) NOT configured properly"

Symptoms

Exapatch fails with following Error "FAILED - yum repo for EECS 2.0.6.2.1 (eecs_20621_updates) NOT configured properly"

INFO: Sat Jun 6 12:30:45 EDT 2015: Skipping custom repo. Custom repodata not found at http://10.0.213.14/shares/export/common//exalogic-lcdata/patches/Virtual/20383732/Infrastructure/BaseTemplateOEL5Supplementary/OracleLinux_5.8/Custom/repodata/repomd.xml
ERROR: Sat Jun 6 12:30:46 EDT 2015: FAILED - yum repo for EECS 2.0.6.2.1 (eecs_20621_updates) NOT configured properly. Please FIX manually before running this script
INFO: Sat Jun 6 12:30:46 EDT 2015: >>>> remove services that launch post patching procedures
cat: /var/log/egbt_post_20621.log: No such file or directory

Cause

If you dig deeper, you can see the issue is “KeyError: 'gpg_key_url'

#yum -c /opt/egbt_upgrade/BaseTemplateOEL5/scripts/eecs_yum.conf repolist | grep eecs_20621_updates
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 309, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 157, in main
base.getOptionsConfig(args)
File "/usr/share/yum-cli/cli.py", line 187, in getOptionsConfig
self.conf
File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 665, in <lambda>
conf = property(fget=lambda self: self._getConfig(),
File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 254, in _getConfig
self.plugins.run('init')
File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 179, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/lib/yum-plugins/rhnplugin.py", line 168, in init_hook
repo = RhnRepo(channel)
File "/usr/lib/yum-plugins/rhnplugin.py", line 301, in __init__
self.gpgkey = get_gpg_key_urls(channel['gpg_key_url'])
File "/usr/share/rhn/up2date_client/rhnChannel.py", line 48, in __getitem__
return self.dict[item]
KeyError: 'gpg_key_url'

Solution/Workaround

This is a known issue ( see Bug 21259957 ) The workaround is to comment out everthing in the /etc/sysconfig/rhn/sources file except the following line.

up2date default

ExaPatch fix targeted in JULY 2015 PSU (2.0.6.2.2). (see bug: 21259957)

____________________________________________________________________________________________________________________________________________________________________________

Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"

Refer to <Note 2092033.1> for details on this known issue.

____________________________________________________________________________________________________________________________________________________________________________

Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade

Refer to <Note 2114516.1> for details on this known issue.

Appendix C: Troubleshooting

ExaPatch prePatchCheck option fails with ERROR:FAILED - /opt/exalogic.tools/CheckHWnFWProfile.

Problem

ExaPatch prePatchCheck option fails with ERROR:

FAILED - /opt/exalogic.tools/tools/CheckHWnFWProfile

CheckHWnFWProfile may have failed due to valid upgraded firmware but /usr/lib/init-exalogic-node/supported_config file still references original firmware version. For e.g.,

Supported disk controller Version: 12.12.0-0079

Current disk controller Version  : 12.12.0-0178

Disk controller is not at the supported firmware version. It requires firmware update...

Supported disk controller Firmware at: /opt/exalogic/firmware/disk_controller/12_12_0_0079.rom

Solution

After making sure there are no other errors, ignore this precheck and continue patching. When patching the compute node, please use the -f flag with Exapatch command line option.

For e.g., to override pre-patching checks while patching compute nodes, run the following command:

[root@compute-node2]# /exalogic-lctools/bin/exapatch -a patch cn -h <compute_node_ip> --force

Appendix D: FAQs

Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching

Refer to <Note 1997466.1> for details

References

<NOTE:2006110.1> - Exalogic: Failure to stop OSWatcher (oswbb) service while applying April 2015 PSU
<NOTE:2006121.1> - Exalogic: Risk of Data Corruption in ZFS Storage Appliance Software (2013.1.3.5) Bundled in April 2015 PSU
<NOTE:2068546.1> - On Exalogic Physical Compute Nodes Interface Hardware Addresses Are Set To Zeros (00:00) For IPoIB Interfaces Using Custom Partitions After Applying Either April Or July 2015 PSU
<NOTE:2087741.1> - Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks
<NOTE:2092033.1> - Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"
<NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade
<NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic
<NOTE:1997466.1> - Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching
<NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic
<NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note

Attachments

This solution has no attachment