Oracle Exalytics Known Issues After Patchset (PS) Install

Asset ID:	1-79-1543836.1
Update Date:	2016-11-21
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1543836.1 : Oracle Exalytics Known Issues After Patchset (PS) Install

Applies to:

Oracle Exalytics Software - Version 1.0.0.1.0 and later
Exalytics In-Memory Machine X3-4 - Version All Versions and later
Exalytics In-Memory Machine X4-4 - Version All Versions and later
Exalytics In-Memory Machine X2-4 - Version All Versions and later
Linux x86-64

Purpose

This document provides detailed information on known installation issues for Oracle Exalytics Patchsets. Affected Patchset (PS) releases are indicated.

Scope

This document is intended for systems administrators or network specialists who may be involved in configuring or patching the Exalytics server. Installation of these patches is accomplished by following the steps in the patch readme files and the Oracle Fusion Middleware Installation and Administration Guide for Oracle Exalytics In-Memory Machine 11g Release 1 (11.1.1). This document includes issues that may be encountered while patching an Exalytics server.

Details

The Exalytics release notes describe known issues and workarounds and should always be checked before doing any new installs or patchset installs. Release notes for each Exalytics release (i.e. 1.0.0.1.0, 1.0.0.4.0, etc.) are available on the Exalytics documentation index. The items listed below are known issues that were found after applying Exalytics Patchsets on an Oracle Exalytics server and that may not be included in the release notes. Affected Patchset (PS) releases are indicated.

FIRMWARE

PS2 Firmware Update Causes Unexpected Server Shutdown and Requires Manual Restart

Symptoms: When applying the firmware updates for Exalytics PS2, as described in Section 7.3.5 Installing the Oracle Exalytics Release 1 Patchset 2 of the Oracle Fusion Middleware Installation and Administration Guide for Oracle Exalytics In-Memory Machine Exalytics X2-4 Release 1 (1.0), the ILOM is disconnected and the server is unavailable. The server is shutdown and does not start automatically.

Affected Releases: Exalytics release 1.0.0.2.0

Bug Number: <Bug 16530596> - EXALYTICS PS2 FIRMWARE UPGRADE CAUSED UNSCHEDULED OUTAGE AND REQUIRED REBOOT

Cause: Bug 16530596 is still under review by Exalytics development and the exact cause is undetermined at this time.

Solution: Undetermined. Bug 16530596 is still under review by Exalytics development.

Workaround: Log into the ADM server (via putty) and restart the server on the command line:

start /SYS
Are you sure you want to start /SYS (y/n)? y
Starting /SYS

Applying PS3, or later, Firmware Update Causes Exalytics Server to Reboot Automatically

Symptoms: When applying firmware update, either as part of Exalytics Patchset (PS) install, or otherwise, the server is automatically rebooted when the update is finished.

Affected Releases: Exalytics release 1.0.0.3.0 and later

Bug Number(s): <Bug 20671945> ADDITIONAL NOTE IN UPGRADE FIRMWARE and <Bug 20565163> HOST SHUTDOWN EVEN BIOS WAS NOT UPDATED DURING ILOM FIRWMARE UPGRADE

Cause: Expected behavior. Firmware updates are expected to automatically reboot the server. The server will restart automatically.

Solution: No action required. Wait for server restart and then continue with any operations.

Workaround: If automatic reboot is not desired, you may select the option "Delay BIOS Upgrade." This is an option that is displayed during the Firmware upgrade process. If you select the option, to "Delay BIOS Upgrade" the server will not be rebooted automatically.
If this option, is not checked, the server will automatically reboot. This will happen irrespective of whether the patch includes a BIOS update.

BASE IMAGE / IMAGE VERSION

Imageinfo Version Unchanged or Incorrect After Patchset (PS) Installation and Shows "Exalogic" Instead of "Exalytics"

Symptoms: After installing Exalytics Patchsets (PS) and image updates, the imageinfo commands continues to show "Exalogic" and may also show an unexpected version. Some examples of this problem are:

After PS1 installation, the version remains unchanged and still shows "Exalogic" in the output:

$imageinfo
Exalogic 1.0.0.3.0 (build:r)
 
Image version       : 1.0.0.3.0
Image build version :
Creation timestamp  : 2011-12-21 21:35:57 -0500
Kernel version      : 2.6.32-100.23.1.el5
Image activated     : 2012-03-25 21:59:21 -0400
Image status        : SUCCESS

After PS 3 install (image version 1.0.0.5.0), the imageinfo command still shows the PS2 version number (1.0.0.4.0) even though the patch was successfully applied:

# /opt/exalogic/usr/sbin/imageinfo
Exalogic 1.0.0.4.0 (build:r)

Image version : 1.0.0.4.0
Image build version :
Creation timestamp : 2013-06-19 11:28:36 +0000
Kernel version : 2.6.32-100.23.1.el5
Image activated :
Image status :

Affected Releases: Exalytics base release and Patchset version PS1, PS2 and PS3

Bug Number(s):

<Bug 14701771> - IMAGEINFO Reports Version 1.0.0.3.0 On Exalytics After Applying 1.0.0.3.1 Patch

<Bug 15933998> Exalytics Base Image 2.0.1.1.0 Has Incorrect Imageinfo

<Bug 14845417> Exalytics Image 1.0.0.4 Has Incorrect Imageinfo

<Bug 17375101> PS4: IMAGEINFO Command Does not Exist

Cause: Oracle Exalytics servers are built on the same basic model as Exalogic servers. In the initial releases, the Exalytics utilities were still referencing the Exalogic machine instead of Exalytics. This issue is corrected in PS3 by introducing a new "exalytics_imageinfo" command instead of "imageinfo."

Solution: In PS1 through PS2, the incorrect version number and "Exalogic" in the imageinfo output can be safely ignored. The misreported versions do not indicate incomplete patch installation. Starting with PS3, the "exalytics_imageinfo" command should be used instead.

For example:

# /opt/exalytics/bin/exalytics_imageinfo
Image version : 1.0.0.5
Creation timestamp : Wed 19 Jun 2013 12:26:49 PM BST
Kernel version : 2.6.32-100.23.1.el5

RPM versions:
kernel-2.6.32-100.23.1.el5
exalytics-container-bm-1.0.0.5-23
exalytics-scripts-1.0.0.5-36
exalytics-flash-1.0.0.5-42

Exalytics PS5 (version 1.0.0.7.0) Image Update Script, update_bm_1.0.0.6_to_1.0.0.7.sh, Fails To Install New RPMS

Symptoms: The update_bm_1.0.0.6_to_1.0.0.7.sh to upgrade the base image to version 1.0.0.7.0 fails with errors:

ERROR: Failed to install new rpms
/opt/exalytics/update/ps3tops4.d/70install_exalytics-deps.sh failed; return
code: 1
Running script: /opt/exalytics/update/ps3tops4.d/72fix_megacli_libsysfs.sh
...
/opt/exalytics/update/ps3tops4.d/72fix_megacli_libsysfs.sh completed OK
Failed to run scripts

Additional errors are in the update log:

file:///mnt/exalyticsPS4/Server/repodata/repomd.xml: [Errno 5] OSError: [Errno 2] No such file or directory: '/mnt/exalyticsPS4/Server/repodata/repomd.xml'
Trying other mirror.
...
Error Downloading Packages:
iwl5000-firmware-8.24.2.12-3.el5.noarch: failure:
iwl5000-firmware-8.24.2.12-3.el5.noarch.rpm from
exalytics_1.0.0.6_x86_64_base: [Errno 256] No more mirrors to try.
perl-XML-Simple-2.14-4.fc6.0.1.noarch: failure:
perl-XML-Simple-2.14-4.fc6.0.1.noarch.rpm from exalytics_1.0.0.6_x86_64_base:
[Errno 256] No more mirrors to try.
...

Bug Number(s): <BUG 20724568> - APPLYING PS5 FAILS WITH ERROR

Cause: As part of the installation of Exalytics Patchset 3 and 4, a local, internal only, yum repository is enabled. Accordingly, any servers with PS4 installed will have this local yum repository enabled. The PS5 installation steps omitted references to the existing repositories and when the image update tries to load, it fails due to a conflict with the existing repositories.

Solution: A documentation bug was opened to correct the missing references and additional steps. Until the documentation bug is implemented, the workaround listed below may be used.

Workaround: Disable the PS3 and PS4 internal yum repository before running the PS5 image update script using the following steps:

1. As root user:

cd /etc/yum.repos.d

2. Edit the file exalytics-1.0.0.5.repo, and change the line 'enabled=1' to enabled=0. Save the changes.
3. Edit the file exalytics-1.0.0.6.repo, and changed the line enabled=1 to enabled=0. Save the changes.
4. Then confirm settings with command:

# yum repolist all
Loaded plugins: security
repo id repo name status
exalytics_1.0.0.5_x86_64_base Exalytics release 1.0.0.5 Base (x86 disabled
exalytics_1.0.0.6_x86_64_base Exalytics release 1.0.0.6 Base (x86 disabled
exalytics_1.0.0.7_x86_64_base Exalytics release 1.0.0.7 Base (x86 enabled: 3,498repolist: 3,498

5. Once confirmed, re-run the upgrade script:

./update_bm_1.0.0.6_to_1.0.0.7.sh file:///mnt/exalyticsPS5/Server

NOTE: This same error has been known to occur when upgrading from PS3 to PS4 on bare metal servers. If you encounter this error, you can use the same workaround, but the revised steps are:

Disable the local/internal yum respository for release 1.0.0.5 (PS3) and then re-attempt the install. Use the following steps:

a. As root user:

cd /etc/yum.repos.d

b. Edit the file exalytics-1.0.0.5.repo, and change the line 'enabled=1' to enabled=0. Save the changes.

c. Re-attempt patch install.

Smartd Error: "smartd[10687]: Device: /dev/sdd [SAT]...Offline uncorrectable sectors"

Symptoms: When using Exalytics, or Oracle Virtual Machine (OVM) on Oracle Exalytics, multiple errors related to the smartd device are reported in the /var/log/messages:

Jun 12 00:15:38 EXALYTICS_OVS02 smartd[10687]: Device: /dev/sdd [SAT], 244302034763776 Offline uncorrectable sectors
Jun 12 00:15:38 EXALYTICS_OVS02 smartd[10687]: Device: /dev/sde [SAT], 223054831550464 Offline uncorrectable sectors
Jun 12 00:15:38 EXALYTICS_OVS02 smartd[10687]: Device: /dev/sdf [SAT], 197830488621056 Offline uncorrectable sectors

Affected Releases: Exalytics versions 1.0.0.0.0 through 1.0.0.3.0; and 1.0.0.0.5 (fixed in PS4, release 1.0.0.4.0 and release 2.0.0.0)

Bug Number(s): Internal, unpublished <Bug 17463303> SMARTD LOGS "OFFLINE UNCORRECTABLE SECTORS" FOR ALL FLASH MODULES IN EXALYTICS and <Bug 20836508> - EXALYTICS SMARTD LOGS "OFFLINE UNCORRECTABLE SECTORS" FOR ALL FLASH MODULES

Cause: Servers with F40 Flash Accelerator cards installed may see smartd messages logged stating that "Offline uncorrectable sectors" are occurring. This is due to a difference in the way that smartd parses the smart data provided by the F40 cards. Smartd is reading the bytes that LSI uses to represent the number of sectors read since power on as the value representing the uncorrectable sectors. Systems experiencing this issue will see messages similar to the following reported for each F40 module (24 drives) and it will repeat every 30 minutes:

Sep 18 09:54:00 exalytics0 smartd[11036]: Device: /dev/sdy, 104917461106688 Offline uncorrectable sectors

Solution: These messages are erroneous and may be ignored. The messages do not indicate any actual hardware failure and do not impact the systems stability, or performance. Workaround is to disable the smartd daemon and prevent messages from being logged. Use the following chkconfig commands. These commands will turn smartd off and will confirm the settings. A reboot is required for this change to take affect.

# chkconfig smartd off
# chkconfig --list smartd
smartd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
# reboot

NETWORK

Loss of connectivity over infiniband after reboot

Symptoms: When the Exalytics server is rebooted, Infiniband connections are lost and IB hosts are not reachable.

Affected Releases: Exalytics releases 1.0.0.1.0, 1.0.0.2.0, 1.0.0.3.0

Bug Number: <Bug 16615228> - EXALYTICS INFINIBAND DOWN AFTER PS2 INSTALL OR REBOOT and Related Exalogic <Bug 13322333> EXALOGIC - INFINIBAND CARDS COULD NOT WORK AFTER REBOOT.

Cause: This matches a known Exalogic bug where network scripts (i.e. openib) are getting loaded after the network servers (netfs). During reboot the load order priority is reset and causes infiniband to not work. This happens because of the line "Required-Start: $local_fs" inside "/etc/init.d/openibd" file. If the line is present, (even though commented), the load order is incorrect and IB will not work. If the line is completely removed, it will load openib before loading the network service and then IB works as expected.

Solution: Bug is still under review by Oracle Exalytics Development.

Note: As temporary measure, you can workaround this issue by doing the following:

a) Edit the "/etc/init.d/openibd" file to completely remove the line that reads:

"# Required-Start: $local_fs"

Modified file would show:

### BEGIN INIT INFO
# Provides:       openibd
# Required-Stop: opensmd
# Default-Start: 2 3 5
# Default-Stop: 0 1 2 6
# Description:    Activates/Deactivates InfiniBand Driver to \
#                 start at boot time.
### END INIT INFO

b) Save changes and then run following commands as root user:

chkconfig openibd off
chkconfig openibd on

c) Verify that only S05openibd exists

ls /etc/rc3.d/S05openibd
ls /etc/rc3.d/S26openibd

d) If s26openibd exists, then run following commands:

# cd /etc/rc.d/rc3.d
# mv S26openibd S05openibd

Failed Connectivity Tests When Running CONFIGURE_NETWORK_FOR_EXALYTICS.SH Script

Symptoms: After running the Exalytics CONFIGURE_NETWORK_FOR_EXALYTICS.SH script, (either as part of PS1 or PS2 installation, or new install), the 10gE interface is not working. When running the network checks at the end of the script execution, you may see messages like:

Warning failed connectivity test to 10.141.123.1 ... [where specified IP is your bond1 10gE IP address]

Affected Releases: Exalytics releases 1.0.0.1.0, 1.0.0.2.0 and 1.0.0.3.0

Bug Number: <Bug 16478734> - CONFIGURE_NETWORK_FOR_EXALYTICS.SH NOT CREATING RULE-BOND1 & ROUTE FILES

Cause: Bug 16478734 - CONFIGURE_NETWORK_FOR_EXALYTICS.SH NOT CREATING RULE-BOND1 & ROUTE FILES is under review by Oracle Exalytics development. It seems that the script is failing to create or update needed network files:

/etc/sysconfig/network-scripts/rule-bond1
/etc/sysconfig/network-scripts/route-bond1
/etc/iproute2/rt_tables

Solution: Bug is still under review by Oracle Exalytics Development and a full solution is pending.

A known workaround is to manually create the /etc/sysconfig/network-scripts/rule-bond1 and /etc/sysconfig/network-scripts/route-bond1 files and then update the /etc/iproute2/rt_tables.

For more information on configuring these files, please see http://www.kernel.org/doc/Documentation/networking/bonding.txt or review the README.bonding on the Exalytics server. The README file is owned by the "root" user and is located in the directory like:

/usr/share/doc/iputils-20020927

For example:

# rule-bond1
from 10.141.133.36/32 table T1 [where specified IPs are your bond1 10gE IP addresses]
to 10.141.133.36/32 table T1

# route-bond1
10.141.132.0/23 dev bond1 src 10.141.133.36 table T1 [where specified IPs are your bond1 10gE IP addresses]
default via 10.141.132.1 dev bond1 table T1

# rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
@ #1      inr.ruhep
@ 1       T1

Server Hangs and Kernel Stack Trace May Show Infiniband (IB) Messages: "destroy_qp_common+0x246/0x260 [mlx4_ib]"

Symptoms: After applying PS5 to Exalytics bare metal 'X series' (X2-4, X3-4, X4-4) machine, server may become unresponsive and appear to hang. Stack trace from the kernel may show messages like:

Mar 21 13:32:52 bdl2 kernel: [<ffffffff812630e1>] list_del+0x11/0x40
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa0371916>] mlx4_bf_free+0xe6/0x100
[mlx4_core]
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa03eeee6>]
destroy_qp_common+0x246/0x260 [mlx4_ib]
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa03eef3a>]
mlx4_ib_destroy_qp+0x3a/0x90 [mlx4_ib]
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa033a104>] ib_destroy_qp+0x44/0x90
[ib_core]
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa05000cc>] sdp_destroy_qp+0x2c/0x60
[ib_sdp]
Mar 21 13:32:52 bdl2 kernel: [<ffffffffa0503978>]
sdp_destroy_work+0x28/0x100 [ib_sdp]
Mar 21 13:32:52 bdl2 kernel: [<ffffffff8108c289>]
process_one_work+0xf9/0x370
Mar 21 13:32:53 bdl2 kernel: [<ffffffffa0503950>] ?
sdp_dreq_wait_timeout_work+0x1e0/0x1e0 [ib_sdp]

Problem is likely to occur when using infiniband and when SDP loads are increased.

Affected Releases: Exalytics 1.0.0.5.0 bare metal servers

Cause: <Bug 20793573> - CEAL: PS5 BOX HUNG IF INFINIBAND SDP IS ENABLED was logged for this issue. This bug will be fixed in the Exalytics Release 2 patch (post PS5).

Solution: Follow steps in <Document 1994907.1>, Exalytics PS5 Bare Metal Server Hangs and Kernel Stack Trace May Show Infiniband (IB) Messages: "destroy_qp_common+0x246/0x260 [mlx4_ib]".

Workaround: Reboot the server.

ORACLE VIRTUAL MACHINE (OVM) ON EXALYTICS

OVM Server Crashed "vector 0x2 is not implemented" and "soft lockup" Errors Displayed on the Console

Symptoms: Exalytics OVM guests instances may became unexpectedly non-responsive. Users may be unable to login to applications and putty or SSH login at the OS level may also fail.
The server console may display messages like:

xen: vector 0x2 is not implemented
Bug: soft lockup - CPU#3 stuck for 22s! [mcelog: 2128]
Modules linked in: xen_pciback tun xen_blkbak xen_netback xen_gntdev xen_evtchm
lock sunrpc mlx4_en(U) bridge stp 11c bonding be2iscsi isci_boot_sysfs _iscsi_....

Affected Releases: Exalytics 1.0.0.1.0 through 1.0.0.5.0

Bug Number(s): Internal, unpublished <Bug 19638775> BUG: SOFT LOCKUP - RIP: XEN_HYPERCALL_SCHED_OP+0XA/0X20 and <Bug 20800586> - EXALYTICS: GUEST OPERATING SYSTEM HANGS WITH SOFT LOCKUP: ERROR ON PS 5 OVM.

Solution: Upgrade the Exalytics server to PS5, (if not already installed), and then upgrade the OVM server to "Oracle Exalytics Base Image 2.0.1.4.0 for Exalytics Oracle VM x86-64." Additional details may be found in <Note 1994681.1> Exalytics OVM Server Crashed and is Unresponsive: "vector 0x2 is not implemented" and "soft lockup" Errors Displayed on the Console.

References

<NOTE:1994681.1> - Exalytics OVM Server Crashed and is Unresponsive: "vector 0x2 is not implemented" and "soft lockup" Errors Displayed on the Console
<NOTE:1512928.1> - Oracle Exalytics IMAGEINFO Command Not Showing Expected Image Version Number After Applying Patchset 1 (PS1) Patch 14301728
<NOTE:1588277.1> - Oracle Exalytics Imageinfo Command Not Showing Expected Versions After Upgrading Exalytics Base Image To 1.0.0.5, Patchset 3 (PS3)
<NOTE:1990668.1> - Exalytics Patchset (PS) 5 Image Update Script Fails With Error: "Failed to install new rpms /opt/exalytics/update/ps3tops4.d/70install_exalytics-deps.sh failed; return code: 1"
<NOTE:1505616.2> - Information Center: Certification Information For Oracle Exalytics
<NOTE:1990982.1> - Exalytics Server Unexpectedly Shuts Down After Applying Patchset (PS) PS5 Firmware Update (Patch 18391569) - No BIOS Changes Made on Server
<NOTE:1910852.1> - Exalytics: Available and Recommended Patches and Patchsets (PS)
<NOTE:1904921.1> - Exalytics /var/log/messages Shows Smartd Error: "smartd[10687]: Device: /dev/sdd [SAT]...Offline uncorrectable sectors"
<NOTE:1994907.1> - Exalytics PS5 Bare Metal Server Hangs and Kernel Stack Trace May Show Infiniband (IB) Messages: "destroy_qp_common+0x246/0x260 [mlx4_ib]"

Attachments

This solution has no attachment