![]() | Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition | ||
|
|
![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Solution Type Predictive Self-Healing Sure Solution 1995236.1 : Exalogic Patch Set Update (PSU) Release 2.0.6.2.1 (Linux - Virtual) for April 2015
In this Document
Applies to:Oracle Exalogic Elastic Cloud X2-2 Hardware - Version X2 to X2 [Release X2]Oracle Exalogic Elastic Cloud Software - Version 2.0.6.0.0 to 2.0.6.2.0 Exalogic Elastic Cloud X3-2 Hardware - Version X3 to X3 [Release X3] Exalogic Elastic Cloud X4-2 Hardware Exalogic Elastic Cloud X5-2 Hardware Linux x86-64 Oracle Virtual Server(x86-64) PurposeOracle Exalogic is an integrated hardware and software system designed to provide a complete platform for a wide range of application types and widely varied workloads. It combines optimized Oracle Fusion Middleware software like WebLogic server, JRockit, Coherence with industry-standard Sun Server and Storage hardware and InfiniBand networking. The purpose of this document is to provide specific information around April 2015 Patch Set Update (PSU) for that system. ScopeThe target audience of this document are engineers and system administrators who plan to apply the Exalogic PSU. This document provides the following:
This document will be kept up-to-date with updates to errata and known issues. DetailsNote:
Patch Download
Released: April 2015 Product Version: 2.0.6.2.1 (on X2-2/X3-2/X4-2/X5-2) for Oracle Exalogic Elastic Cloud infrastructure PATCH 20383732 - EXALOGIC VIRTUAL 2.0.6.2.1 (on X2-2/X3-2/X4-2/X5-2) PATCH SET UPDATE (PSU) FOR APRIL 2015 Patch Readme DocumentationRefer to the readme documentation attached 20383732-Virtual.zip to this Document on how to upgrade the Exalogic infrastructure:
The readme content structure layout of the 20383732-Virtual.zip: 20383732-Virtual Known Issue with data corruption in ZFS Storage Appliance Software (2013.1.3.5) bundled in April 2015 PSUThere is a known issue with data corruption in the ZFS version (2013.1.3.5) bundled in the April 2015 PSU. Please refer to the following MOS Note for details: <Note 2006121.1>: Exalogic: Risk of Data Corruption in ZFS Storage Appliance Software (2013.1.3.5) Bundled in April 2015 PSU
All applications of April 2015 PSU MUST apply the updated ZFS version provided through patch 20983458 as a replacement to the version included in the PSU. Pre-requisitesO2CB Cluster TimeoutBefore patching, increase the cluster heartbeat timeout as mentioned in the Document ID : 1995593.1 - Increase O2CB Cluster Heartbeat Timeout on Exalogic Virtual PCIe Renumbering check on ZS3-ES storage heads in Exalogic x4-2 and x5-2 racksRefer to following Note which has information on this important pre-requisite check. <Note 2087741.1>: Patching ZS3-ES Renumbering Check On Exalogic x4-2 And x5-2 Racks
The pre-requisite step to update Weblogic Server configuration properties for non-default passwords prior to upgrading Exalogic Control Services failsRefer to "Appendix B: Patching Known Issues" section below for information on this pre-requisite step failure known issue. AppendicesAppendix A: Fixed Bugs ListPlease review Document ID : 1995246.1 - Exalogic Infrastructure April 2015 PSU - Fixed Bugs List. Appendix B: Patching Known IssuesDuring the April 2015 PSU patching, compute nodes will crash if the IB master subnet manager is not running on an NM2-GW switch on the Exalogic rackSymptoms Cause The problem is that when an IB master subnet manager's host IB switch is being upgraded it needs to reboot. Therefore, before the upgrade, the subnet manager daemon on the NM2-GW switch is disabled. The other IB components on the rack running the subnet manager daemon negotiate a new master subnet manager. If the new master subnet manager ends up on a device other than an NM2-GW switch on the Exalogic rack, compute nodes will lose their IB connectivity and then crash due to the fact that they cannot communicate with the cluster. Solution/Workaround The workaround is to follow the solution documented in following MOS Note. <Note 1682501.1>: Setting up the subnet manager in a multirack configuration containing Exalogic/BDA and Exadata/SSC/Expansion Rack.
Note that the configuration must be correct on ALL IB subnet managers on the IB fabric, and is not limited to the IB switches on the Exalogic rack. _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Exalogic control services upgrade failing for April 2015 PSU patchingSymptoms While attempting to patch Exalogic Virtual version 2.0.6.0.x with the April 2015 PSUs with the following command: /exalogic-lctools/bin/exapatch -a runExtension -p Exalogic_Control/emoc_patch_extension.py exapatch_descriptor.py ERROR: EMOC-OVMM-service 10.2.76.11 is not running the expected version: 3.2.8.733 [wldeploy] Caused by: javax.naming.AuthenticationException [Root exception is java.lang.SecurityException: User: weblogic, failed to be authenticated.] Cause WebLogic password was reset Solution/Workaround Review following MOS Note to fix this issue. <Note 1915756.1>: Exalogic Patching Failing Due To EMOC-OVMM Service Not Running the Expected Version After Changing the WebLogic User Password
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ After April 2015 PSU patching , when attempting to start guest vServers through the EMOC, they hang during the start-upSymptoms As part of the April 2015 PSU (2.0.6.2.0 ), OVM is upgraded from 3.2.1 to 3.2.8. When attempting to start any of the guest vServers from the EMOC, they are hanging during the start-up. Upon inspecting OVM's AdminServer.log file, it contained the following error message: ####<Aug 23, 2014 7:15:55 AM EST> <Error> <com.oracle.ovm.mgr.api.job.InternalJob> <XX01-elcontrol> <AdminServer> <Odof Tcp Client Thread: /127.0.0.1:54321/98> <<anonymous>> <> <0000KVz^_mCBh4SMyEvX6G1JxvB4000002> <1408742155355> <BEA-000000> <Job: AutoDiscoverTask_1408742154199, Time: 1408742154205, Internal Error (Operation) OVMAPI_6000E Internal Error: OVMAPI_4021E Server discover conflict at IP address: 192.168.23.4. The manager already has a server: XXX01cn04.XXX.XXX.XXX, at this IP address, with SMBIOS UUID: 08:00:20:ff:ff:ff:ff:ff:ff:ff:18:9e:c0:28:21:00.
Cause This is a known OVM defect: UNPUBLISHED Bug 16221689 - DISCOVER ISSUE DUE TO SERVER SMBIOS UUID CHANGING AFTER UPGRADE
When seen "fakeuuid=os:bi:sm:# : i:mp:le:me:nt:at:io:ns: n:ew:er: t" in the Oracle VM Server (dom0) /etc/ovs-agent/agent.ini. Solution/Workaround Please review the workaround documented in following MOS Note. <Note 1531611.1>: Errors after Oracle VM Server upgrade "IllegalOperationException: OVMAPI_6000E Internal Error: If the IP address of this server has changed.
____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Upgrading NM2 Switch Firmware on full rack Exalogic systems or multi-rack Exa systems causing Network outage
NOTE: The issue does not occur when Exalogic racks (one eighth, quarter, half ) are connected to Exadata racks. Symptoms Cause The NM2 Port ID space has increased four times from firmware version 2.0 to 2.1. Therefore, port ID collisions may occur if NM2 switches with firmware version 2.0 and 2.1 are run on the same fabric. This issue can cause network outages on the InfiniBand fiber. Solution/Workaround In order to prevent this issue, manipulation of the switch 'GWInstance' value may be required to ensure unique port IDs are generated by performing the following steps: # showgwconfig
Note the GWInstance "Running Value" on each of the switches. 1. Ensure that all the GWInstance values are even numbers [root@nm2gw-ib01 ~]# setgwinstance 16
Stopping Bridge Manager.. [ OK ] Starting Bridge Manager. [ OK ] root@nm2gw-ib01 ~]# For example, if the GWInstance values of four NM2 switches are 10, 20, 30 & 40, it will be necessary to change the GWInstance value of 10 to 16. Patching Exalogic Control Services fails due to incorrect version of PC/EC vservers in post-patch checksSymptoms Patching Exalogic Control Services fails due to incorrect version of PC/EC vservers in post-patch checks Cause Caused by the presence of backup files of certain property files, due to which the patching script skips the code to upgrade version in those property files. Solution/Workaround The workaround is to check for the presence of the following backup files and if present, rename them to something else and reapply the patch. /opt/sun/n1gc/lib/XVM_AGENT.properties.<old_version>
/opt/sun/n1gc/lib/XVM_PROXY.properties.<old_version> /opt/sun/n1gc/lib/XVM_SATELLITE.properties.<old_version> (present only in EC vserver, not in PCs) /opt/sun/n1gc/lib/XVM_SERVER.properties.<old_version> /opt/sun/n1gc/lib/XVM.properties.<old_version> /n1gc-setup/.version.properties.<old_version> where <old_version> is the version of EMOC before applying the patch . For e.g. - 12.1.4.2500 EMOC jobs are taking a long time to completeRefer to <Note 2016700.1> for information on this known issue. Failure to stop OSWatcher (oswbb) service while applying April 2015 PSURefer to <Note 2006110.1> for information on this known issue. The pre-requisite step to update Weblogic Server configuration properties for non-default passwords prior to upgrading Exalogic Control Services failsSymptoms A pre-requisite step is included in section entitled "1.1 Prerequisites" to configure Weblogic Server prior to upgrading Exalogic Control Services to handle non-default passwords configured for the weblogic user. Running the command as documented results in an error as shown below: [oracle@elctrl ~]$ /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh weblogic <non-default-password>
-bash: /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh: Permission denied Invoking it using sh as follows runs into ClassNotFoundException, as shown in the sample below: [oracle@elctrl ~]$ sh /u01/app/oracle/ovm-manager-3/weblogic/wlconfig.sh weblogic welcome1
Exception in thread "main" java.lang.NoClassDefFoundError: weblogic.WLST at gnu.java.lang.MainThread.run(libgcj.so.7rh) Caused by: java.lang.ClassNotFoundException: weblogic.WLST not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}} at java.net.URLClassLoader.findClass(libgcj.so.7rh) at java.lang.ClassLoader.loadClass(libgcj.so.7rh) at java.lang.ClassLoader.loadClass(libgcj.so.7rh) at gnu.java.lang.MainThread.run(libgcj.so.7rh) [oracle@elctrl ~] Cause A step to set the WLS environment variables is missing in the documentation. Also, the wlconfig.sh script is not executable; it needs to be invoked using "sh". Solution/Workaround
Running exapatch with force option (-f) after applying OSWatcher patch exits indicating that it is already at version 2.0.6.2.1Symptoms After applying patch 20949658 to resolve the issue with OSWatcher and re-running exapatch with the force option (-f), the following error is encountered: Solution/Workaround Refer to <Note 2006110.1> which has been recently updated with information on this known issue. Guest vServer update fails during yum update of rpms with 'up2dateErrors'Refer to Exalogic PSU Known Issues <Note 1571367.1>, section "Guest vServer update fails during yum update of rpms with 'up2dateErrors'" for information on this known issue.______________________________________________________________________________________________________________________________________________________________________________________________________________________________________ PSU Fails to Patch Compute Nodes Due to a Conflict with a file from package vmpinfo-sosreport-1.0.3-1.noarchRefer to for <Note 2078002.1> details. ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Exapatch PrePatchCheck On Guest vServer Failing With "Unable to identify applicable pre/post patch checks" Error During April 2015 PSU (2.0.6.2.1) UpgradeRefer to for <Note 2086832.1> details. ______________________________________________________________________________________________________________________________________________________________________ Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range"Refer to <Note 2092033.1> for details on this known issue. ______________________________________________________________________________________________________________________________________________________________________ Exalogic Guest vServer Patching To April 2015 PSU Failing With Error: "python-devel-2.4.3-56.el5.i386: failure: python-devel-2.4.3-56.el5.i386.rpm from eecs_20621_updates: [Errno 256] No more mirrors to try."Refer to <Note 2100724.1> for details on this known issue. ______________________________________________________________________________________________________________________________________________________________________ Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU UpgradeRefer to <Note 2114516.1> for details on this known issue. ______________________________________________________________________________________________________________________________________________________________________ Exalogic Guest vServers Start Without Networks After Upgrading To April 2015 PSU 2.0.6.2.1 With "WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/." Message On ConsoleRefer to <Note 2122009.1> for details on this known issue. Appendix C: Errata1. Migration of assets managed by Proxy ControllersIn the troubleshooting MOS note section "Problem: components are listed under both the ProxyControllers(PC)" Original Text In the Exalogic Control BUI, if it is observed that any of the component asset is listed under both the Proxy Controllers (PC), they need to be migrated to a single PC. Updated Text For a given switch (NM2-GW switches, NM2-36p switch), only one proxy controller needs to be managing it. If a particular switch appears to be managed by both PC1 and PC2, it must be migrated to one of the proxy controllers. It does not matter whether it is migrated to PC1 or PC2. One or more compute nodes may appear in the Managed Assets list of both proxy controllers. For instance, el01cn01.example.com may appear as being managed by both PC1 and PC2. This is expected behavior; no migration is required for the compute nodes. Other non-switch assets, such as ZFS storage heads and PDU may also appear in the Managed Assets list of both proxy controllers. Migration is not required for these assets prior to applying the April 2014 PSU; migration is required only for switches. Login to EMOC BUI and navigate to "Administration" item in the left panel and find the entries for 'PC1' and 'PC2' vServers. Select a Proxy Controller say, "PC1" in the left panel. In the center panel click on the "Managed Assets" tab and set the Asset Type Filter to "Network Switches" to get list of switches managed by the 'PC1' Proxy Controller. Select the switch that you wish to migrate to the other ProxyController "PC2". Click on the icon that provides the option to "Migrate Assets". A confirmation dialog shows up, select 'Migrate' button to proceed. Once the migration completes finishes, a notification pop-up appears at the bottom right corner of the EMOC BUI, confirming the successful migration. Appendix D: TroubleshootingExapatch prePatchCheck option fails with ERROR:FAILED - /opt/exalogic.toos/CheckHWnFWProfileProblem ExaPatch prePatchCheck option fails with ERROR: FAILED - /opt/exalogic.tools/tools/CheckHWnFWProfile
CheckHWnFWProfile may have failed due to valid upgraded firmware but /usr/lib/init-exalogic-node/supported_config file still references original firmware version. For e.g. Supported disk controller Version: 12.12.0-0079 Current disk controller Version : 12.12.0-0178 Disk controller is not at the supported firmware version. It requires firmware update... Supported disk controller Firmware at: /opt/exalogic/firmware/disk_controller/12_12_0_0079.rom Solution After making sure there are no other errors, ignore this precheck and continue patching. When patching the compute node, please use the -f flag with Exapatch command line option. For e.g., to override pre-patching checks while patching compute nodes, run the following command: [root@compute-node2]# /exalogic-lctools/bin/exapatch -a patch cn -h <compute_node_ip> --force
Appendix D: TroubleshootingInformation About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic PatchingRefer to <Note 1997466.1> for details Guest vServer patching FAQ'sRefer to <Note 2031749.1> for details.
References<NOTE:2027098.1> - Exalogic Patch Set Update (PSU) Release 2.0.6.2.2 (Linux - Virtual) for July 2015<NOTE:2122009.1> - Exalogic Guest vServers Start Without Networks After Upgrading To April 2015 PSU 2.0.6.2.1 With "WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/." Message On Console <NOTE:1449226.1> - Exachk Health-Check Tool for Exalogic <NOTE:2092033.1> - Exalogic Compute Node or Guest vServer PSU Upgrade Using Exapatch Fails With Error "IndexError: list index out of range" <NOTE:2016700.1> - Exalogic: vServer Creation Takes a Long Time or Fails After a Few Days of Uptime <NOTE:2100724.1> - Exalogic Guest vServer Patching To April 2015 PSU Failing With Error: "python-devel-2.4.3-56.el5.i386: failure: python-devel-2.4.3-56.el5.i386.rpm from eecs_20621_updates: [Errno 256] No more mirrors to try." <NOTE:2114516.1> - Exalogic X2-2 Compute Nodes ILOM Network Settings Lost After PSU Upgrade <NOTE:2078002.1> - PSU Fails to Patch Compute Nodes Due to a Conflict with a file from package vmpinfo-sosreport-1.0.3-1.noarch <NOTE:1329262.1> - How to Perform a Healthcheck on Exalogic <NOTE:2006110.1> - Exalogic: Failure to stop OSWatcher (oswbb) service while applying April 2015 PSU <NOTE:2006121.1> - Exalogic: Risk of Data Corruption in ZFS Storage Appliance Software (2013.1.3.5) Bundled in April 2015 PSU <NOTE:1314535.1> - Exalogic Patch Set Updates (PSU) Master Note <NOTE:1571367.1> - Exalogic Infrastructure PSU Upgrade - Known Issues <NOTE:1997466.1> - Information About Usage Of Exapatch Force ( -f or --force ) Option Used During Exalogic Patching <NOTE:2031749.1> - FAQs Regarding vServer Patching Part Of Exalogic PSU Upgrade Attachments This solution has no attachment |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|