SuperCluster- July 2014 Quarterly Full Stack Download Patch (QFSDP) Companion Document

Asset ID:	1-79-1910505.1
Update Date:	2015-03-23
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1910505.1 : SuperCluster- July 2014 Quarterly Full Stack Download Patch (QFSDP) Companion Document

Applies to:

Oracle SuperCluster T5-8 Full Rack - Version All Versions and later
Oracle SuperCluster T5-8 Half Rack - Version All Versions and later
SPARC SuperCluster T4-4 Full Rack - Version All Versions and later
SPARC SuperCluster T4-4 Half Rack - Version All Versions and later
Oracle SuperCluster T5-8 Hardware - Version All Versions and later
Oracle Solaris on SPARC (64-bit)
This document is to detail known issues and workarounds for a specific QFSDP . This is a living document that will be updated as and we discover any issues. Do do not save as a hard copy please review prior to any patching assessment , patching plan build out and within 24 hours before a patching window.

Purpose

This document is intended to work in conjunction with SuperCluster - Patching Best Practices For The Quarterly Full Stack Download Patche (QFSDP) <Document 1569461.1> . It will give you a list of known issues and their current accepted workarounds. Some of them will be duplicates of items mentioned in the readme files of the QFSDP but are restated here because Oracle SuperCluster engineering felt them important enough to stress. This does not replace reading and understanding all of the README files provided.

Scope

This document is scoped to only account for known items for the July 2014 QFSDP.

This document is intended for any patching engineer, support engineer, patching Coordinator, patch planner or project manager in or Outside of Oracle involved in the planning and execution of QFSDP application.

This Document assumes you have read and are familiar with the contents of the QFSDP readmes and of the following articles:

SuperCluster Patching Best Practices For The Quarterly Full Stack Download Patch (QFSDP) <Document 1569461.1>

SuperCluster Critical Issues <Document 1452277.1>

Note: Of the next three notes you only need to use the one that is for your hardware type.

Oracle SuperCluster Supported Software Versions - All Hardware Types <Document 1567979.1>

Details

Goal

To provide a list of known issues for each QFSDP that we have discovered before it is released and that we find out after it is released. This one is specific to the July 2014 QFSDP.

Solution

Readme's can also be obtained without downloading the entire patchset by using the attachments at the bottom of the following documents

Patch Bundle	Readme MOS Note
July 2014	Supercluster QFSDP July 2014 SCMU Readme <Document 1910077.1>
April 2014	Supercluster QFSDP April 2014 SCMU Readme <Document 1682460.1>

Prescribed patching order.

The following installation order of the QFSDP components must be followed:
        1.CX-2 HCA InfiniBand Card Firmware Update (T4 SuperClusters only)
        2. Database Server Update
        3. Exadata Storage Server Update
        4. NM2 Infiniband Switch Update
        5. SuperCluster Maintenance Update
        6. All Other Components

Engineering designated "pay very close attention to" sections.

"Installation Procedure Phase 1" section ot the following README files:

./Infrastructure/SuperClusterMaintenanceUpdate/SCMU_2014.04/README.T5-8_scmu

./Infrastructure/SuperClusterMaintenanceUpdate/SCMU_2014.04/README.T4-4_scmu

Know Issues with the April 2014 QFSDP

Infiniband switch patching

The information on this is contained in README ./Infrastructure/ExadataStorageServer/11.2.3.3.0/README.html. However, it is not very helpful it simply directs you to the Exadata Owners Guide under chapter "Maintaining Oracle Exadata Racks" section "Upgrading InfiniBand Switch Firmware"

You can find the documentation by reviewing How do I find the SuperCluster T4-4, T5-8 and M6-32 documentation, such as Owner and User Guide? <Document 1605591.1>

Known issue: the documentation states that you can use patchmgr to upgrade and down grade the switches. However, on Oracle SuperCluster this is not true. The patchmgr utility does not know about the 2.0.6. If you need to downgrade a switch for any reason lease open a service request to Oracle Support for guidance. Running the patchmgr as is will down revision your switches to aversion not supported on Oracle SuperCluster and could damage your infiniband partitioning. This is documented in <Bug 18329680> - supercluster patchmgr does not allow for downgrade ib switches to 2.0.6

Known issue : The documentation states to use a lst file if only doing a sunset of switches. On Oracle SuperCluster we have found it much easier to use this file even when doing them all.

Known issue: This issue impacts both infiniband switch patching and Exadata storage cell patching. Do not run patchmgr from a /net/… or nfsv4 mount because it can lead to a zero permissions dostep.sh file and errors if the share ACLS are not accurate

--Depending on your naming convention you can do

# cat /etc/hosts |grep "sw-ib" > ibswitches.lst

# cat ibswitches.lst
10.141.177.113   orlt5sw-ib1.us.oracle.com          orlt5sw-ib1
10.141.177.114   orlt5sw-ib2.us.oracle.com          orlt5sw-ib2
10.141.177.115   orlt5sw-ib3.us.oracle.com          orlt5sw-ib3

--Verify you have all your expected switches if not change the grep patter for your rig.

--Verify that the spine switch is the first in the list. If it is not vi the file to make it the top one.

--Run from the ./Infrastructure/ExadataStorageServer/11.2.3.3.0/patch_11.2.3.3.0.131014.1 directory.

Note: The -upgrade -ibswitch_precheck may fail.

# ./patchmgr –ibswitches ibswitches.lst -upgrade -ibswitch_precheck

--Correct the known issues below and then run the upgrade

# ./patchmgr -ibswitches ibswitches.lst –upgrade

Known issue : The -upgrade -ibswitch_precheck may fail due to insufficient free memory and due to the hostname and FQDN being reversed in the /etc/hosts file on the switches.

Remediation:

Log into the switches and run:

--Drop pagecache, dentries and inodes to free up memory without rebooting. This saves you one switch reboot. We have tested this on many switches and it does not have an impact since the upgrade does the one reboot anyway.
#echo 3 > /proc/sys/vm/drop_caches

--vi /etc/hosts to swap the order of the hostname and FQDN if you get that error

Solaris operating system patching.

Known issue 1: ensure you shut down all monitoring agents the readmes explicitly call out the OpsCenter agents however we have seen other third party monitoring software to write to mount points before items like beadm mount phase of the patching operation and the mounting of zones at boot can get to them causing the operation to fail.

Known Issue 2: If you have an IDR applied that is not a standard SuperCluster IDR as determined during the assessment remove it.

# pkg info idr<number>

---Get whole command from above and edit for odd line breaks,if necessary. If you get an error saying certain packages are not in the Operating Sytem remove them from the list out of info and run the command again. Then proceede with the install_smu script.

Known Issue 3: If you had previously applied the workaround described in note 1617956.1, it will need to be reapplied as the file /usr/lib/fm/fmd/plugins/eft.conf will be overwriten while upgrading

Known Issue 4: SCRIT PLACEHOLDER

Feeedback

Provide feedback on the QFSDP process in the Oracle SuperCluster Quarterly Full Stack Download Patch (QFSDP) feedback discussion on the MOS community.

References

<NOTE:1452277.1> - SuperCluster Critical Issues
<NOTE:1569461.1> - SuperCluster - Patching Best Practices For The Quarterly Full Stack Download Patch
<NOTE:1682460.1> - SuperCluster - QFSDP April 2014 SCMU Readme
<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types

Attachments

This solution has no attachment