Oracle Database Quarterly Full Stack Download Patch for SuperCluster (Jul 2017) Known Issues

1.1 SuperCluster Pre-Installation Notes

The following are general SuperCluster QFSDP pre-installation notes:

1.1.1 - Installation of the Quarterly Full Stack Download Patch for Oracle SuperCluster (Jan 2016) - patch 22336131 is a prerequisite to installing this patch.

1.1.2 - Pay particular attention to the installation order prescribed in sub-section 4 ”Required Installation Order for QFSDP Components”, which is part of ”Section 2. Installing Quarterly Full Stack (Jul 2017 - 11.2 and 12.1)”.

1.1.3 - Use the Checklists contained in this QFSDP to assist users in understanding the installation process and for tracking progress during the update. See QFSDP Checklistdocuments.

1.1.4 - Oracle recommends running Exachk before and after performing planned maintenance to review and cross reference collected data against supported version levels and recommended Oracle Exadata best practices. See My Oracle Support <Note 1070954.1> "Oracle Exadata Database Machine exachk or HealthCheck".

1.1.5 - Beginning with the Oct 2016 QFSDP, management of Solaris 11 IDRs is simplified by replacing IDRs with a supercluster-solaris Custom Incorporation (CI). This is a superset of the Solaris 'entire' incorporation, and includes the payloads from the recommended SuperCluster IDRs.

Additional critical fixes may be released after the QFSDP is released. Please periodically check MOS Note 2086278.1, "SuperCluster Recommended Custom Incorporations, IDRs, and CVEs Addressed" for the latest recommended Custom Incorporation for Solaris 11 SRU16.3 and install any later Custom Incorporation specified (if any) with this QFSDP on all Solaris 11 instances.

Starting with the Jan 2017 QFSDP, a later Custom Incorporation can be downloaded and added to the SuperCluster's Solaris IPS repository and used by the QFSDP install_smu script in place of the Custom Incorporation included in the QFSDP itself. That saves having to first install the QFSDP and then update to a later recommended Custom Incorporation afterwards.

Please refer to the SCMU README.solaris file for instructions on how to add a downloaded Custom Incorporation version to the Solaris IPS repository.

1.1.6 - Check the README.solaris `Change Manifest and History` for any changes impacting install_smu behavior.

1.1.7 - Where the instructions in the QFSDP READMEs contradict instructions in the individual patch READMEs of included components, the QFSDP READMEs have precedence in a SuperCluster context.

1.1.8 - Starting with the April 2017 QFSDP, two NM2 IB switch firmware versions will be included with the QFSDP: IB FW 2.1.8-1 supporting Exadata cell versions 12.1.2.3.x only and IB FW 2.2.x-y for Exadata cell versions 12.2.1.1.x. IB FW 2.1.8-1 is part of the SuperClusterMaintenanceUpdate and should be applied following the instructions in the README files under SCMU_2017.07, IB FW 2.2.x-y is part of the ExadataStorageServer 12.2.1.1.x bundle and the Exadata procedures apply for updates to this FW version.

1.1.10 - For information on the use of the osc-config-backup utility, including Known Issues, please see MOS <Note 1934129.1>

Due to bug 26544468 osc-config-backup has been removed from the QFSDP install_smu process for the Jul 2017 QFSDP. For a workaround for bug 26544468 and instructions on how to manually update/install osc-config-backup, please see MOS<<Doc 1934129.1>>.

1.1.11 - If you are doing ZFSSA replication, or have shares off the ZFSSA to external hosts, using the management interface, plan on taking an outage for these services during QFSDP application. Before applying the ZFSSA patch take note of which storage head is servicing such operations and ensure it is the active head when your patching activities are over.

1.1.12 - ZFSSA Deferred Updates should be applied. This is particularly relevant for older SuperCluster T4, T5, M6 systems. See https://docs.oracle.com/cd/E56021_01/html/E55850/goxdn.html#scrolltoc. In particular, "Multiple Initiator Groups per LUN" and "Support for Large Block Sizes" are required for SuperCluster and must be applied. For other Deferred Updates, if the system is being upgraded from a version later than the Deferred Update, apply it as it's applicable to both the "from" and "to" versions and hence won't prevent rollback to the earlier version in the unlikely event that it is necessary. Where the Deferred Update is later than the version "from" which you are upgrading, you may wish to wait several weeks to ensure the version updated "to" is working well, before applying the Deferred Update in the unlikely case there's a need to revert to the version you upgraded "from".

1.1.13 - General SuperCluster documentation updates are now available from http://docs.oracle.com/en/engineered-systems/ and will no longer be included in the QFSDP.

1.1.14 - The Jul 2017 QFSDP introduces Exafamily 2.4. For further information, please refer to the SuperCluster I/O Domain Administartion Guide at https://docs.oracle.com/cd/E58626_01/html/E53172/index.html.

1.1.15 - Please review the issues detailed in the SuperCluster Critical Issues (MOS <Note 1452277.1>) prior to applying this QFSDP.

1.2 SuperCluster Known Issues for this release

The following are known issues for this QFSDP release:

1.2.1 - If the Oct 2016 QFSDP was ever installed on the system and the system configuration contains IO domains, please follow the steps outlined in MOS <Note 2223954.1> "Remediation steps for 25386898 for SuperClusters with the Oct 2016 QFSDP applied" before installing this QFSDP.

1.2.2 - Install picl in any Database non global zones using 'pkg install picl' (if not already installed) prior to running opatch to update the Database as opatch now uses jre-8 and installing picl improves performance.

1.2.3 - Solaris 11.3 SRU11 and later include LDoms 3.4.0.1 which by default sets verfied boot_policy=warn. Due to Bug 21463646, the warning messages such as the following may be seen in /var/adm/messages on boot:

These warnings can be ignored. Do not set verified boot_policy=fail. If desired, the warning messages can be suppressed as follows, but it is not necessary to do so:

On the control domain:
    ldm set-domain boot-policy=none <domain_name>
    save the SP config
Then reboot the guest domain.

1.2.4 - Solaris 11.3 SRU11 and later includes LDoms 3.4.0.1 which by default sets verified boot_policy=warn. On Solaris 10 guest domains, "Unsupported bootblk image" messages may be seen (Bug 24825049). These can be ignored.

If desired, the messages can be suppressed as follows, but it is not necessary to do so:

On the control domain:
    ldm set-domain boot-policy=none <S10_domain_name>
    save the SP config
Then reboot the guest domain.

1.2.5 - The following messages which may be seen on the console can be ignored (bug 24923073):

1.2.7 - Read MOS <Note 2043654.1> "Bug 21417505 - Rebooting of IB switch causing local zones going to shutting down state" and follow the steps indicated before applying this QFSDP.

This issue has the potential to cause database node evictions on domains or global zones with a large number of RDS connections when an IB switch is rebooted. There are a number of factors which impact the number of RDS connections, in particular Network Resource Management (that is, Exadata QOS / TOS), which is enabled in DB 12.1.0.2, Solaris 11.2 SRU1, and 12.1.2.x.x Exadata cells. These versions, or later, are included in this Jan2017 QFSDP.

The referenced My Oracle Support note provides information for users to determine their current number of RDS connections and prediction formulae to determine the likely number of RDS connections after applying this QFSDP. This should be done well in advance of the patching window.This Jul 2017 QFSDP will be suitable for most SuperCluster customers to install as is. If the prediction formulae indicate that applying this QFSDP would exceed the current safe threshold of RDS connections, then file an SR and wait for further advice.

1.2.8 - If the bug 23138854 "NON-ROLLING CELL UPGRADE 12.1.2.3.1 TIMEDOUT IN REACTIVATION GRIDDISKS" is encountered, workaround as follows:

For non-rolling upgrade, first make sure databases and ASM are shutdown.

To inactivate all griddisks on a cell using cellcli command :

For rolling upgrade, you need to check all cells with all griddisks having asmdeactivationoutcome=Yes with cellcli command :

1.2.9 - Some customers temporarily modify passwords during maintenance windows, for example to enable Platinum Patching, and set them back when patching is complete. If this is the case, it is recommended to temporarily disable monitoring of the Exadata storage cells in the Oracle Engineered System Hardware Monitoring OESHM) utility or update the passwords. This is necessary as otherwise OESHM attempts to login to the storage cells will fail and, by default, the cells will lock out for 10 minutes after repeated failed login attempts. To temporarily disable storage cell monitoring in OESHM, login to https://<IP of master root domain>:8001, go to the Storage Server tab, select each storage cell in turn, and click the "Stop Monitoring" button and confirm. The cell Hardware Health and Configuration status will change to "In Maintenance". After reverting to the original passwords at the end of the maintenance window, re-enable monitoring of the cells by repeating the process, clicking the "Start Monitoring" button.

1.2.10 - When running OEDA, bug 24945782 "STEP 11 : INITIALIZE CLUSTER SOFTWARE HANGING INTERMITTENTLY" may occasionally be encountered. The workaround is simply to run the step again.

1.2.11 - Customers may see SQLs failing with ORA-27626 (Exadadata error: 242) in 11gR2 database after upgrading the storage server to version 12.1.2.3.4 or higher. See MOS <Note 2250760.1> for details.

1.2.12 - Mitigation issues can occur with ZFSSA after updating ssctuner and osc-domcreate to the Jul 2017 QFSDP. For details and workaround, please see MOS <Note 2297850.1>

1.2.13 - IB switch FW version 2.2.6-2 has not been qualified for SuperCluster yet and is not supported as part of the July 2017 QFSDP.

1.2.14 - Due to bug 26678971 it might not be recommended to update IB switch 2.2.x-y FW at this point. Please refer to MOS <Note 2280595.1> "Infiniband switch running 2.2.x will not boot..." and <Note 2301054.1> "How to prevent an Infiniband Switch being rendered un-bootable..." for details and how to determine if a IB switch might be vulnerable to this issue.

1.2.15 - An Oracle Clusterware   one-off patch will be required to address bug 26512962 <Patch 26512962>. This will have to be applied to each Grid Infrastructure Home as part of the patching cycle.

1.2.16 - Please also review the issue detailed in <Note 2085653.1>: 12.1 : Datapatch Fails with ERROR "KUP-04004,KUP-04017,KUP-04118,KUP-04095,ORA-29913"," fatal: libjli.so or libpicl.so.1: open failed"