Sun ZFS Storage 7320
 
Oracle Exadata Storage Server Software
 
Oracle SuperCluster T5-8 Half Rack
 
Solaris Operating System
 
Solaris Cluster
 
SPARC SuperCluster T4-4 Half Rack
 
Oracle SuperCluster T5-8 Full Rack
 
SPARC SuperCluster T4-4 Full Rack
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 
_Old GCS Categories>ST>Server>Engineered Systems>SPARC SuperCluster>Install
 
_Old GCS Categories>ST>Server>Engineered Systems>SPARC SuperCluster>Patching
 

SuperCluster Patching Best Practices

In this Document

Purpose

Scope

Companion Note Table

Readme's can also be obtained without downloading the entire patchset by using the attachments at the bottom of the following documents

Details

Planning

Assessment

What type of data should I provide before engaging Oracle?

Work to be done no less than 1 week prior to the patching window.

Items to do 48 hours before the patching window

Items to do the day of patching

Start of patching window

Post patching obligations

Generic feedback

References

Applies to:

Purpose

This note is to cover any best practices associated with the complete project planning and implementation of Oracle SuperCluster Quarterly Full Stack Download Patch. This is generic to all QFSDP  releases starting January 2014. However there will be a table at the end of the document that links you to the specific companion note that has known issues for a specific QFSDP.

Scope

This note will cover items associated in the planning assment and delivery of the Oracle SuperCluster Quarterly Full Stack Download Patch.

 Companion Note Table

Readme's can also be obtained without downloading the entire patchset by using the attachments at the bottom of the following documents

Details

Planning

The Oracle SuperCluster Quarterly Full Stack Download Patch, hereinafter referred to as QFSDP, may contain components that require the shutdown and restart of various components of the engineered system. This includes, but is not limited to, the storage cells, zones, logical domains (LDoms) and entire computes nodes or physical domains (PDoms). Taking this into consideration if you have deployed any software that is not configured for high availability you will have to plan to fail those items to another location within or outside of the Oracle SuperCluster or schedule an outage of the same.

Oracle Support requests that you start planning for your patching at least six weeks prior to your desired patching window so you can ensure all required change management processes are in place and that you can get the assessment phase completed four weeks prior to the patching window . This will include filling out the attached Oracle SuperCluster Patching Assessment Worksheet attached to this document.

The reason for the four weeks is that any given system may have one off defect fixes applied that are not part of the standard Oracle SuperCluster software deployment. Your organization will need time to provide this information to Oracle Support so that we may determine if these fixes are in your target patching level and if not give us sufficient lead time for you to have the necessary additional fixes built and ready at least two to seven days before the patching window.

Assessment

If the QFSDP will be applied by Oracle Platinum Services or Oracle Advanced Customer Services they will assist you in completing the Oracle SuperCluster Patching Assessment Worksheet.  However it would help expedite items if you could complete the General Information tab, Host Information tab, and Current Version column of the Component Versions and Counts tab to the best of your ability. If you are patching on your own and targeting a yet to be released QFSDP, you will need to do the aforementioned and then open a Service Request to Oracle Support using your Super Cluster CSI and have a support engineer assist you in completing the assessment. Again, please give the four to six weeks lead time.

What type of data should I provide before engaging Oracle? 

Database patch level information – use this information to find out if your one off patches are included or require a patch to be delivered on the target patching level.

Compute Node/ Operating System information - use this information to find out if your IDRs are included or require an IDR to be delivered on the target patching level.  The Solaris 10 information will have to be a separate attachment as it is too verbose to fit in the assessment worksheet.

LDom information – use this on the primary LDom to determine the number of LDoms for each compute node and in turn the total for the Oracle SuperCluster

Zone Information – use this on each LDom from above to determine the zone information.

Infiniband HCA firmware version – use this information to determine if your HCA firmware is at the latest supported revision.

 Fault information – use this to determine if there are any reported hardware faults or critical software faults reported on an LDom.

Compute Node firmware version – use this information to determine if your firmware is at the latest supported version for your hardware type.

Exadata storage cell information – use this information to determine version information and to ensure that there is no defective hardware or other serious fault on the cells. 

Power Distribution Unit Information –use this to obtain version information. 

Infiniband Switch information –use this to obtain version information and to verify the integrity of the Infiniband partitioning. If the Infiniband partitioning is invalid you would probably know because portions of the Oracle SuperCluster software would not be functioning. Never enter a patching window with any defective infiniband partitioning.

 
ZFS Storage Appliance –use this information to determine if your storage heads are at the right AK software level. 

Once you have all the information detailed in this section you will use the Oracle SuperCluster Patching Assessment Worksheet and the README of the QFSDP you are going to apply to determine which and how many of your components are going to be patched. Then use the time calculations to determine estimated windows for patching.  

Gather the assessment worksheet and all other detailed data too large for it and provide it to the parties doing your patching so they can determine if there are defect fixes they will have to ask for on top of the target QFSDP. The request for these items should be done with two days of completing the assessment. The request for these items should explicitly state the target date of patching in them.

Work to be done no less than 1 week prior to the patching window. 

Log into the target SuperCluster and verify the health of the IPS repository via the document SuperCluster- Best Practices for local Image Packaging System (IPS) software package repository. <Document 1625719.1>. 

If the repository check above in this section shows that the ssc project is already present skip this code block. If not create it. Modify the IP address or hostname as required.  

If running database in zones, exavm,  check for connectivity in each zone to the Infiniband storage network in accordance with SuperCluster- Best Practices for creating 8503 infiniband storage network IPMP interface in Solaris 11 local zones (application or exavm) <Document 1630341.1>. 

Create a share on the integrated ZFS Storage Appliance, a mount point on the primary LDom of the first compute node and mount it with nfsv3. Modify for your IP address hostname and subnet where applicable.

Copy the QFSDP patch pieces, any additional Database one off patches and IDRs into this QFSDP directory and follow the instructions for merging the pieces and extracting. Once everything is extracted do a chmod to all the database bundle patch items, opatch archive, and additional database related patches, excluding the Exadata Storage cell patches to the Grid Infrastructure home user and group. If for any reason you could not create the storage network in the zones, nor the nfs mount point, do the same to/u01/patches on the primary LDom of all compute nodes. And then push the database bits into /u01/patches of all exavm database zones. 

Ensure sufficient disk space.  Substitute with the zfs pool that your /u01 mount point is on. For T5-8 and later installs this may  a mount point  in rpool.  

Check for free space in the rpool of each LDom and zone. There is no hard number but we typically like to see them <75% full. If they are over this speak to the systems administrators and have them clean up what they can. 

While you are on the system do another sanity check of the storage cell hardware to make sure nothing went bad since the initial assessment. Do the same for the operating system by checking fmadm. If you notice anything get a Service Request opened immediately to address the faulty hardware. Do not ever go into a patching window with faulty hardware.

Items to do 48 hours before the patching window

Do checks on statuses for any database one off patches or operating system  IDRs that you have not received.  If you cannot get a confirmed ETA of before the patching window escalate the Service Request to a severity 1 if it is not already and start evaluating the possibility of other patching windows. This will require close coordination with application teams and any business units served.

Do another sanity check of the storage cell hardware to make sure nothing went bad since the initial assessment. Do the same for the LDoms by checking fmadm. If you notice anything get a Service Request opened immediately to address the faulty hardware. Do not ever go into a patching window with faulty hardware.

Check for free space in the rpool of each LDom and zone. There is no hard number but we typically like to see them <75% full. If they are over this speak to the systems administrators and have them clean up what they can. 

If you do not have the additional required database one off patches nor operating system IDRs escalate again to Oracle Support and request a guarantee that you will have them by your patching window.

Items to do the day of patching

Obtain the additional required database one off patches and/or IDRs. If they are not available you will have to evaluate the risk of running without them and if it is too high you will have to cancel the patch window.

Do another sanity check of the storage cell hardware to make sure nothing went bad since the initial assessment. Do the same for the operating system by checking fmadm. If you notice anything get a Service Request opened immediately to address the faulty hardware. Work with the Service request owner and the assigned field engineer to determine how much, if any delay, this will put on your patch window.  Work with your project managers and application teams and business units so they know what the status is throughout the day. Do not ever go into a patching window with faulty hardware.

Refresh the SuperCluster Supported Versions note for your hardware type, The SuperCluster Critical Issues note and most importantly the SuperCluster QFSDP Companion note for your QFSDP for any updates. 

Start of patching window

Connect to the Exadata storage cell and compute nodes ILOMs and reset the SP.

Confirm with all application teams and database administrators that it is OK to patch the targeted nodes.  Patch in accordance with your action plan derived from the assessment, patching method (rolling or non-rolling), the QFSDP README files, and the patching.  In case if failure during the window raise a Service Request immediately  at a severity commiserate with business impact. Please provide any log or trace information at the time of filing to expedite resolution. 

Post patching obligations

Once you have completed patching and ensured all components are at target levels and are operational please go through the README files one more time and ensure all steps were complete. Receive the iso images into the local repository if you did not do it during the preparation work.  This would typically be Solaris SRU iso, exafamily iso, IDRs and Solaris Cluster iso. There may be more or less depending on what was patched in the particular QFSDP. Instructions are clearly defined in the README files. Ensure that ssctuner is functioning and reporting output of its work in the /var/log/messages file and ensure all database LMS, LGWR and VKTM processes are at scheduling class of FX-60 for all databases and ASM instances where applicable. Check to make sure that the entries in /etc/system are time stamped earlier that the latest reboot.

Unless you did the install from the NFS mount clean up all installation files transferred to /u01/patch.  No matter the method chosen make sure none of the rpools on any of the LDoms nor zones are at or over 85% capacity. If they are look for areas to clean up such as /var/tmp, $OARCLE_BASE/diag for those systems where /u01 is a volume under rpool, extraneous Boot Environments (BEs).  What is extraneous is up to each shops best practice. Ensure the integrated ZFS Storage Appliance is left with the same head active as when you entered the patching window. The reason for this is some systems actually have NFS mounts over the management network and the management network does not fail over with the head. Document all findings during the patching exercise and document them in the applicable service requests. Also provide feedback for the companion document or the overall assessment process.

Generic feedback

For general feedback about this note or the QFSDP companion notes or general positive and negative feedback about the Oracle SuperCluster patching process please let us know via the Oracle SuperCluster MOS Community QFSDP thread. Do not use this thread for technical troubleshooting please do all that via service requests. Also do not provide any personal or customer specific information that you do not want have seen by possibly all MOS users.

References

This solution has no attachment