How To Back Up the Oracle SuperCluster Configuration with osc-config-backup

Asset ID:	1-79-1934129.1
Update Date:	2018-05-30
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 1934129.1 : How To Back Up the Oracle SuperCluster Configuration with osc-config-backup

Applies to:

Oracle SuperCluster T5-8 Full Rack - Version All Versions to All Versions [Release All Releases]
Solaris Operating System - Version 10 1/13 U11 to 11.3 [Release 10.0 to 11.0]
Oracle SuperCluster T5-8 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M7 Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)

Purpose

This article describes how to back up Oracle SuperCluster configuration data using the Oracle SuperCluster osc-config-backup tool. To recover SuperCluster configuration from osc-config-backup data, refer to the article called How To Recover SuperCluster Components that were Backed Up with osc-config-backup (Doc ID 1934130.1)

Scope

Detailed instructions and examples that you can use to back up SuperCluster components using osc-config-backup can found in the Oracle SuperCluster Configuration Backup Utility - Backup Guide available in the Oracle SuperCluster M7 Series Documentation library. The initial release of osc-config-backup is available for M7 SuperClusters running with 2.2 software onwards or from MOS via Patch 24415676.

Details

Overview

The osc-config-backup tool simplifies the process of backing up SuperCluster component configurations. The backup data is stored on the internal ZFS storage appliance.

Common use cases

Obtain an initial backup snapshot of the components immediately following the SuperCluster deployment.
Back up components prior to and after performing a Quarterly FullStack Download Patch (QFSDP) upgrade.
Save the component configurations prior to unforeseen hardware or software failures, thereby providing a means of recovery.
Components Covered by osc-config-backup:

Domains – Configuration, ZFS pools (rpool, bpool if present, and u01-pool for Database Domains)
Solaris zones – Configuration, ZFS pools (rpool and u01-pool for Database Zones)
IB switches – Configuration
Ethernet management switch – Configuration
ZFS storage appliance – Configuration
SuperCluster – Configuration information (ssc-cu.zip file)
Explorer – Data from each Database and Application Domain

Note - Data located outside the rpool and u01-pool zpools in zones and domains is not backed up by osc-config-backup.

Supported Systems

osc-config-backup is supported on these SuperCluster models as long as the system also meets the prerequisites:

Oracle SuperCluster T5-8
Oracle SuperCluster M6-32 (starting Jan 2017 QFSDP)
Oracle SuperCluster M7
Oracle SuperCluster M8

Prerequisites

Oracle SuperCluster must be running the July 2016 QFSDP update or later
Oracle SuperCluster running with 2.2 software or later.

Best Practices

Version information can be found by running osc-config-backup.pl -v

If you are running v1.1 or v1.1.1 please ensure you upgrade to v1.1.2 as detailed on SuperCluster Critical Issues (Doc ID 1452277.1)

V1.0.x
Oracle SuperCluster T5-8

The default steps provided offer a high-level of backup security and a performant restore for domains by providing an additional backup of domains using a LUN mirror concept.

Steps 7-11 can be omitted to greatly reduce the time for the backup, space consumption, and the time taken for the internal ZFS storage appliance to failover should the need occur.

run osc-config-backup.pl -s 12 -f to skip steps 7-11 after step 6 is complete.

The domains will continue to be backed-up and stored in a ZFS stream format during the "Backup Domains and Zones datasets" step.

Oracle SuperCluster

To perform additional Oracle SuperCluster configuration backups you may like to rotate the log/audit trail and configuration file.

v1.0.x
# mv /var/log/osc-config-backup-data /var/log/osc-config-backup-data.`/usr/bin/date '+%Y.%m.%d.%H.%M'`
# mv /opt/oracle.supercluster/osc-config-backup/conf/osc-config-backup.conf /opt/oracle.supercluster/osc-config-backup/conf/osc-config-backup.conf.`/usr/bin/date '+%Y.%m.%d.%H.%M'`

v1.1.x
osccfbck@host: mv /var/log/osc-config-backup-data $HOME/osc-config-backup-data.'/usr/bin/date '+%Y.%m.%d.%H.%M''
osccfbck@host: mv /opt/oracle.supercluster/osc-config-backup/conf/osc-config-backup.conf $HOME/osc-config-backup.conf.'/usr/bin/date '+%Y.%m.%d.%H.%M''

Note: If you have moved the files away as root and are running or upgraded to v1.1.x, please ensure that the /var/log/osc-config-backup-data directory is in place and owned by user/group osccfbck
# mkdir /var/log/osc-config-backup-data
# chown osccfbck /var/log/osc-config-backup-data
# chgrp osccfbck /var/log/osc-config-backup-data

Known Issues

SuperCluster platform 2.5.13 April 2018 QFSDP

Note: This applies only in the case of an upgrade and NOT for a fresh install

With the fix in SuperCluster platform 2.5.12 of
Bug 26847416 pkg verify - fails for osc-config-backup

and now in SuperCluster platform 2.5.13 of the fix for
Bug 27844160 osc-config-backup should use a different UID/GID

we are introducing a new UID (61012) for the osccfbck user and a new GID (61012) for the osccfbck group.
This means that the following additional steps are needed after upgrading from any release prior to SuperCluster platform 2.5.13

All following commands must be run after login in as root:

1. On all control nodes where the osc-config-backup tool is installed, run:

# chown -R osccfbck:osccfbck /export/home/osccfbck

2. Run once, on every node listed in the variable BKP_NODES in /opt/oracle.supercluster/osc-config-backup/conf/osc-config-backup.conf:

# groupmod -o -g 61012 osccfbck
# usermod -g 61012 oscbackR
# chown -R oscbackR:osccfbck /export/home/oscbackR
# chown root:bin /export/home/oscbackR/scripts/prtfmt
# chown -R oscbackR:osccfbck /var/log/osc-config-backup-data

Note: Ignore any errors that may occur. You may skip this step if the oscbackR account does not exist on that node.

3. Change file ownership on the Storage Appliance:
- Connect to the BUI of the active ZFSSA head as root
- Go to Maintenance -> Workflows
- Upload the attached chown_workflow.akwf file by clicking on the "+" icon at the top of the table
- Run the newly added workflow named "Change osc-config-backup file ownership" by clicking on the appropriate icon.
The workflow takes no parameters so you can dismiss the warning that appears upon execution.

4. On the master control domain, you might need to re-apply the workaround described below in Known Issues v1.1.x step 1

# chgrp osccfbck /var/tmp/ssc-data/config/*

This issue will be fixed in the SuperCluster platform 2.6 release

SuperCluster platform 2.5.13 April 2018 QFSDP

Bug 27828005 - step 6 throws invalid command message on console
The "aksh: invalid command" error messages printed on the console during step 6 can be safely ignored.
They are purely cosmetic and incur no loss of functionality

SuperCluster platform 2.4.13 + July '17 QFSDP

v1.1.3 ; exa-family 2.4.0.1219

If you are installing for the first time you may see an error regarding the osccfbck user.
Please download to the primary domain of node 1 and as root run the preinstall script here, and retry the package installation.

v1.1.x Step 1	With the new security model we no longer access files as 'root', to further automate the discovery of your system for backup you may like to ensure that the /var/tmp/ssc-data/config/ files are readable to the osccfbck user. chgrp osccfbck /var/tmp/ssc-data/config/* chmod g+r /var/tmp/ssc-data/config/* Alternatively you may populate the configuration file manually as described in the process.
Step 2	For convenience osc-config-backup is able to utilise the SuperCluster Virtual Assistant to describe the necessary SuperCluster components to back-up. If the SuperCluster Virtual Assistant service is unavailable osc-config-backup is able to leverage the initial installation data files. In this circumstance during the execution of Step 2 you may see: Error: cannot get ZFS-SA heads when your configuration file is missing the ZFS-SA head details: /opt/oracle.supercluster/osc-config-backup/conf/osc-config-backup.conf export BKP_ZFSHEAD1=; export BKP_ZFSHEAD2=; Resolution: You should check the configuration file above for correctness and populate the two ZFSSA head hostnames manually in the configuration file, and re-run step 2.
Step 2	If you are using telnet for connection to the management switch (opposed to SSH) you may see an error. If you verify that 'admin' user can connect then please ignore the error and Step 6 will verify your connection.
Prior to v1.1.3 Step 2	If you receive a password error for the Service Processors (SP) and you confirm the password is correct, you may like to increase the authentication timeout as detailed below. cd /opt/oracle.supercluster/osc-config-backup/scripts cp validatePass.exp validatePass.exp.bak gsed -i 's/timeout 20/timeout 40/g' validatePass.exp
v1.1/v1.1.1 Step 2	Before Step 2 - please run cd /opt/oracle.supercluster/osc-config-backup/scripts cp setup_ssh_equiv.sh setup_ssh_equiv.sh.bak gsed -i 's/500/700/g' setup_ssh_equiv.sh md5sum setup_ssh_equiv.sh MD5 sums should be: v1.1: 35a213e875c640f6490c90111395cd25 V1.1.1: 57b72a972fc01d77998c0db58e6302d0
Prior to v1.1.4 Step 9	Depending on your configuration the ZFSSA backup may take longer than 30seconds and you may receive an error. When logging into the ZFSSA head(s) the prompt then may look like :maintenance system conf_backup step0> You can cancel the backup with "abort" to return the state back to normal. In that case increase the timeout time as below: cd /opt/oracle.supercluster/osc-config-backup/scripts cp sshAKSH.exp sshAKSH.exp.bak gsed -i 's/30/90/g' sshAKSH.exp and rerun the step.
v1.0.x Step 10	During step 10 it's possible to encounter Bug 19308779 - Unable to split rpool: pool already exists or Unable to split rpool: log device has unplayed intent logs - This situation arises due to the ZFS intent log on the original BE not being replayed on boot due to the fact that we've booted from our ABE. In the unlikely event this issue is encountered, then the previous BE just needs to be mounted and umounted ie beadm mount BE /mnt; beadm umount BE
v1.0.x Step 10	Once step 10 has completed and the zpool split has run Bug 20052910 - resolve unsolicited hotplug hostage situation via force detach is likely to be encounted - The risk of encountering this issue is high if the osc-backup iscsi lun is removed from the ZFS-SA. Until this bug is addressed in a future QFSDP, then the only workaround is to reboot the domain(s) affected.
Step 11	At this time, osc-config-backup is unable to run the Oracle Explorer Data Collector on Solaris 10 nodes
	The final step "Cleanup SSH equivalence" may fail with "removeSSHPubKey.aksh is missing" Please update the filename below and rerun the step: mv /opt/oracle.supercluster/osc-config-backup/scripts/removeSSHPubKey.akshr /opt/oracle.supercluster/osc-config-backup/scripts/removeSSHPubKey.aksh

References

<NOTE:1567979.1> - Oracle SuperCluster Supported Software Versions - All Hardware Types

Attachments

This solution has no attachment