Oracle SuperCluster Maintenance

Test on non-production first

Preserve SQL Execution Plans with SQL Plan Management

Develop and Test a Recovery Plan

Obtain Required Account Access

Generic requirements:

• If you must update software a later version to meet the patching requirements for the 11.2.0.4 upgrade, then install the most recent version indicated in the SuperCluster Supported Version note for your hardware type.
• Apply all overlay and additional patches for the installed Bundle Patch when required.  The list of required overlay and additional patches can be found in SuperCluster Supported Version note for your hardware type, Exadata Critical Issues <Document 1270094.1>, ans SuperCluster Critical Issues <Document 1452277.1>.
• Verify that one-off patches currently installed on top of  11.2.0.3 are fixed in 11.2.0.4. Review the list of fixes provided with 11.2.0.4. For a list of provided fixes on top of 11.2.0.4 review the README. 
  
  • If you are unable to determine if a one-off patch is still required on top of 11.2.0.4 then contact Oracle Support.
• Do not place the new ORACLE_HOME under /opt/oracle.
  • If this is done then see <Document 1281913.1> for additional steps required after software is installed.
• Data Guard only - If there is a physical standby database associated with the databases being upgraded, then the following must be true:
  • The standby database is running in real-time apply mode as determined by querying v$archive_dest_status and verifying recovery_mode='MANAGED REAL TIME APPLY' for the local archive destination on the standby database. If there is a delay or real-time apply is not enabled then see Data Guard Concepts and Administration guide on how to configure these setting and remove the delay

• Oracle Database 11g, Release 2 (11.2.0.4) Patch Set 3:
• "Oracle Database", typically the installation media for the database comes with two zip files.
• "Oracle Grid Infrastructure"
• "Deinstall tool"

See the README of <Patch 13390677> for the exact filenames you need for Solaris SPARC.
     

• When available: download the latest Bundle Patch for 11.2.0.4
  • Typically recommended to download the latest OPatch release when applying a Bundle Patch. See <Patch 6880880> - download p6880880_112000_SOLARIS64.zip (for 11.2)

• Patches to be applied on top of 11.2.0.4 
  • The latest 11.2.0.4 Bundle Patch. This note uses 11.2.0.4 BP1 (<patch 17628025>) as an example

Apply patches / updates where required before upgrading proceeds

Update OPatch in existing 11.2 Grid Home and existing 11.2 Database Homes on All Database Servers

Apply required patches / updates identified in the section  'Upgrade Prerequisites' 

 Use Exachck's upgrade module to check for additional upgrade best practices and last minute patch requirements. See the Exachk documentation via <Document 1070954.1> for more information. As of this authoring it will check DB/RAC concerns.

Early stage pre-upgrade check: analyze your databases to be upgraded with the Pre-Upgrade Information Tool

During the pre-upgrade steps, the pre-upgrade tool (preupgrd.sql) will warn to set the CLUSTER_DATABASE parameter to FALSE. Ignore this warning.

(root)# dcli -g ~/dbs_group -l root mkdir -p /u01/app/11.2.0.4/grid/
(root)# dcli -g ~/dbs_group -l root chown oracle /u01/app/11.2.0.4/grid
(root)# dcli -g ~/dbs_group -l root chgrp -R oinstall /u01/app/11.2.0.4

(oracle)$ unzip -q /u01/app/oracle/patchdepot/p13390677_112040_SOLARIS64_3of7.zip \
          -d /u01/app/oracle/patchdepot

(oracle)$ unset ORACLE_HOME ORACLE_BASE ORACLE_SID
(oracle)$ export DISPLAY=<your_xserver>:0
(oracle)$ cd /u01/app/oracle/patchdepot/grid
(oracle)$ ./runInstaller
Starting Oracle Universal Installer...

1. On Download Software Updates screen, select Skip software updates, and then click Next. 
   
2. On Select Installation Options screen, select Upgrade Oracle Grid Infrastructure or Oracle Automatic Storage Management, and then click Next.
3. On Select Product Languages screen, select languages, and then click Next.
4. On Grid Infrastructure Node Selection screen, verify all database nodes are shown and selected, and then click Next.
5. On Privileged Operating System Groups screen, verify group names and change if desired, and then click Next.
6. On Specify Installation Location screen, enter /u01/app/11.2.0.4/grid as the Software Location for the Grid Infrastructure home, and then click Next.
• The directory should already exist and it is recommended that the Grid Infrastructure home NOT be placed under /opt/oracle
7. On Summary screen, verify information presented about installation, and then click Install.
8. On Install Product screen, monitor installation progress.
9. On Execute Configuration scripts screen, do NOT execute the scripts yet but first perform the following steps in order:

Apply the latest Bundle Patchset  and the workaround for item DB_10 in SuperCluster Critical Issues

  

<Document 1452277.1> to the Grid Infrastructure Home using 'napply' before running rootupgrade.sh on all nodes in the cluster. Please note that is you do the workaround before the bundle patch you may have to redo it following the bundle patch.

 

In order to prevent against an extra stop and start of the Grid Infrastructure and Databases later on in the process, the latest 11.2.0.4 Bundle Patch needs to be applied to the Grid Infrastructure home on all nodes in the cluster at this stage. In this section 11.2.0.4 BP1 (<patch 17628025>) is used as an example.

The current phase the installation is in does not allow applying the Bundle Patch using the 'opatch auto' functionality. In this phase the patches contained in the Bundle Patch can only be applied as the GI home owner using the 'opatch napply' command for each patch component. 

Consult the Bundle Patch README (section 'Manual Steps for Apply/Rollback Patch') for further instructions on what patch components need to be applied for the Grid Infrastructure home using 'opatch napply' as this differs per Bundle Patch.

Note that at this stage README step 1 ("Stop the GI managed resources running from DB home") and step 2 ("Run the pre root script") as well as README steps 4,5,6,7 and 8 must NOT be executed. An example of what remains is applying 11.2.0.4 BP1 components using 'napply' on all nodes in the cluster as follows (this step assumes the latest Opatch was installed in the new GI home):

(oracle)$ export PATH=$PATH:/u01/app/11.2.0.4/grid/OPatch
(oracle)$ opatch napply -oh $ORACLE_HOME -local /u01/app/oracle/patchdepot/17628025/17628006
(oracle)$ opatch napply -oh $ORACLE_HOME -local /u01/app/oracle/patchdepot/17628025/17629416

NOTE: No other steps other than updating OPatch to the latest release and running 'opatch napply' as specified by the README should be done at this stage. Don't execute the commands rootcrs.pl -unlock and rootcrs.pl -patch for example. Also for Exadata 11.2.0.4 already has RDS linked-in by default.

NOTE: See <Document 1410202.1> for more information on how to apply a Grid Infrastructure Patch Before root script (root.sh or rootupgrade.sh) is executed. Note that <Document 1410202.1> talks about Patch Set Updates (PSU) while Exadata only has bundle patches (BP)

If available apply 11.2.0.4 Bundle Patch Overlay Patches to the Grid Infrastructure Home as specified in  SuperCluster Supported Versions for your hardware type

Review  SuperCluster Supported Versions for your hardware type  to identify and apply patches that must be installed on top of the Bundle Patch just installed.

Apply Customer-specific 11.2.0.4 One-Off Patches to the Grid Infrastructure Home

If there are one-offs that need to be applied to the environment and they are approved by Oracle Support, then apply them now.

Change SGA memory settings for ASM

As sysasm, adjust sga_max_size and sga_target to a value of 2G. The values will become active with a next start of the ASM instances.

SYS@+ASM1> alter system set sga_max_size = 2G scope=spfile sid='*';
SYS@+ASM1> alter system set sga_target = 2G scope=spfile sid='*';

Verify values for memory_target and memory_max_target

Values should be as follows:

SYS@+ASM1> col value format a30
SYS@+ASM1> col name format a30
SYS@+ASM1> col sid format a5
SYS@+ASM1> select sid, name, value from v$spparameter
where name in
('memory_target','memory_max_target');

SID    NAME                      VALUE
------ ------------------------- -----------------------------------
*      memory_target             0
*      memory_max_target

When the values are not as expected, change them as follows:

SYS@+ASM1> alter system set memory_target=0 sid='*' scope=spfile;
SYS@+ASM1> alter system set memory_max_target=0 sid='*' scope=spfile /* required workaround */;
SYS@+ASM1> alter system reset memory_max_target sid='*' scope=spfile;

 Checks to do before executing rootupgrade.sh on Each Database Server

Before running rootupgrade.sh verify no active rebalance is running

Query gv$asm_operation to verify no active rebalance is running. A rebalance is running when the result of the following query is not equal to zero :

SYS@+ASM1> select count(*) from gv$asm_operation;

COUNT(*)
----------
0

Verify the workaround for item DB_10 in SuperCluster Critical Issues <Document 1452277.1>

 Before running rootupgrade.sh please review     SuperCluster - How to recover from a failed 11.2.0.3 to 11.2.0.4 upgrade due to HAIP (Doc ID 1996379.1) and make sure that you have made the changes form the cause section before preceding.

Execute rootupgrade.sh on each database server, as indicated in the Execute Configuration scripts screen the script must be executed on the local node first. The rootupgrade.sh script shuts down the earlier release installation, updates configuration details, and starts the new Oracle Clusterware installation. 

When rootupgrade.sh fails it is recommended to check the following output first to get more details :

• output of rootupgrade.sh script itself
• ASM alert.log
• /u01/app/11.2.0.4/grid/cfgtoollogs/crsconfig/rootcrs_<node_name>.log

For rootupgrade.sh executions that fail, refer to <document 969254.1> - How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure (CRS) on Linux/Unix.

 

After rootupgrade.sh completes successfully on the local node, you can run the script in parallel on other nodes except for the last node. When the script has completed successfully on all the nodes except the last node, run the script on the last node.  Do not run rootupgrade.sh on the last node until the script has run successfully on all other nodes.

 

• First node rootupgrade.sh will complete with this output

Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

• Last node rootupgrade.sh will complete with this output

Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0

ASM upgrade has finished on last node.

 

NOTE: if the following error message is seen when running rootupgrade.sh, it can be ignored:

acfsutil registry: CLSU-00100: Operating System function: open64 failed with error data: 2
acfsutil registry: CLSU-00101: Operating System error message: No such file or directory
acfsutil registry: CLSU-00103: error location: OOF_1
acfsutil registry: CLSU-00104: additional error information: open64 (/dev/ofsctl)
acfsutil registry: ACFS-00502: Failed to communicate with the ACFS driver. Verify the ACFS driver has been loaded.

 

11. On Finish screen, click Close.

Perform an extra check on the status of the Grid infrastructure post upgrade by executing the following command from one of the compute nodes:

(root)# /u01/app/11.2.0.4/grid/bin/crsctl check cluster -all

The above command should show an online status for Cluster Ready Services, Cluster Synchronization Services and Event Manager on all nodes in the cluster, example as follows:

**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

When the cluster is not showing an online status for any of the components on any of the nodes, the issue needs to be researched before continuing. For troubleshooting see the MOS notes mentioned in the reference chapter.

• Install Database 11.2.0.4 software on the primary system first then the standby system.
• Install Database 11.2.0.4 software on the standby system first then the primary system.
• Install Database 11.2.0.4 software on both the primary and standby systems simultaneously.

(oracle)$ unzip -q /u01/app/oracle/patchdepot/p13390677_112040_SOLARIS64_1of7.zip -d /u01/app/oracle/patchdepot
(oracle)$ unzip -q /u01/app/oracle/patchdepot/p13390677_112040_SOLARIS64_2of7.zip -d /u01/app/oracle/patchdepot

(root)# dcli -g ~/dbs_group -l root mkdir -p /u01/app/oracle/product/11.2.0.4/dbhome_1
(root)# dcli -g ~/dbs_group -l root chown -R oracle:oinstall /u01/app/oracle/product/11.2.0.4
(oracle)$ unset ORACLE_HOME ORACLE_BASE ORACLE_SID
(oracle)$ export DISPLAY=<your_xserver>:0
(oracle)$ cd /u01/app/oracle/patchdepot/database
(oracle)$ ./runInstaller

1. On 'Configure Security Updates' screen, fill in required fields, and then click Next.
2. On 'Download Software Updates' screen, select 'Skip software updates', and then click Next.
3. On 'Select Installation Option' screen, select 'Install database software' only, and then click Next.
4. On 'Grid Installation Options' screen, select 'Oracle Real Application Clusters database installation', click 'Select All'. Verify all database servers are present in the list and are selected, and then click Next.
5. On 'Select Product Languages' screen, select languages, and then click Next.
6. On 'Select Database Edition', select 'Enterprise Edition', click 'Select Options to choose components to install', and then click Next.
7. On 'Specify Installation Location', enter '/u01/app/oracle/product/11.2.0.4/dbhome_1' as the Software Location for the Database home, and then click Next.
• It is recommended that the Database home NOT be placed under /opt/oracle.
8. On 'Privileged Operating System Groups' screen, verify group names, and then click Next.
9. On 'Perform Prerequisite Checks' screen, resolve any failed check or warning.
10. On 'Summary screen', verify information presented about installation, and then click Install.
11. On 'Install Product' screen, monitor installation progress.
12. On 'Execute Configuration Scripts' screen, execute root.sh on each database server as instructed, and then click OK
13. On 'Finish' screen, click Close.

Install Latest 11.2.0.4 Bundle Patch for Oracle  Exadata Database Machine to the Database Home  - Do Not Perform Post-Installation Steps

Run Pre-Upgrade Information Tool

At this point the database is still running with 11.2.0.3 software. Connect to the database with your environment set to 11.2.0.3 and run the pre-upgrade information tool that is located in the 11.2.0.4 database home, as follows:

SYS@PRIM1> spool preupgrade_info.log
SYS@PRIM1> @/u01/app/oracle/product/11.2.0.4/dbhome_1/rdbms/admin/utlu112i.sql

During the pre-upgrade steps, the pre-upgrade tool (preupgrd.sql) will warn to set the CLUSTER_DATABASE parameter to FALSE. However when using DBUA this is done automatically so the warning can be ignored.

Handle obsolete and underscore parameters

Obsolete and underscore parameters will be identified by the pre-upgrade information tool.  During the upgrade, DBUA will remove the obsolete and underscore parameters from the primary database initialization parameter file.  Some underscore parameters that DBUA removes will be added back in later after DBUA completes the upgrade.

Data Guard only - DBUA will not affect parameters set on the standby, hence obsolete parameters and some underscore parameters must be removed manually if set. Typical values that need to be unset before starting the upgrade are as follows:

SYS@STBY1> alter system reset cell_partition_large_extents scope=spfile;
SYS@STBY1> alter system reset "_backup_ksfq_bufsz" scope=spfile;
SYS@STBY1> alter system reset "_backup_ksfq_bufcnt" scope=spfile;
SYS@STBY1> alter system reset "_lm_rcvr_hang_allow_time" scope=spfile;
SYS@STBY1> alter system reset "_kill_diagnostics_timeout" scope=spfile;
SYS@STBY1> alter system reset "_arch_comp_dbg_scan" scope=spfile;

Review pre-upgrade information tool output

Review the remaining output of the pre-upgrade information tool.  Take action on areas identified in the output.

• The standby database is running in real-time apply mode.
• The value of the LOG_ARCHIVE_DEST_n database initialization parameter on the primary database that corresponds to the standby database must contain the DB_UNIQUE_NAME attribute, and the value of that attribute must match the DB_UNIQUE_NAME of the standby database.

Flush all redo generated on the primary and disable the broker

To ensure all redo generated by the primary database running 11.2.0.3 is applied to the standby database running 11.2.0.3, all redo must be flushed from the primary to the standby.

First, verify the standby database is running recovery in real-time apply.  Run the following query connected to the standby database.  If this query returns no rows, then real-time apply is not running. Example as follows:

SYS@STBY1> select dest_name from v$archive_dest_status
  where recovery_mode = 'MANAGED REAL TIME APPLY';

DEST_NAME
------------------------------
LOG_ARCHIVE_DEST_2

Shutdown the primary database and restart just one instance in mount mode, as follows:

(oracle)$ srvctl stop database -d PRIM -o immediate
(oracle)$ srvctl start instance -d PRIM -n dm01db01 -o mount

Verify the primary database has specified db_unique_name of the standby database in the log_archive_dest_n parameter setting, as follows:

SYS@PRIM1> select value from v$parameter where name = 'log_archive_dest_2';

VALUE
-------------------------------------------------------------------------------
service="gih_stby" LGWR SYNC AFFIRM delay=0 optional compression=disable max_fa
ilure=0 max_connections=1 reopen=300 db_unique_name="STBY" net_timeout=30 valid
_for=(all_logfiles,primary_role)

SYS@PRIM1> alter system flush redo to 'STBY';

Shutdown the primary database.

Wait until the 'End-of-Redo' on the standby is confirmed, as follows:

End-Of-REDO archived log file has not been recovered
Incomplete recovery SCN:0:1371457 archive SCN:0:1391461
Physical Standby did not apply all the redo from the primary.
Tue May 22 13:03:51 2013
Media Recovery Log +RECO/prim/archivelog/2011_11_22/thread_2_seq_39.1090.767883831
Identified End-Of-Redo (move redo) for thread 2 sequence 39 at SCN 0x0.153b65
Resetting standby activation ID 338172592 (0x14281ab0)
Media Recovery Waiting for thread 2 sequence 40
Tue May 22 13:03:52 2013
Standby switchover readiness check: Checking whether recoveryapplied all redo..
Physical Standby applied all the redo from the primary.
Standby switchover readiness check: Checking whether recoveryapplied all redo..
Physical Standby applied all the redo from the primary.

Then shutdown the primary database, as follows:

(oracle)$ srvctl stop database -d PRIM -o immediate

Shutdown the standby database and restart it in an 11.2.0.4 database home

Perform the following steps on the standby database server:

Shutdown the standby database, as follows:

(oracle)$ srvctl stop database -d stby

Copy required files from 11.2.0.3 home to the 11.2.0.4 database home.
The following example shows the copying of the password file, but also other files like init.ora files may be required to copy:

(oracle)$ dcli -l oracle -g ~/dbs_group \
          'cp /u01/app/oracle/product/11.2.0/dbhome_1/dbs/orapwstby* \
          /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs'

Edit standby environment files

• Edit the standby database entry in /var/opt/oracle/oratab  to point to the new 11.2.0.4 home.
• On both the primary and standby database servers, ensure the tnsnames.ora entries are available to the database after it has been upgraded.  If using the default location for tnsnames.ora, $ORACLE_HOME/network/admin, then copy tnsnames.ora from the old home to the new home.
• If using Data Guard Broker to manage the configuration, then modify the broker required SID_LIST listener.ora entry on all nodes to point to the new ORACLE_HOME.  For example

SID_LIST_LISTENER =
(SID_LIST =
  (SID_DESC =
  (GLOBAL_DBNAME=PRIM_dgmgrl)
  (SID_NAME = PRIM1)
  (ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1)
  )
)

• After this reload the listener on all nodes, as follows:

(oracle)$ lsnrctl reload listener

Update the OCR configuration data for the standby database by running the 'srvctl upgrade' command from the new database home as follows.

(oracle)$ srvctl upgrade database -d stby -o /u01/app/oracle/product/11.2.0.4/dbhome_1

Start the standby, as follows (add "-o mount" option for databases running Active Data Guard):

(oracle)$ srvctl start database -d stby

Start all primary instances in restricted mode

DBUA requires all RAC instances to be running from the current database home before starting the upgrade. To prevent an application from accidentally connecting to the primary database and performing work causing the standby to fall behind, startup the primary database in restricted mode, as follows:

(oracle)$ srvctl start database -d PRIM -o restrict

Oracle recommends removing the value for the init.ora parameter 'listener_networks' before starting DBUA. The value will be restored after running DBUA. Be sure to obtain the original value before removing, as follows:

SYS@PRIM1> set lines 200
SYS@PRIM1> select name, value from v$parameter where name='listener_networks';

If the value for parameter listener_networks was set, then the value needs to be removed as follows:

SYS@PRIM1> alter system set listener_networks='' sid='*' scope=both;

Run DBUA from the new 11.2.0.4 ORACLE_HOME as follows:

(oracle)$ /u01/app/oracle/product/11.2.0.4/dbhome_1/bin/dbua

1. On 'Welcome screen', click Next.
2. On 'Select Database' screen, select the database to be upgraded, and then click Next.
• Enter a local instance name if requested.
3. On 'Upgrade Options' screen, select the desired options, and then click Next.
• If you have a standby database, then do NOT select to turn off archiving during the upgrade.
4. On 'Recovery and Diagnostic Locations' screen, click Next.
5. On 'Management Options' screen, select the option of choice
6. On 'Summary' screen, verify information presented about the database upgrade, and then click Finish.
7. On 'Progress' screen, when the upgrade is complete, click OK.
8. On 'Upgrade Results' screen, review the upgrade result and investigate any failures, and then click Close.

• Update Environment Variables
• Upgrade the Recovery Catalog
• Upgrade the Time Zone File Version when not already done earlier.
• For upgrades done by DBUA tnsnames.ora entries for that particular database will be updated in the tnsnames.ora in the new home. However entries not related to a database upgrade or entries related to standby database will not be updated as part of any DBUA action. The synchronisation of these entries needs to be done manually. Ifile directives used in tnsnames.ora, for example in the grid home, need to be updated to point to the new database home.

SYS@PRIM1> select distinct(value) from gv$parameter where name = '_file_size_increase_increment';

VALUE
--------------------------------------------------------------------------------
2143289344

The value for the init.ora parameter 'listener_networks' removed before the upgrade needs to be restored now as follows:

SYS@PRIM1> alter system set listener_networks='<original value>' sid='*' scope=both;

Data Guard only - DBUA will not affect parameters set on the standby, hence previously set underscore parameters will remain in place, however, since the values were reset in a previous step they need to be restored now. For standard installations, only the following additional underscore parameters need to be added back for a standby database.

SYS@STBY1> alter system set "_file_size_increase_increment"=2143289344 sid='*' scope=both;

Run catbundle.sql to load the required bundle patch SQL, as follows.

(oracle)$ cd $ORACLE_HOME
SYS@PRIM1> @?/rdbms/admin/catbundle.sql exa apply

Navigate to the <ORACLE_HOME>/cfgtoollogs/catbundle directory (if ORACLE_BASE is defined then the logs will be created under <ORACLE_BASE>/cfgtoollogs/catbundle), and check the following log files for any errors like "grep ^ORA <logfile> | sort -u". If there are errors, then refer to Section 3 of the README, "Known Issues". Here, the format of the <TIMESTAMP> is YYYYMMMDD_HH_MM_SS

catbundle_EXA_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_EXA_<database SID>_GENERATE_<TIMESTAMP>.log

SYS@PRIM1> alter system set dg_broker_start = true sid='*';
SYS@STBY1> alter system set dg_broker_start = true sid='*';

DGMGRL> enable configuration
DGMGRL> enable fast_start failover

Obtain latest mount-dbfs.sh script from Document 1054431.1

Download the latest mount-dbfs.sh script attached to <Document 1054431.1>, place it in a (new) directory and update the CRS resource:

(oracle)$ dcli -l oracle -g ~/dbs_group mkdir -p /home/oracle/dbfs/scripts
(oracle)$ dcli -l oracle -g ~/dbs_group -f /u01/app/oracle/patchdepot/mount-dbfs.sh -d /home/oracle/dbfs/scripts
(oracle)$ crsctl modify resource dbfs_mount -attr "ACTION_SCRIPT=/home/oracle/dbfs/scripts/mount-dbfs.sh"

Edit mount-dbfs.sh script and Oracle Net files for the new 11.2.0.4 environment

Using the variable settings from the original mount-dbfs.sh script, edit the variable settings in the new mount-dbfs.sh script to match your environment.  The setting for variable ORACLE_HOME must be changed to match the 11.2.0.4 ORACLE_HOME (/u01/app/oracle/product/11.2.0.4/dbhome_1). 

Edit tnsnames.ora used for DBFS to change the directory referenced for the parameters PROGRAM and ORACLE_HOME to the new 11.2.0.4 database home.

fsdb.local =
(DESCRIPTION =
   (ADDRESS =
     (PROTOCOL=BEQ)
     (PROGRAM=/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/oracle)
     (ARGV0=oraclefsdb1)
     (ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=BEQ)))')
     (ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1,ORACLE_SID=fsdb1')
   )
   (CONNECT_DATA=(SID=fsdb1))
)

If the location of Oracle Net files changed as a result of the upgrade, then change the setting of TNS_ADMIN in shell scripts and login files.
If using wallet-based authentication, recreate the symbolic link to /sbin/mount.dbfs. If you are using the Oracle Wallet to store the DBFS password, then run the following commands:

(root)# dcli -l root -g ~/dbs_group ln -sf \
/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/dbfs_client /sbin/mount.dbfs
(root)# dcli -l root -g ~/dbs_group ln -sf \
/u01/app/oracle/product/11.2.0.4/dbhome_1/lib/libnnz11.so /usr/local/lib/libnnz11.so
(root)# dcli -l root -g ~/dbs_group ln -sf \
/u01/app/oracle/product/11.2.0.4/dbhome_1/lib/libclntsh.so.11.1 /usr/local/lib/libclntsh.so.11.1
(root)# dcli -l root -g ~/dbs_group ldconfig

• There are no databases configured to use the home.
• The home is not a configured Grid Infrastructure home.
• ASM is not detected in the Oracle Home.

(oracle)$ unzip -q /u01/app/oracle/patchdepot/p10098816_112020_SOLARIS64_7of7.zip -d /u01/app/oracle/patchdepot

(oracle)$ cd /u01/app/oracle/patchdepot/deinstall
(oracle)$ ./deinstall -checkonly -home /u01/app/oracle/product/11.2.0/dbhome_1/
(oracle)$ ./deinstall -home /u01/app/oracle/product/11.2.0/dbhome_1/

(root)# dcli -l root -g ~/dbs_group chmod -R 755 /u01/app/11.2.0/grid
(root)# dcli -l root -g ~/dbs_group chown -R oracle /u01/app/11.2.0/grid
(root)# dcli -l root -g ~/dbs_group chown oracle /u01/app/11.2.0

(oracle)$ cd /u01/app/oracle/patchdepot/deinstall
(oracle)$ ./deinstall -checkonly -home /u01/app/11.2.0/grid/
(oracle)$ ./deinstall -home /u01/app/11.2.0/grid/

• <Document 1050908.1> - Troubleshoot Grid Infrastructure Startup Issues