Upgrade 11.2.0.x to 11.2.0.y and 12.1.0.1 Failure Of Rootupgrade Script on previously cloned GI

Asset ID:	1-72-1485970.1
Update Date:	2015-09-07
Keywords:

Solution Type Problem Resolution Sure

Solution 1485970.1 : Upgrade 11.2.0.x to 11.2.0.y and 12.1.0.1 Failure Of Rootupgrade Script on previously cloned GI

Applies to:

Exadata Database Machine X2-2 Half Rack - Version All Versions and later
Exadata Database Machine X2-2 Hardware - Version All Versions and later
Exadata Database Machine X2-2 Qtr Rack - Version All Versions and later
Exadata Database Machine X2-8 - Version All Versions and later
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
This issue could occur for those in the middle of an upgrade or outside of an upgrade.

Symptoms

Scenario 1 - During an Upgrade on an Oracle Grid Infrastructure Home that was previously cloned

When upgrading from 11.2.0.x to 11.2.0.y and 12.1.0.1 the Oracle Grid Infrastructure Home was previously cloned, you may encounter this error:

Can't open <path to Grid Infrastructure cloned from> No such file or directory at /u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 14843.
/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install
/u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

NOTE: This situation can occur when upgrading an Oracle Grid Infrastructure Home that has previous been cloned from an older version. The user will see the <GRID_HOME>/log/crflogd/crflogdOUT.log of the cloned TO Oracle Grid Infrastructure Home filled with "No such file or directory" messages refering to the path of the previous Oracle Grid Infrastructure Home (cloned FROM). Example as follows :

<path of the cloned TO Grid Infrastructure Home>/log/crflogd/crflogdOUT.log:

<path of the cloned FROM Grid Infrastructure Home>/crf/db/dm01db01: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/dm01db01: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/db01db01: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/dm01db01/__db.001: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/dm01db01: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/dm01db01/__db.001: No such file or directory
<path of the cloned FROM Grid Infrastructure Home>/crf/db/dmdb01: No such file or directory

Scenario 2 - After Cloning an Oracle Grid Infrastructure Home (outside of an upgrade):

The problem with the references to non-existing files as seen in scenario 1 can also occur when not upgrading. In those cases the same error message in the crflogdOUT.log file will be seen. Perform the following actions for your environment to verify if you are exposed to this issue after cloning an Oracle Grid Infrastructure Home:

(root)# cat <path of the cloned TO Grid Infrastructure Home>/crf<node name>.ora | egrep "CRFHOME|BDBLOC"

In those failure cases, the output will be:

BDBLOC=<path of the cloned FROM Grid Infrastructure Home>/crf/db/<node name>
CRFHOME=<path of the cloned FROM Grid Infrastructure Home>

Example as follows:

(root)# egrep "CRFHOME|BDBLOC" /u01/app/11.2.0.3/grid/crf/admin/crfdm01db01.ora
BDBLOC=/u01/app/11.2.0.2/grid/crf/db/dm01db01
CRFHOME=/u01/app/11.2.0.2/grid

In this case the location of BDBLOC and CRFHOME is not equal to the location of the cloned TO Oracle Grid Infrastructure home. Additional to this the user may also see the same "No such file or directory" as mentioned in scenario 1.

Cause

Due to unpublished bug 14168708, the mentioned locations for CRFHOME and BDBLOC may still point to the Grid Infrastructure that was previously cloned from.

Solution

Follow step a) below if the issues as described earlier are encountered during an upgrade or after cloning and using an Oracle Grid Infrastructure Home.

Follow step b) below if you want to prevent against the issues described earlier and want to be sure the Oracle Grid Infrastructure Home is prepared properly

If you should encounter this error in the middle of an upgrade or after cloning an Oracle Grid Infrastructure Home, you will need to update the 'crf'-files in the cloned TO Oracle Grid Infrastructure Home as described as follows:
Note: Be sure to stop the crf resource on all nodes within the cluster as oracle GI software owner, example as follows: # $ORACLE_HOME/bin/crsctl stop res ora.crf -init


Manually edit <GRID_HOME>/crf/admin/crf<node>.ora in the cloned TO Oracle Grid Infrastructure Home and change the values for BDBLOC and CRFHOME, as follows :

Old values:
BDBLOC=/<path to Grid Infrastructure cloned FROM>/crf/db/dm01db01
CRFHOME=/<path to Grid Infrastructure cloned FROM>

New values need to match the new cloned grid home:
BDBLOC=/<path to Grid Infrastructure cloned TO>/crf/db/dm01db01
CRFHOME=/<path to Grid Infrastructure cloned TO>
Note: Be sure to start the crf resource on all nodes within the cluster as oracle GI software owner, example as follows: # $ORACLE_HOME/bin/crsctl start res ora.crf -init

Notes:
- This same change needs to be done on all nodes in the cluster to the file referenced above if it exists.
- As a precaution take backup copies of the files before making any change.
- FOR UPGRADES ONLY: Once the change has been done on all nodes, the upgrade can be re-attempted by restarting the rootupgrade.sh script.

b. Refer to Oracle® Clusterware Administration and Deployment Guide Chapter 5 Cloning Oracle Clusterware

In step 3 follow all instructions carefully for the type of copy you want to make (uncompressed and compressed). The steps will include instructions to remove the 'crf'-files.

References

<NOTE:1364947.1> - How to Proceed When Upgrade to 11.2 Grid Infrastructure Cluster Fails
<BUG:14373758> - ROOTUPGRADE SCRIPT FAILED WHEN TRYING TO REMOVE CRF FILES IN OLDER CRSHOME
<NOTE:1373255.1> - 11.2.0.1/11.2.0.2 to 11.2.0.3 Grid Infrastructure and Database Upgrade on Exadata Database Machine
<NOTE:330358.1> - Oracle Clusterware 10gR2/ 11gR1/ 11gR2/ 12cR1 Diagnostic Collection Guide
<NOTE:1364946.1> - How to Downgrade 11.2.0.3 Grid Infrastructure Cluster to Lower 11.2 GI or Pre-11.2 CRS

Attachments

This solution has no attachment