SuperCluster - Unable to create new databases or start existing databases against > 12.1.2.1.1 cells due to control file errors such as ora-00200 or ora-00205

Asset ID:	1-72-2044088.1
Update Date:	2015-08-21
Keywords:

Solution Type Problem Resolution Sure

Solution 2044088.1 : SuperCluster - Unable to create new databases or start existing databases against > 12.1.2.1.1 cells due to control file errors such as ora-00200 or ora-00205

Applies to:

Oracle SuperCluster T5-8 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 Full Rack - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 Half Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster T5-8 Full Rack - Version All Versions to All Versions [Release All Releases]
Oracle Solaris on SPARC (64-bit)

Symptoms

New Database creations can fail with stacks similar to

DBCA_PROGRESS : 4% Creating and starting Oracle instance DBCA_PROGRESS : 5% DBCA_PROGRESS : 6% DBCA_PROGRESS : 7% DBCA_PROGRESS : 8% ORA-01501: CREATE DATABASE failed ORA-00200: control file could not be created ORA-00202: control file: '+DATAC5' ORA-15045: ASM file name '+DATAC5' is not in reference form ORA-17502: ksfdcre:5 Failed to create file +DATAC5

If following storage cell patching existing database startup could fail as follows

WARNING: failed to open a disk[o/192.168.8.17;192.168.8.18/DATA7_SI_CD_09_orlt5cel01] ORA-00204: error in reading (block 1, # blocks 1) of control file ORA-00202: control file: '+DATA7/dbm01/control01.ctl' ORA-15081: failed to submit an I/O operation to a disk

Changes

New install with 12.1.2.1.2 Exadata Storage Server software or Storage cells were recently patched to this level.

Cause

A code change was done in 12.1.2.1.2 that enforced Global DB_UNIQUE_NAME across all virtual hosts sharing cells.

This was put in as a fix for Bug 19775528 - core dumps in kutyxtt_traverse_type and kutyxtte_deserialize_internal_old which basically broke cloud style setups where different virtual hosts sharing the cells have the same db unique name and enhancement has been filed to fix this Bug 20602164 : CLOUD: CELLSRV NEED TO SUPPORT CONFLICTING DB NAME IN DIFFERENT CLUSTER

Solution

New DB Creation case

1) Pick a different SID / DB_UNIQUE_NAME for the database creation that is unique across the entire enviroment sharing the cells. This may not always be possible especially for canned applications.

2) Edit the cellint.ora on each storage cell to add in _cell_db_unique_name_check=false and then restart all cell services. This can be done rolling one cell at a time.

Problems starting existing database post patching and need real time restoration getting them back up.

1) alter cell events="immediate cellsrv.cellsrv_setparam('_cell_db_unique_name_check',false)";

2) Edit the cellint.ora on each storage cell to add in _cell_db_unique_name_check=false so the value stays persistant following a cellsrv restart or a cell boot

Disabling this check does possibly expose the customer to  Bug 19775528 - core dumps in kutyxtt_traverse_type and kutyxtte_deserialize_internal_old.
However to encounter that they would need to meet the following conditions of having two databases not only with the same DB_UNIQUE_NAME but also each of those running at two different DB versions 112.x and 12.1.x. 
To this date we have not encountered a costumer to hit this on SuperCluster. So disabling the feature can be viewed as low risk but not completely risk free.

References

<NOTE:1452277.1> - SuperCluster Critical Issues

Attachments

This solution has no attachment