Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-2060753.1
Update Date:2015-10-02
Keywords:
Solution Type  Problem Resolution Sure

Solution  2060753.1 :   Exadata: Warning Cellsrv Rejecting Open Request from Host Due to DB Name Conflict  

Related Items 
    

  • Oracle Database - Enterprise Edition
    •  
  • Oracle SuperCluster M6-32 Hardware
    •  
  • SPARC SuperCluster T4-4 Half Rack
    •  
  • Oracle SuperCluster T5-8 Half Rack
    •  
  • Oracle SuperCluster T5-8 Full Rack
    •  
  • Oracle Exadata Storage Server Software
    •  
  • SPARC SuperCluster T4-4 Full Rack
    •  
Related Categories 
    

  • PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
    •  






Created from <SR 3-11416949331>


Applies to:  

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.2 [Release 11.2 to 12.1]
Oracle Exadata Storage Server Software - Version 11.2.2.3.3 to 12.1.2.2.0 [Release 11.2 to 12.1]
Oracle SuperCluster T5-8 Full Rack - Version All Versions to All Versions [Release All Releases]
Oracle SuperCluster M6-32 Hardware - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 Full Rack - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.



Symptoms


1.  Database cloning (duplicate) process is reporting I/O errors (in the database alert.log) when accessing the ASM disks (Exadata GridDisks) associated to all the CELL Storage Servers:


a. I/O errors accessing the GridDisks in the  oracloudcel03 CELL:


Errors in file /app/oracle/diag/rdbms/ORCL3/ORCL3i1/trace/ORCL3i1_ora_28844.trc:
ORA-15025: could not open disk "o/192.190.35.09;192.190.35.04/DATA_BCP_CD_08_oracloudcel03_adm"                
"o/192.190.35.09;192.190.35.04/DATA_BCP_CD_08_oracloudcel03_adm"  
WARNING: disk locally closed resulting in I/O error [0xf4]                                                                                
disk locally closed resulting in I/O error [0xf4]      


b. I/O errors accessing the GridDisks in the  oracloudcel02 CELL:


Errors in file /app/oracle/diag/rdbms/ORCL3/ORCL3i1/trace/ORCL3i1_ora_28844.trc:
ORA-15025: could not open disk "o/192.190.35.07;192.190.35.08/DATA_BCP_CD_08_oracloudcel02_adm"              
"o/192.190.35.07;192.190.35.08/DATA_BCP_CD_08_oracloudcel02_adm"
WARNING: disk locally closed resulting in I/O error [0xf4]                                                                                
disk locally closed resulting in I/O error [0xf4]      


c. I/O errors accessing the GridDisks in the  oracloudcel01 CELL:  


Errors in file /app/oracle/diag/rdbms/ORCL3/ORCL3i1/trace/ORCL3i1_ora_28844.trc:
ORA-15025: could not open disk "o/192.190.35.06;192.190.35.05/DATA_BCP_CD_01_oracloudcel01_adm"              
"o/192.190.35.06;192.190.35.05/DATA_BCP_CD_01_oracloudcel01_adm"  
WARNING: disk locally closed resulting in I/O error [0xf4]                                                                                
disk locally closed resulting in I/O error [0xf4]


 

2. Previous errors were generated as result of a database cloning (duplicate) process from one RAC cluster to another different RAC cluster (through RMAN). 


 


3. Both clusters (source and target) are sharing/using the same Exadata Storage CELLs.


 


4. Whereas, all the CELL servers are generating the following errors (as reported by the Cellsrv process) during the database cloning (duplicate) process:


Warning: Cellsrv rejecting open request from host <compute node name>[Pid:<#>] (Cid: <#>) due to DB name conflict (Db: <Name> Cid: <#>), disk <disk Name>



a. oracloudcel01 CELL is reporting the next errors many times:


.
.

Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_01_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_02_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_03_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_04_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_05_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_06_oracloudcel01_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_07_oracloudcel01_adm
Last alert is happening too often (20 allowed in 300 seconds), it will be suppressed for 300 seconds
.
.
.



b. Same behavior for oracloudcel02 CELL:


.
.
.
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_00_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_01_oracloudcel02_adm
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_02_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_03_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_04_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_05_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_06_oracloudcel02_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_07_oracloudcel02_adm
Last alert is happening too often (20 allowed in 300 seconds), it will be suppressed for 300 seconds
.
.


c. And same in oracloudcel03 CELL:


.
.

Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_01_oracloudcel03_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_02_oracloudcel03_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_03_oracloudcel03_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_04_oracloudcel03_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_05_oracloudcel03_adm
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_06_oracloudcel03_adm
Tue Sep 22 10:58:28 2015
Warning: Cellsrv rejecting open request from host oracloudd2z1[pid:28844] (cid: d59d757a2422eff2ff29752203fd6f9b) due to DB name conflict (db: ORCL3 cid: 5398ef3cc4536f71bfb5e1a95d41a02e), disk DATA_BCP_CD_07_oracloudcel03_adm
Last alert is happening too often (20 allowed in 300 seconds), it will be suppressed for 300 seconds
.
.


 


5.  RMAN reported the next errors during cloning (duplicate) process:


RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 09/22/2015 10:07:46
RMAN-05501: aborting duplication of target database
RMAN-06136: ORACLE error from auxiliary database: ORA-00200: control file could not be created
ORA-00202: control file: '+DATA_BCP'
ORA-15045: ASM file name '+DATA_BCP' is not in reference form
ORA-17502: ksfdcre:5 Failed to create file +DATA_BCP
ORA-15081: failed to submit an I/O operation to a disk


 



6. But, all the Griddisks in all the Exadata Storage CELLs were validated and are in good shape (they are not reporting any real I/O error or problem):



a.  “oracloudcel01_adm” CELL:


        name:                   DATA_BCP_CD_00_oracloudcel01_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_01_oracloudcel01_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_02_oracloudcel01_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_03_oracloudcel01_adm
        errorCount:             0
        status:                 active
.
.
.
        name:                   RECO_UAT_CD_09_oracloudcel01_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_10_oracloudcel01_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_11_oracloudcel01_adm
        errorCount:             0
        status:                 active



b.  “oracloudcel02_adm” CELL:  


        name:                   DATA_BCP_CD_00_oracloudcel02_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_01_oracloudcel02_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_02_oracloudcel02_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_03_oracloudcel02_adm
        errorCount:             0
        status:                 active
.
.
.
.
        name:                   RECO_UAT_CD_09_oracloudcel02_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_10_oracloudcel02_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_11_oracloudcel02_adm
        errorCount:             0
        status:                 active


 

c.  “oracloudcel03_adm” CELL:



        name:                   DATA_BCP_CD_00_oracloudcel03_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_01_oracloudcel03_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_02_oracloudcel03_adm
        errorCount:             0
        status:                 active
        name:                   DATA_BCP_CD_03_oracloudcel03_adm
        errorCount:             0
        status:                 active
.
.
.
.
        name:                   RECO_UAT_CD_09_oracloudcel03_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_10_oracloudcel03_adm
        errorCount:             0
        status:                 active
        name:                   RECO_UAT_CD_11_oracloudcel03_adm
        errorCount:             0
        status:                 active




7. All the ASM disks are reported in good shape too (at ASM layer):



a.  ASM disks from “oracloudcel01_adm” CELL:


DATA_BCP_CD_00_oracloudcel01_adm active ONLINE Yes
DATA_BCP_CD_01_oracloudcel01_adm active ONLINE Yes
DATA_BCP_CD_02_oracloudcel01_adm active ONLINE Yes
DATA_BCP_CD_03_oracloudcel01_adm active ONLINE Yes
.
.
.
RECO_UAT_CD_08_oracloudcel01_adm active ONLINE Yes
RECO_UAT_CD_09_oracloudcel01_adm active ONLINE Yes
RECO_UAT_CD_10_oracloudcel01_adm active ONLINE Yes
RECO_UAT_CD_11_oracloudcel01_adm active ONLINE Yes



b.  ASM disks from “oracloudcel02_adm” CELL:


DATA_BCP_CD_00_oracloudcel02_adm active ONLINE Yes
DATA_BCP_CD_01_oracloudcel02_adm active ONLINE Yes
DATA_BCP_CD_02_oracloudcel02_adm active ONLINE Yes
DATA_BCP_CD_03_oracloudcel02_adm active ONLINE Yes
.
.
.
RECO_UAT_CD_08_oracloudcel02_adm active ONLINE Yes
RECO_UAT_CD_09_oracloudcel02_adm active ONLINE Yes
RECO_UAT_CD_10_oracloudcel02_adm active ONLINE Yes
RECO_UAT_CD_11_oracloudcel02_adm active ONLINE Yes



c. ASM disks from “oracloudcel03_adm” CELL:


DATA_BCP_CD_00_oracloudcel03_adm active ONLINE Yes
DATA_BCP_CD_01_oracloudcel03_adm active ONLINE Yes
DATA_BCP_CD_02_oracloudcel03_adm active ONLINE Yes
.
.
.
RECO_UAT_CD_08_oracloudcel03_adm active ONLINE Yes
RECO_UAT_CD_09_oracloudcel03_adm active ONLINE Yes
RECO_UAT_CD_10_oracloudcel03_adm active ONLINE Yes
RECO_UAT_CD_11_oracloudcel03_adm active ONLINE Yes


 



8. Therefore, the I/O errors (reported by the database instance) accessing the griddisks (in all the CELLs) are misleading - i.e., there are no problems at the physical/ASM disks layer.



Cause


1. DB_UNIQUE_NAME is being globally tracked by the CELLSRV process to guarantee uniqueness among the entire Exadata Storage (in all the CELLs).

2. This is documented in the following unpublished bug (currently being worked by Development):
 


 Bug 20602164 - CLOUD: CELLSRV NEED TO SUPPORT CONFLICTING DB NAME IN DIFFERENT CLUSTER


 


Solution


This MOS note will be updated once the bug is fixed in a future code line.  


In the meantime, use the following workarounds until the fix for Bug 20602164 is released:


Option #1:

Set “_cell_db_unique_name_check=false” in all the CELL storage servers as follows:


SQL> alter cell events="immediate cellsrv.cellsrv_setparam('_cell_db_unique_name_check',false)";


 


Note: In order to provide persistence on every cellsrv restart or a cell boot, you will need to set the "_cell_db_unique_name_check=false" parameter in the cellint.ora (in all the Exadata Storage CELLs)


 
Option #2:

Use a different database name to avoid this database name conflict.








   


Note: By disabling this parameter if the two databases with the same name are different versions (e.g. one 12c and the other 11g), then the databases are exposed to Bug 19775528 - core dumps in kutyxtt_traverse_type and kutyxtte_deserialize_internal_old


  




 


Community Discussions


Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)


Click here to open in main browser window




References
 <BUG:21342978> - CELLSRV IS REJECTING IO REQUEST WHILE TRYING TO RESTORE A DATABASE WITH RMAN
 <BUG:20602164> - CLOUD: CELLSRV NEED TO SUPPORT CONFLICTING DB NAME IN DIFFERENT CLUSTER
 <BUG:19637262> - CELLCLI LIST DATABASE DOESN'T WORK WITH NON-UNIQUE DB NAMES
 <BUG:20839618> - X5-2: CELLSRV BLOCKING IO FOR AN ALREADY DELELED DB IN A DECONFIGURED CLUSTER



Attachments

This solution has no attachment


















       

Copyright © 2018 Oracle, Inc.  All rights reserved.

 Feedback