Exadata: Datafile Copy Using RMAN to Diskgroup Fails with ORA-15041 Although Required Free Space is Available

Asset ID:	1-72-2075870.1
Update Date:	2017-05-01
Keywords:

Solution Type Problem Resolution Sure

Solution 2075870.1 : Exadata: Datafile Copy Using RMAN to Diskgroup Fails with ORA-15041 Although Required Free Space is Available

Applies to:

Oracle Database - Enterprise Edition - Version 11.1.0.7 to 12.1.0.2 [Release 11.1 to 12.1]
Oracle Exadata Storage Server Software - Version 11.1.0.3.0 to 12.1.2.2.0 [Release 11.1 to 12.1]
Exadata Database Machine X2-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Datafile copy operations from one diskgroup to another diskgroup (using RMAN) fails with the following error:

RMAN> copy datafile 5 to '+DATA_CLOUD';

Starting backup at 21-AUG-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2273 instance=ora1 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=2465 instance=ora1 device type=DISK
allocated channel: ORA_DISK_3
.
.
.
.
channel ORA_DISK_19: SID=1142 instance=ora1 device type=DISK
allocated channel: ORA_DISK_20
channel ORA_DISK_20: SID=1328 instance=ora1 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/ora_stb/datafile/USERS_ORA.656.883740831
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 08/21/2015 08:07:30
ORA-19504: failed to create file "+DATA_CLOUD"
ORA-17502: ksfdcre:4 Failed to create file +DATA_CLOUD
ORA-15041: diskgroup "DATA_CLOUD" space exhausted
channel ORA_DISK_1 disabled, job failed on it will be run on another channel
channel ORA_DISK_2: starting datafile copy
input datafile file number=00005 name=+DATA/ora_stb/datafile/USERS_ORA.656.883740831
RMAN-03009: failure of backup command on ORA_DISK_2 channel at 08/21/2015 08:07:31
ORA-19504: failed to create file "+DATA_CLOUD"
ORA-17502: ksfdcre:4 Failed to create file +DATA_CLOUD
ORA-15041: diskgroup "DATA_CLOUD" space exhausted

Note: This problem was reported on an Exadata configuration but it is also applicable to Non-Engineered Systems configurations.

Detailed description of this problem:

a) Diskgroup reports the following USABLE_FILE_MB free space (11,598,114):

[grid@ex02cloud01 ~]$ asmcmd lsdg DATA_CLOUD
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB USABLE_FILE_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 4194304 25989120 23629380 433152 11598114 0 N DATA_CLOUD/

b) This is the source file (which is about 7,681,024.01 MB in size before taking redundancy in account):

[grid@ex02cloud01 ~]$ asmcmd ls -ls
+data/ora_stb/datafile/USERS_ORA.656.883740831
Type Redund Striped Time Sys Block_Size Blocks Bytes Space Name
DATAFILE MIRROR COARSE OCT 07 11:00:00 Y 16384 491585537 8054137438208 16108555862016 USERS_ORA.656.883740831

c) Therefore +DATA_CLOUD diskgroup should be able to allocate this datafile, but it fails with the following error:

RMAN> copy datafile 5 to '+DATA_CLOUD';

Starting backup at 07-OCT-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1325 instance=ora1 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005
name=+data/ora_stb/datafile/USERS_ORA.656.883740831
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 10/07/2015
11:22:17
ORA-19504: failed to create file "+DATA_CLOUD"
ORA-17502: ksfdcre:4 Failed to create file +DATA_CLOUD
ORA-15041: diskgroup "DATA_CLOUD" space exhausted

Diagnostic:

1) "Check all repair" health check reports no errors on the target diskgroup:

SQL> alter diskgroup DATA_CLOUD check all repair
NOTE: starting check of diskgroup DATA_CLOUD
Tue Oct 06 12:36:09 2015
GMON checking disk 0 for group 2 at 25 for pid 36, osid 92616
GMON checking disk 1 for group 2 at 26 for pid 36, osid 92616

GMON checking disk 77 for group 2 at 78 for pid 36, osid 92616
GMON checking disk 78 for group 2 at 79 for pid 36, osid 92616
GMON checking disk 79 for group 2 at 80 for pid 36, osid 92616
GMON checking disk 80 for group 2 at 81 for pid 36, osid 92616
GMON checking disk 81 for group 2 at 82 for pid 36, osid 92616
GMON checking disk 82 for group 2 at 83 for pid 36, osid 92616
GMON checking disk 83 for group 2 at 84 for pid 36, osid 92616
SUCCESS: check of diskgroup DATA_CLOUD found no errors
SUCCESS: alter diskgroup DATA_CLOUD check all repair

2) AMDU dump from the DATA_CLOUD diskgroup shows the following free space:

---------------------- SUMMARY FOR DISKGROUP DATA_CLOUD
-----------------------
          Allocated AU's: 589940
               Free AU's: 5907340   <(==== Here
      AU's read for dump: 234
      Block images saved: 221987
       Map lines written: 234
         Heartbeats seen: 3
Corrupt metadata blocks: 0
       Corrupt AT blocks: 0

*******************************

END OF REPORT
********************************

Free space = 5907340 AUs x 4 MB = 23,629,360 MB = 23,075.54 GB = 22.53 TB.

3) Currently the +DATA_CLOUD diskgroup has about 11.26 TB (22.53 TB / 2) of free usable space (after redundancy).

4) ASMCMD also reports "11,598,114 MB" about 11.06 TB of "USABLE_FILE_MB", which matches the amdu calculation (above):

5) And per the output from the script1.out (Doc ID 1551288.1), "Cell Usable File MB" reports 8,955,887 MB (8.54 TB):

------------------------------------------------------------------------------
----------------------------------------------------------------------
|          |         |     |          |            |            |            |Cell Req'd |Disk Req'd |            |            |    |    |       |
|          |DG       |Num |Disk Size |DG Total    |DG Used     |DG Free    |Mirror Free |Mirror Free |Disk Usable |Cell Usable |    |    |PCT    |
|DG Name   |Type     |Disks|MB        |MB          |MB          |MB         |MB          |MB          |File MB     |File MB     |DFC |CFC |Util   |
------------------------------------------------------------------------------
----------------------------------------------------------------------
|DATA_CLOUD |NORMAL   |   60|   433,152| 25,989,120|   2,359,740| 23,629,380|   5,717,606|     504,774| 11,562,303|   8,955,887|PASS|PASS| 9.1%|
------------------------------------------------------------------------------
----------------------------------------------------------------------

6) As a reminder, the size of the affected file is about 7.32 TB (the one we need to copy to the DATA_CLOUD diskgroup):

NAME BYTES
-------------------------------------------------- ----------

+data/ora_stb/datafile/USERS_ORA.656.883740831 8.0531E+12

BYTES= 8.0531E+12 = 8,053,100,000,000 = 7.32 TB

7) Thus that means, the DATA_CLOUD diskgroup has enough free usable space (about 11 TB) to allocate the "+data/ora_stb/datafile/USERS_ORA.656.883740831" file.

Cause

This problem is due to the fact that next bug:

Bug 18009564 RMAN overestimates backup size twice as much

Solution

1) Apply the Patch:18009564 to fix this bug.

2) Any of the following patchset (which include this fix) on both Grid Infrastructure and RDBMS Oracle Homes:

11.2.0.4.4
11.2.0.4.BP10
12.1.0.2
12.2.0.0
WIN:B107P59
WIN:B204P10

Community Discussions ASM

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

Community Discussions Exadata

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

Community Discussions ODA

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

References

<BUG:21827704> - EXADATA:ORA-15041 COPYING FILE FROM ON DISK GROUP TO ANOTHER
<NOTE:18009564.8> - Bug 18009564 - RMAN overestimates backup size twice as much (False RMAN-3009 possible)
<BUG:18009564> - RMAN IS SENDING WRONG INFORMATION TO MML, THUS MML IS FAILING

Attachments

This solution has no attachment