Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1673290.1
Update Date:2014-05-19
Keywords:
Solution Type  Problem Resolution Sure

Solution  1673290.1 :   ODA Trace files Show ORA 15041 ; Error 19504 Creating Archive log ; ORA 312 after crash - Some Activity Caused Archivelog Full and Now Multiple Databases Will Not Come Up :  

Related Items 
    

  • Oracle Database Appliance X4-2
    •  
  • Oracle Database Appliance
    •  
  • Oracle Database Appliance Software
    •  
  • Oracle Database Appliance X3-2
    •  
Related Categories 
    

  • PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Database Appliance>DB: ODA_EST
    •  




ODA ASM diskgroups are shared by ALL databases so the impact of filling up a diskgroup can effect all databases. While the impact is severe the ability to fix the problem of releasing storage can be considered more flexible than other ASM configurations. You can choose to free space in ANY of the databases.  Used, abandoned but not deleted, test, as well as the instigating database. This note will excerpt trace files, alert.logs, and other evidence to help provide an example of where to look to find and confirm the source of the problem. We will also point to existing notes that can assist in corrective actions once the problem source is confirmed

Created from <SR 3-9012885571>

Applies to:

Oracle Database Appliance X4-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X3-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance Software - Version 2.1.0.1 to 2.10.0.0
Information in this document applies to any platform.

Symptoms

This problem is the result of filling up the diskgroup storage: (RECO)

Sometimes the problem source is more evident than others as error messages may or may not point directly to the problem source
In this case during an Import it appears that the archivelog destination was full and now all databases will hang


As the symptom includes the database(s) crashing we can look in the ALERT.LOGs for evidence

 DATABASE ALERT.LOG

Fri May 16 07:26:21 2014
Thread 1 advanced to log sequence 1608 (LGWR switch)
  Current log# 2 seq# 1608 mem# 0: +REDO/dwprod/onlinelog/group_2.309.832503665
  Current log# 2 seq# 1608 mem# 1: +RECO/dwprod/onlinelog/group_2.1906.832503665
Fri May 16 07:26:21 2014
Archived Log entry 2453 added for thread 1 sequence 1607 ID 0xff40bce9 dest 1:
Fri May 16 07:31:37 2014                                                                 <<<<<<<<<<<    First evidence of the problem
DIA0 detected that there is an archiving problem on the cluster. Several
     processes including LGWR and at least one ARC process are hung.
     Instance termination is not allowed.  External intervention
     is required.
Fri May 16 07:52:37 2014
DIA0 detected that there is an archiving problem on the cluster. Several
     processes including LGWR and at least one ARC process are hung.
     Instance termination is not allowed.  External intervention
     is required.
...
...
Fri May 16 09:20:43 2014
Shutting down instance (immediate)
...
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:43681:59938} */
ALTER DATABASE OPEN /* db agent *//* {1:43681:59938} */
This instance was first to open
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database dwprod and diskgroup resource ora.DATA.dg is established
Beginning crash recovery of 1 threads
 parallel recovery started with 2 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 2: logseq 855, block 188413, scn 13810639530
Recovery of Online Redo Log: Thread 2 Group 3 Seq 855 Reading mem 0
  Mem# 0: +REDO/dprod/onlinelog/group_3.308.832504379
  Mem# 1: +RECO/dprod/onlinelog/group_3.1911.832504379
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 2: logseq 855, block 188413, scn 13810659531
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
...
...
Unable to create archive log file '+RECO'
ARCH: Error 19504 Creating archive log file to '+RECO'                                   <<<< First ERROR # Reported
Errors in file /u01/app/oracle/diag/rdbms/dprod/dprod1/trace/dprod1_ora_25461.trc:
ORA-16038: log 4 sequence# 854 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 4 thread 2: '+REDO/dprod/onlinelog/group_4.307.832504381'
ORA-00312: online log 4 thread 2: '+RECO/dprod/onlinelog/group_4.1912.832504381'
ORA-16038 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:43681:59938} */...
 
Shutting down instance (abort)
License high water mark = 3
USER (ospid: 26752): terminating the instance

---

ASM - Stays up and generates message and Trace Files
           No specific error messages are reported in the ALERT.LOG

  

ASM1

Fri May 16 09:21:00 2014
NOTE: client dprod1:dprod deregistered
Fri May 16 09:32:43 2014
NOTE: client dwprod1:dprod registered, osid 25413, mbr 0x2
Fri May 16 09:33:02 2014
NOTE: ASM client dwprod1:dwprod disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25413.trc
 ...
Fri May 16 09:48:23 2014
NOTE: client dwprod1:dprod registered, osid 29248, mbr 0x2
Fri May 16 09:48:41 2014
NOTE: ASM client dwprod1:dprod disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_29248.trc

 

ASM2  

Fri May 16 09:30:46 2014
NOTE: ASM client dprod2:dwprod disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_24959.trc
...
...
Fri May 16 09:48:43 2014
NOTE: ASM client dwprod2:dprod disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_28572.trc

 

 

ASM ALERT.LOG points to the following Trace file


DDE rules only execution for: ORA 15041 <<<

----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
----- START DDE Actions Dump -----
Executing SYNC actions
Executing ASYNC actions
----- END DDE Actions Dump (total 0 csec) -----
*** 2014-05-16 09:32:57.105 4329 krsh.c
Unable to create archive log file '+RECO'
*** 2014-05-16 09:32:57.105 4329 krsh.c

  ARCH: Error 19504 Creating archive log file to '+RECO'

*** 2014-05-16 09:32:57.105 2917 krsi.c

  krsi_dst_fail: dest:1 err:19504 force:0 blast:1 <<
  DDE rules only execution for: ORA 312 <<<

 

Changes

Any database activity which requires the creation of logfiles, backups or increased storage demands can result in this problem
It can be the result of a single event ( such as Import in our example) but should not be considered a defect or problem due to otherwise normal database functionality

  

Cause

Diskgroup Storage exceeded during database operations


NOTE: This problem can be due to the addition of "one more logfile" as the problem is a Diskgroup storage capacity problem
              The ODA ran out of Diskgroup space during [in this case ] an import which generated multiple logfiles

  

Solution

 Determine the timeline of the problem using the trace files and alert.log entries
These files should confirm the time of the crash, last generated Logfile or other related errors

 

Corrective actions can be found in the following notes

1. How To Delete Archive Log Files Out Of +Asm? <Document 300472.1>
     or:   ASMCMD> rm -rf <filename>

2. Complete Steps To Delete Archivelogs Using The Rman Utility <Document 794383.1>

3. ORA-15041 Diskgroup Space Exhausted <Document 1367078.1>

  

 

References

<NOTE:300472.1> - How To Delete Archive Log Files Out Of +Asm?
<NOTE:794383.1> - Complete Steps To Delete Archivelogs Using The Rman Utility
<NOTE:1367078.1> - ORA-15041 Diskgroup Space Exhausted
<NOTE:351117.1> - Information to gather when diagnosing ASM space issues

Attachments 
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback