Exadata: ORA-27603: Cell Storage I/O Error in Database Alert.log- ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] in $CELLTRACE/alert.log

Asset ID:	1-72-1611602.1
Update Date:	2014-01-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1611602.1 : Exadata: ORA-27603: Cell Storage I/O Error in Database Alert.log- ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] in $CELLTRACE/alert.log

Applies to:

Exadata Database Machine X2-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Following errors may be reported in database instance's alert log:

Errors in file /u01/app/oracle/diag/rdbms/prdcmsmy/prdcmsmy1/trace/prdcmsmy1_ckpt_98444.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.1.7/DATA_GEXA_CD_10_gexacel03 at offset 10729029632 for data length 16384
ORA-27626: Exadata error: 201 (Generic I/O error)
WARNING: Write Failed. group:1 disk:49 AU:2558 offset:0 size:16384
path:o/192.168.1.7/DATA_GEXA_CD_10_gexacel03
incarnation:0xf0f071aa asynchronous result:'I/O error'
subsys:OSS iop:0xfffffd7ffd5c2200 bufp:0xfffffd7ffd622e00 osderr:0xc9 osderr1:0x0
Errors in file /u01/app/oracle/diag/rdbms/prdcmsmy/prdcmsmy1/trace/prdcmsmy1_ckpt_98444.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 3 of virtual extent 5 logical extent 2 of file 486 in group 1 on disk 49 allocation unit 2558
NOTE: process _ckpt_prdcmsmy1 (98444) initiating offline of disk 49.4042289578 (DATA_GEXA_CD_10_GEXACEL03) with mask 0x7e[0x7] in group 1
Thu Aug 01 05:13:20 2013

The $CELLTRACE/alert.log of the storage cell referenced in the above errors shows:

Sat Aug 03 19:19:19 2013
Cellsrv encountered a fatal signal 11
Errors in file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/gexacel03/trace/svtrc_23810_81.trc (incident=305):
ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] [11] [0x000000000] [] [] []
Incident details in: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/gexacel03/incident/incdir_305/svtrc_23810_81_i305.trc
Exception [type: 11] [ADDR:0x0]
QuarantineMgr: Fault does not have QM protection (threadID=81, beingMonitored=false)
Cellsrv encountered a fatal signal 11

Alerthistory of one or more storage cells logged the following errors:

32 2013-07-18T21:37:50+08:00 critical "ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] [11] [0x000000000] [] [] []"
33 2013-07-22T20:34:17+08:00 critical "ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] [11] [0x000000000] [] [] []"
34 2013-07-25T23:03:28+08:00 critical "ORA-07445: exception encountered: core dump [_ZN13DestBufferCtl25allocDefaultPayloadHeaderEP9Cacheable()+49] [11] [0x000000000] [] [] []"

svtrc*.trc incident trace log from the cell shows the following call stack:

----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ossex_dump_stack()+ call kgdsdst() 000000000 ? 000000000 ?
1130 062F806B8 ? 000000001 ?
000000001 ? 000000003 ?
ossex_dump_all()+68 call ossex_dump_stack() 000000000 ? 000000003 ?
000000003 ? 000000001 ?
000000001 ? 000000003 ?
ossex_exception_han call ossex_dump_all() 000000003 ? 000000003 ?
dler()+3114 000000003 ? 000000001 ?
000000001 ? 000000003 ?
__sighandler() call ossex_exception_han 00000000B ? 0630868B0 ?
dler() 000000002 ? 000000042 ?
000000400 ? 000000003 ?
_ZN13DestBufferCtl2 signal __sighandler() 7FDA008485E0 ? 7FDA008485E0 ?
5allocDefaultPayloa 00000000D ? 001AC35B0 ?
dHeaderEP9Cacheable 7FDAA0147F00 ? 7FDAA0221A50 ?
()+49
_ZN15PredicateFilte call _ZN13DestBufferCtl2 7FD9FD5F87C8 ? 7FDA008485E0 ?
r7processEv()+361 5allocDefaultPayloa 00000000D ? 001AC35B0 ?
dHeaderEP9Cacheable 7FDAA0147F00 ? 7FDAA0221A50 ?
()
_ZN10UserThread8mai call _ZN15PredicateFilte 7FD9FD5C6380 ? 7FDA008485E0 ?
nLoopEv()+521 r7processEv() 00000000D ? 001AC35B0 ?
7FDAA0147F00 ? 7FDAA0221A50 ?
_ZN10UserThread3run call _ZN10UserThread8mai 7FDB07DF6228 ? 7FDA008485E0 ?
Ev()+722 nLoopEv() 00000000D ? 001AC35B0 ?
7FDAA0147F00 ? 7FDAA0221A50 ?
_ZN9Scheduler8sched call _ZN10UserThread3run 7FDB07DF6228 ? 7FDA008485E0 ?
uleEv()+128 Ev() 00000000D ? 001AC35B0 ?
7FDAA0147F00 ? 7FDAA0221A50 ?
_Z16kernelThreadMai call _ZN9Scheduler8sched 7FDB07DF6920 ? 7FDA008485E0 ?
nPv()+4 uleEv() 00000000D ? 001AC35B0 ?
7FDAA0147F00 ? 7FDAA0221A50 ?
oracle_fp_thread_ma call _Z16kernelThreadMai 7FDB07DF6920 ? 7FDA008485E0 ?
in()+110 nPv() 00000000D ? 001AC35B0 ?

Changes

Issue can occur when a cell is performing a very high number of IOs which are split across multiple celldisks, not a common occurrence.

Cause

Unpublished Bug 14514449 : ORA-7445 _ZN13DESTBUFFERCTL25ALLOCDEFAULTPAYLOADHEADEREP9CACHEABLE which is marked as duplicate of unpublished Bug 14457249 : DEFERRED BUFFER ALLOCATION FAILURE ON SUBMIT PATH PROPAGATED TO DATABASE .

Solution

A fix for the bug is included in the Exadata Storage Server Software 11.2.3.2.1 version and later.

Attachments

This solution has no attachment