Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1469932.1
Update Date:2018-01-05
Keywords:

Solution Type  Problem Resolution Sure

Solution  1469932.1 :   Sun Storage 7000 Unified Storage System: Issue With Shadow Migration. Oracle DB ORA-01578 ORA-01110 errors  


Related Items
  • Sun ZFS Storage 7420
  •  
  • Sun Storage 7110 Unified Storage System
  •  
  • Oracle ZFS Storage ZS3-2
  •  
  • Sun Storage 7210 Unified Storage System
  •  
  • Oracle ZFS Storage ZS4-4
  •  
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun Storage 7310 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
  • Oracle ZFS Storage ZS3-4
  •  
  • Oracle ZFS Storage Appliance Racked System ZS4-4
  •  
  • Sun ZFS Storage 7320
  •  
  • Sun Storage 7720 Unified Storage System
  •  
  • Oracle ZFS Storage ZS3-BA
  •  
Related Categories
  • PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
  •  




In this Document
Symptoms
Changes
Cause
Solution
References


Created from <SR 3-5856001001>

Applies to:

Oracle ZFS Storage Appliance Racked System ZS4-4 - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7720 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

We have initiated a shadow copy to Sun storage :adc07stor21 from a netapp filer and it got finished @ below time

2012-6-10 11:25:53 6c121f9a-2626-ca71-98a8-eb14811a77ae Migration of 'psatoc_21/i_oradata' from 'nfs://adc07ntap29-bkp/vol/psatoc_29_data_3_2M0A0G0_MIGSRC_EXP_17062012/i_oradata' completed.

 

After that

Starting from Jun 11 22:33 Hrs EDT - customer started getting index corruption errors :      at the path : /psatoc/oradata/data01/a_txn_ind04.dbf


Below is the error reported:

Mon Jun 11 22:33:37 2012
Hex dump of (file 405, block 82827) in trace file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc
Corrupt block relative dba: 0x6541438b (file 405, block 82827)
Completely zero block found during buffer read
Reading datafile '/psatoc/oradata/data01/a_txn_ind04.dbf' for corruption at rdba: 0x6541438b (file 405, block 82827)
Reread (file 405, block 82827) found same corrupt data (no logical check)
Mon Jun 11 22:33:37 2012
Corrupt Block Found
  TSN = 382, TSNAME = APPS_TS_TX_IDX
  RFN = 405, BLK = 82827, RDBA = 1698775947
  OBJN = -1, OBJD = 299514, OBJECT = APPS_TS_TX_IDX, SUBOBJECT =
  SEGMENT OWNER = , SEGMENT TYPE = Temporary Segment
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc (incident=65081):
ORA-01578: ORACLE data block corrupted (file # 405, block # 82827)
ORA-01110: data file 405: '/psatoc/oradata/data01/a_txn_ind04.dbf'
Incident details in: /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc:

 

Changes

The block corruption on this system was caused by ALL ZEROS within the block as reported in the alert log:

Mon Jun 11 22:33:37 2012
Hex dump of (file 405, block 82827) in trace file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc
Corrupt block relative dba: 0x6541438b (file 405, block 82827)
Completely zero block found during buffer read  ->>> ALL ZEROS
Mon Jun 11 22:33:37 2012
Hex dump of (file 405, block 82827) in trace file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc
Corrupt block relative dba: 0x6541438b (file 405, block 82827)
Completely zero block found during buffer read
Reading datafile '/psatoc/oradata/data01/a_txn_ind04.dbf' for corruption at rdba: 0x6541438b (file 405, block 82827)
Reread (file 405, block 82827) found same corrupt data (no logical check)
Mon Jun 11 22:33:37 2012
Corrupt Block Found
     TSN = 382, TSNAME = APPS_TS_TX_IDX
     RFN = 405, BLK = 82827, RDBA = 1698775947
     OBJN = -1, OBJD = 299514, OBJECT = APPS_TS_TX_IDX, SUBOBJECT =
     SEGMENT OWNER = , SEGMENT TYPE = Temporary Segment
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/trace/PSATOC_j000_32366.trc  (incident=65081):
ORA-01578: ORACLE data block corrupted (file # 405, block # 82827) <-------------------------------------------------------------
ORA-01110: data file 405: '/psatoc/oradata/data01/a_txn_ind04.dbf'
Incident details in: /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc
Errors in file /psatoc/log/diag/rdbms/psatoc/PSATOC/incident/incdir_65081/PSATOC_j000_32366_i65081.trc:

 

How to I diagnose and fix this ORA-01578 error?

Answer: 
An ALL ZEROS block corruption is described in Note 1545366.1 ; this is a physical corruption (Note 840978.1) and as such is resolved by either applying media recovery or recreating the segment (Note 1578.1)

Since you are getting a ORA-01578 running Oracle on a personal computer (Windows), it's a safe bet that you have a disk crash.  Oracle is intended to run in any platform as it was designed; this includes Windows.

Cause

We have initiated a shadow copy to Sun storage :adc07stor21 from a netapp filer and it got finished @ below time


2012-6-10 11:25:53 6c121f9a-2626-ca71-98a8-eb14811a77ae Migration of 'psatoc_21/i_oradata' from 'nfs://adc07ntap29-bkp/vol/psatoc_29_data_3_2M0A0G0_MIGSRC_EXP_17062012/i_oradata' completed.
 
http://www.dba-oracle.com/t_ora_01578_data_block_corrupted.htm

In this case, the ORA-01578 block corruption error is:

ORA-01578: ORACLE data block corrupted

To diagnose the ORA-01578 error, use the oerr utility to look-up the ORA-01579 error:

ORA-01578: ORACLE data block corrupted (file # string, block # string)

Cause: The data block indicated was corrupted, mostly due to software errors.

 

NOLOGGING blocks are not zeroed out so ORA-26040 will not be produced. NOLOGGING blocks have a especial designed format recognized by Oracle that does not includes zeros.

 

Action: Try to restore the segment containing the block indicated. This is resolved by either applying media recovery or recreating the segment (Note 1578.1)

This may involve dropping the segment and recreating it. If there is a trace file, report the errors in it to your ORACLE representative.

Normally, block corruption is often due to a hardware error, and there is likely a trace file or log entry that was made when it happened.

 

By examining the trace file, you can often tell the root cause of the corruption, or an Oracle Representative must now tell you the root cause of the corruption.

Beware, you may need to perform an RMAN recovery on this database, up to the point of the corruption, usually a disk failure.
 

Solution

You will need to open a Call with https://support.oracle.com/ with your Oracle Database CSI# to help salvage or clean up the ORA-01578: ORACLE data block corrupted (file # 405, block # 82827)

Action Plan:  This is resolved by either applying media recovery (Note 342972.1) or recreating the segment (Note 1578.1)


There is not a need to reference bbed as the block description. The error already explains it was corrupt with-in the block.  bbed is also deprecated in 11g; it is an internal tool not inteded to be used by customers.


For NAS support tech receiving this type of SR. Link this doc and move the parent SR to the Oracle DB team PLA Product code 5 : Oracle Database - Enterprise Edition  > RDMS

Oracle technical Support can use the block editor (BBED) utility to inspect the data blocks and see EXACTLY why they became corrupted. You can also inspect the blocks with BBED yourself.


 

References

<NOTE:840978.1> - Physical and Logical Block Corruptions. All you wanted to know about it.
<NOTE:342972.1> - HOW TO PERFORM BLOCK MEDIA RECOVERY (BMR) WHEN BACKUPS ARE NOT TAKEN BY RMAN.
<NOTE:1578.1> - OERR: ORA-1578 "ORACLE data block corrupted (file # %s, block # %s)" Master Note
<NOTE:729433.1> - Cannot Reuse a Corrupt Block in Flashback Mode, ORA-1578
<NOTE:1545366.1> - Physical Corrupted Blocks consisting of all Zeroes indicate a problem with OS, HW or Storage

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback