ALERT Diskgroup Corruption Due to Invalid ASM Block Header [ENDIAN_KFBH] for Devices Larger Than 2TB with ADVM Volume on X5-2 ODA

Asset ID:	1-77-2038152.1
Update Date:	2017-03-06
Keywords:

Solution Type Sun Alert Sure

Solution 2038152.1 : ALERT Diskgroup Corruption Due to Invalid ASM Block Header [ENDIAN_KFBH] for Devices Larger Than 2TB with ADVM Volume on X5-2 ODA - 12.1.2.2 and 12.1.2.3 Only

Applies to:

Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance Software - Version 12.1.2.2 to 12.1.2.3 [Release 12.1]
Information in this document applies to any platform.

Description

Critical ACFS data corruption Bug 21369858 - ORA-15196: INVALID ASM BLOCK HEADER [KFC.C:29297] [ENDIAN_KFBH].

The corruption can occur for devices larger than 2TB in an asm disk group where an advm volume is used on ODA 12.1.2.2 and 12.1.2.3.

INTERNAL RELATED 21465186 21481512 21503355

Occurrence

Requirements to encounter this bug:

ODA X5-2
AND
ODA software version12.1.2.2 or 12.1.2.3

Please notice the issue does not exist on ODA V1 X3-2 and X4-2.

Symptoms

ASM ALERT.LOG and /or TRACE FILES:

ORA-15196 and [endian_kfbh] errors in ASM trace directory +ASM1 or +ASM2

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_xxxxxx.trc:
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh]
...
NOTE: a corrupted block from group DATA was dumped to ......
WARNING: cache read (retry) a corrupt block: group=1(DATA) fn=1 blk=277
disk=0 (HDD_E0_S07_123456...P1)

MESSAGES / ACSF logs: "ASSERTION FAILURE:

physA->startBlk >= auLimit File"

kernel: [Oracle OKS] ASSERTION FAILURE:
physA->startBlk >= auLimit File:

Potentially - Unable to start CRS

crsctl start cluster -all

CRS-2672: Attempting to start 'ora.evmd' on 'Node01'
CRS-2672: Attempting to start 'ora.cssd' on ......
...
CRS-4705: Start of Clusterware failed on node XYZ
CRS-4000: Command Start failed, or completed with errors.

Workaround

The FIX is included in ODA bundles 12.1.2.4, 12.1.2.5 and higher - see Note 888888.1 for further patch information [08/18 /2015]

Alternatively: Apply the provided patch for either 12.1.2.2 or 12.1.2.3.

NOTE: The fix provided will only avoid future corruptions.
Once the corruption has occurred, the only way to resolve the problem is to redeploy the system.

Patches

The FIX is included in 12.1.2.4 which is now available - see Note 888888.1 for further patch information [08/18 /2015]

The related patches for 12.1.2.2 and 12.1.2.3 can be applied in rolling mode. IF you have guest VM using share storage, please stop or failover the VM on the related Node before do the shutdown CRS step as mentioned in the readme and restart related VM after apply the fix.

ODA 12.1.2.2

If using ODA 12.1.2.2 please:
Download below two files in the attachement:

"Patch readme for 12.1.2.2"
"ACFS fix for ODA 12.1.2.2"

Follow the README steps to apply the fix.

ODA 12.1.2.3

If using ODA 12.1.2.3 please:
Download below two files in the attachment:

"Patch readme for 12.1.2.3"
"ACFS fix for ODA 12.1.2.3"

Follow the README steps to apply the fix.

The related RPM patch in the Note also including the fix for acfs bug: 21163764. So if we suspect the customer is hitting acfs bug 21163764, we can also let customer follow related readme to apply related patch. The Note is created for ODA x5-2 which is the only version using disk sizes of 2T+ but the patch can apply onto x4-2/x3-2 or V1 HW in case the ODA version and ACFS version match the requirement in the readme.

History

a. First Draft for review
b. Second Draft uploaded related RPM and readme.
c. Corrected minor spelling and grammatical issues
d. Add statement stop/restart VM.

Attachments

This solution has no attachment