ASM Rebalance stuck due to incorrect file type

Asset ID:	1-72-1489981.1
Update Date:	2014-08-13
Keywords:

Solution Type Problem Resolution Sure

Solution 1489981.1 : ASM Rebalance stuck due to incorrect file type

Applies to:

Exadata Database Machine V2 - Version All Versions and later
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.4 [Release 11.2]
Information in this document applies to any platform.

Symptoms

Diskgroup created on Exadata Storage
A rebalance operation getting stuck after running for couple of hours
No signs of a hang. Dumping hanganalyze did not report any chain
Using psstack to review stacks of ARB0 process, it showed different functions, which is a sign that process was not stuck

run #1

kfk_transit_waitIO
kfk_transitIO
kffRelocateWait
kffRelocate
kfdaExecute
kfgbRebalExecute
kfgbDriver
ksbabs
kfgbRun
ksbrdp
opirip
opidrv
sou2o
opimai_real
ssthrdmain
main

run #2

kfk_reap_ios
kfk_iol
kfkrequest
kfk_transitIO
kffRelocateWait
kffRelocate
kfdaExecute
kfgbRebalExecute
kfgbDriver
ksbabs
kfgbRun
ksbrdp
opirip
opidrv
sou2o
opimai_real
ssthrdmain
main

Timestamp for ARB0 trace file was not changing

Changes

Rebalance was triggered by a Storage Cell failure that was unavailable more than the disk repair timer.

Cause

Reviewing the different ARB0 trace files, because rebalance was restarted multiple times, it was observed that the last line on trace file was referencing always the same ASM file number.
Looking into v$asm_file, it was discovered the type of file was ASMVOL.
Type ASMVOL is related to a VOLUME created inside of a diskgroup to be used by ACFS (before 12.1.0.2).
ACFS is not supported before 12.1.0.2 on Exadata environments, but the command to create a VOLUME on a diskgroup using Exadata storage will not fail.

Solution

Drop the VOLUME from the diskgroup using command

alter diskgroup <diskgroup name> drop volume <volume name>;

The volume was dropped while rebalance was running and before re-allocating extents from the ASMVOL file, allowing rebalance to continue.

Attachments

This solution has no attachment