ASM DROPS DISKS EVEN AFTER CONFINEMENT OFFLINE IS CLEARED

Asset ID:	1-72-1663577.1
Update Date:	2014-04-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1663577.1 : ASM DROPS DISKS EVEN AFTER CONFINEMENT OFFLINE IS CLEARED

Applies to:

Exadata X3-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

<p >*Symptoms
<span >Briefly describe the symptoms of the problem. Remember to delete any customer specific information

The disks enter into confined offline. The alert is cleared in the alerthistory after a short time.
But ASM dropped the disk after disk repair time

Alerthistory.out

76_1     2014-04-08T14:06:16+02:00     info         "Data hard disk entered confinement status. The LUN 0_8 changed status to warning - confinedOnline. CellDisk changed status to normal - confinedOnline.

76_2     2014-04-09T11:55:53+02:00     warning      "Data hard disk entered confinement offline status. The LUN 0_8 changed status to warning - confinedOffline. CellDisk changed status to normal - confinedOffline.

76_3     2014-04-09T12:07:25+02:00     clear        "Data hard disk status changed to normal.

From above we can see that the alert is cleared at 2014-04-09T12:07:25

From ASM alert log we can see that the disks were droped

Wed Apr 09 15:33:45 2014
WARNING: PST-initiated drop of 1 disk(s) in group 1(.2812041431))
SQL> alter diskgroup DATA_DM06 drop disk DATA_DM06_CD_08_DM06CEL08 force /*
ASM SERVER */
NOTE: GroupBlock outside rolling migration privileged region

But cell alert log shows the below :

Wed Apr 09 12:07:24 2014
CDHS: Do cd health state change after confinement CD_08_dm06cel08 testFailed 0
CDHS: Do cd health state change CD_08_dm06cel08 from HEALTH_BAD_OFFLINE to newState HEALTH_GOOD
[CDP] Set CD perf state normal.
..
..
NOTE: Initiating ASM Instance operation: ASM OFFLINE disk on 3 disks
CDHS: Done cd health state change CD_08_dm06cel08 from HEALTH_BAD_OFFLINE to newState HEALTH_GOOD

Cause

After confinement test passed, cellsrv still published OFFLINE for the corresponding cell disks. Later
these disks are expired on ASM side (due to disk repair time ) and got dropped.

This issue is due to bug 17615340

Solution

This issue is due to bug 17615340.

The disks will be automatically re-added and come back online

The process can also be triggered by killing the XDMG process on all DB Nodes. It will automatically be respawned.

Do a "ps -ef |grep -i xd" and kill the xdmg processes on each of the compute nodes.

A patch is available for 11.2.3.3.0 for the Database Machine

The patch is included in 11.2.3.3.1 and later releases of the Exadata Software

If the patch do not exist for an earlier version, please open an SR.

References

<BUG:17615340> - X2-8: AUTO MGMT DID NOT BRING ALL THE DISK ONLINE AFTER DISK RESTORED TO NORMAL
<BUG:18608408> - ASM DROPS DISKS AFTER CONFINEMENT OFFLINE IS CLEARED

Attachments

This solution has no attachment