CRS-4662 CRS-4000 Errors While Trying To Delete Crashed Node

Asset ID:	1-72-1381059.1
Update Date:	2018-03-21
Keywords:

Solution Type Problem Resolution Sure

Solution 1381059.1 : CRS-4662 CRS-4000 Errors While Trying To Delete Crashed Node

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.2 [Release 11.2]
Exadata Database Machine V2 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

A node in the cluster has crashed due to hardware or OS problems and is not recoverable. You are attempting to remove the node from the cluster, but you receive the following errors:

[root@dbm1db01 ~]# /u01/app/11.2.0.2/grid/bin/crsctl delete node -n dbm1db04
CRS-4662: Error while trying to delete node dbm1db04.
CRS-4000: Command Delete failed, or completed with errors.

Changes

One of the nodes has crashed irreversibly.

Cause

This could be due to either of the following internal bugs:

Bug 11705438 - CRSCTL DELETE NODE RECEIVES CRS-4662 AND CRS-4000 - The voting file has become inconsistent and cannot be changed without the crashed node.

Bug 10014392 - CRSCTL DELETE NODE FAILS WITH CRS-4662 & CRS-4000 - The latest disk heartbeat information has become inconsistent and cannot be changed without the crashed node.

Solution

Internal Bug 11705438 is fixed in 11.2.0.2 GI PSU5, 11.2.0.3 and 12.1.

Internal Bug 10014392 is fixed in 11.2.0.2 GI PSU1, 11.2.0.3 and 12.1.

Per Development: As a workaround, you can move the voting file to other location (another ASM diskgroup, supported cluster filesystem or supported NFS):

crsctl replace votedisk <new-location>

This can be done on a running cluster. You can check the removal by running olsnodes to see if the node is gone. You can also run the following cluster verify utility command:

[oraclenodedb01 bin]$ ./cluvfy stage -post nodedel -n nodedb03 -verbose

Performing post-checks for node removal

Checking CRS integrity...
The Oracle Clusterware is healthy on node "nodedb02"
The Oracle Clusterware is healthy on node "nodedb01"
The Oracle Clusterware is healthy on node "nodedb08"
The Oracle Clusterware is healthy on node "nodedb07"
The Oracle Clusterware is healthy on node "nodedb06"
The Oracle Clusterware is healthy on node "nodedb05"
The Oracle Clusterware is healthy on node "nodedb04"

CRS integrity check passed
Result:
Node removal check passed

Post-check for node removal was successful.

Please see OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) <Document 428681.1> for more details.

References

<BUG:11705438> - CRSCTL DELETE NODE RECEIVES CRS-4662 AND CRS-4000
<NOTE:428681.1> - OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)
<BUG:10014392> - CRSCTL DELETE NODE FAILS WITH CRS-4662 & CRS-4000

Attachments

This solution has no attachment