"Volume Failure Message" and "Bad Health" shown in Cloudera Manager for DataNode

Asset ID:	1-72-1596121.1
Update Date:	2016-04-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1596121.1 : "Volume Failure Message" and "Bad Health" shown in Cloudera Manager for DataNode

Applies to:

Big Data Appliance Hardware - Version All Versions and later
Big Data Appliance Integrated Software - Version 2.0.1 and later
Linux x86-64

Symptoms

The Cloudera Manager shows the DataNode with "bad health". The message is: "The data node has 1 volume of failure(s). Critical threshold any."

The bdacheckhw on that node showed that disk 10 is bad with the following error.

ERROR: Wrong disk 10 status : Failed No alert

Cause

This was caused by a failed disk. The errors indicate that the disk 10 is bad on one of the nodes in the Oracle Big Data Appliance cluster.

Solution

For versions 2.0.1 to 2.1.2:

1. Replace the disk using the steps in:
Steps for Replacing a Disk Drive and Determining its Function on the Oracle Big Data Appliance V2.0.1/2.1.0/2.1.1/2.1.2 (Doc ID 1514231.1)

AND

2. Complete the configuration using steps in:

How to Configure a Server Disk After Replacement as an HDFS Disk on Oracle Big Data Appliance V2.0.1/2.1.0/2.1.1/2.1.2 (Doc ID 1514996.1)

For versions 2.2.0 and higher:

1. Replace the disk using the steps in:
Steps for Replacing a Disk Drive and Determining its Function on the Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.0.0/V3.0.1/V3.1.0/V4.0.0/V4.1.0/V4.2.0/V4.3.0 (Doc ID 1581331.1)

AND

2. Complete the configuration using steps in:

How to Configure a Server Disk After Replacement as an HDFS Disk or Oracle NoSQL Database Disk on Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.0.0/V3.0.1/V3.1.0/V4.0.0/V4.1.0/V4.2.0/V4.3.0 (Doc ID 1581583.1)

References

<NOTE:1514231.1> - Steps for Replacing a Disk Drive and Determining its Function on the Oracle Big Data Appliance V2.0.1/2.1.0/2.1.1/2.1.2
<NOTE:1514996.1> - How to Configure a Server Disk After Replacement as an HDFS Disk on Oracle Big Data Appliance V2.0.1/2.1.0/2.1.1/2.1.2
<NOTE:1581331.1> - Steps for Replacing a Disk Drive and Determining its Function on the Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.0.0/V3.0.1/V3.1.0/V4.0.0/V4.1.0/V4.2.0/V4.3.0
<NOTE:1581583.1> - How to Configure a Server Disk After Replacement as an HDFS Disk or Oracle NoSQL Database Disk on Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.0.0/V3.0.1/V3.1.0/V4.0.0/V4.1.0/V4.2.0/V4.3.0

Attachments

This solution has no attachment