Big Data Appliance X3-2 Full Rack
 
Big Data Appliance X3-2 Hardware
 
Big Data Appliance X3-2 In-Rack Expansion
 
Big Data Appliance Hardware
 
Big Data Appliance X3-2 Starter Rack
 

PLA-Support>Eng Systems>BDA>Big Data Appliance>DB: BDA_EST
 

In this Document

Purpose

Troubleshooting Steps

BDA Utilities to check status of disks

MegaCli64 Commands to Assist in Troubleshooting

Gathering Information about the Disk Drives on the Server

Troubleshooting a Flashing Amber LED on an Oracle Big Data Appliance

Checking the Disk Service Faults

Errors Reported on Disk Seen in bdacheckcluster Output

Errors Reported on Disk Seen in bdacheckhw Output

Summary

References

Applies to:

Purpose

 This document provides information on troubleshooting disk errors, warnings, info messages, failing, and failed disks on Oracle Big Data Appliance.

Troubleshooting Steps

Oracle Big Data Appliance (BDA) utilities such as bdacheckcluster, bdacheckhw, and bdachecksw when run may result in errors, warnings, and info messages about Oracle Big Data Appliance disks. The information below will show some helpful utility commands and output that may be seen and how to further troubleshoot whether the disk is in fact already failed or is in the process of failing. The information below will help determine whether a service request should be opened to get a new disk drive shipped and replaced. The spares kit contains one or two spare disks that can be used depending on the BDA Rack shipped. The spare disks may be used but it is important to open a service request to get the failed disk replaced.

BDA Utilities to check status of disks

Please note all utilities should be run as the root user.

1. The bdacheckcluster utility should be run on node01 of the BDA Cluster as root user.

To run bdacheckcluster enter the following:

Sample output showing hardware checks failing on a node:

If an error is seen continue investigating with bdacheckhw on the node found to have the error.

2. If bdacheckcluster reports hardware checks failing, find out if those are related to disks. Do this by running the bdacheckhw command on the node / server found to have the error.

To run bdacheckhw enter the following:

If bdacheckhw reports a disk issue continue investigating with the MegaCli64 commands shown below.

Sample output showing an error with wrong disk 5 status:

3. The bdachecksw utility can also be run on each individual server to find any disk related errors. In general the bdachecksw will show information about partitions and mounts.

To run bdachecksw enter the following:

Sample output showing warnings and errors with the partitions due to disk errors follows. This output does not show /u04 or /u05 which may indicate a problem with two of the disks.  Further investigation with the MegaCli64 commands below should be helpful.

MegaCli64 Commands to Assist in Troubleshooting

There are a number of MegaCli64 Commands which can assist in troubleshooting disk failure.

Here are some of the MegaCli64 commands that can be used and sample output.

To check the Firmware state issue the following command: 

Sample output showing Firmware state: Unconfigured(good), Spun Up: 

In the instance of seeing Firmware state: Unconfigured(good) or Firmware state: Unconfigured(bad) the disk will need to be replaced. Log a service request to replace the disk. Please note there is no need to be concerned about Firmware state: Online, Spun Up. 

The physical drive information can also be stored in a file for review:

Open the pdinfo.tmp file you created in a text editor and search for the following:

• Disks that have a Foreign State of Foreign.
• Disks that have a Firmware state of Unconfigured.
• Disks that have a Drive has flagged a S.M.A.R.T alert : Yes
• Disks that have a Media Error Count: non 0
• Disks that have a Other Error Count: non 0
• Disks that have a Predictive Failure Count: non 0 

The following example shows a Firmware state of Unconfigured(bad) and a Foreign State of Foreign:

Here is an example of a disk that shows a Firmware state of Unconfigured(good):

Use MegaCli LdPdInfo a0 to verify mapping of logical to physical drive numbers.

Example output: 

For Predictive Failure Counts check the value of "Predictive Failure Count" as returned for a disk when using:

A "Predictive Failure Count: 0" indicates no predictive failures.  Anything non-zero triggers an alert.

Gathering Information about the Disk Drives on the Server

The "lsscsi", "mount -l", and "ls -l /dev/disk/by-label" commands can provide information about disks that are missing, not showing up in output, or mountpoints which are not found by the system during disk failure.

Verify if the disk is recognized using the "lsscsi" command:

Sample output showing where slots 3 and 4 are not listed:

If observing similar output where slots are not listed then log a service request to investigate further if the disk needs to be replaced.

Determine if the mountpoints are all listed using "mount -l" command.

Sample output showing the mountpoints where /dev/sdd1 on /u04 type ext4 (rw,nodev,noatime) [/u04] is missing:

If observing similar output with missing mount points then log a service request to investigate if the disk needs to be replaced.

Use "ls -l /dev/disk/by-label" to determine if any device is missing. The following sample shows u03 -> ../../sdc1 is missing:

 If observing similar output with missing devices then log a service request to investigate further replacement of the disk.

Troubleshooting a Flashing Amber LED on an Oracle Big Data Appliance 

If a flashing amber LED is seen on a server in the BDA cluster please see the troubleshooting tips in the following document:

Troubleshooting Flashing Amber LED on Oracle Big Data Appliance V2.0 for Non-Failure Status (Doc ID 1537798.1).

Checking the Disk Service Faults

To check the Disk Service Faults as root issue the following command:

Output for the case where all disks are healthy:

Any ONs in the above is indicative that the disk requires replacing. File a service request.

Errors Reported on Disk Seen in bdacheckcluster Output

If executing bdacheckcluster shows an info message about disk errors on one of the BDA nodes like:

And running the "MegaCli64 ldpdinfo a0" command on the node shows the following:

Solution

Then no disk replacement is needed. The Media Error Count, can be ignored. These are not failures but represent recoverable read/write errors. More information can be found in "Running bdacheckcluster utility on Oracle Big Data Appliance Reports: INFO: Errors Reported On Disk 7 : 8 0 (Doc ID 1568792.1)."

Errors Reported on Disk Seen in bdacheckhw Output

If running bdacheckhw on a BDA node errors with:

And disk 2 is a HITACHI disk but looks ok:

However the "MegaCli64 LdPdInfo a0" and "MegaCli64 pdlist a0" report:

Further checking in the example output from running "MegaCli64 LdPdInfo a0" command shows Firmware state: Online, Spun Up:

The output of "lsscsi" and "mount -l" look ok.

Solution

The disk should be replaced at the soonest opportunity. Log a service request to have a disk drive shipped and replaced on the server.

The Media Error Count, can be ignored. These are not failures and represent recoverable read/write errors. More information can be found in "Running bdacheckcluster utility on Oracle Big Data Appliance Reports: INFO: Errors Reported On Disk 7 : 8 0 (Doc ID 1568792.1)."

The disk however is failing as shown in the Predictive Failure Count: 5 and the S.M.A.R.T alert: Yes. A predictive failure due to a SMART alert indicates a failing drive which should be replaced. However a disk showing predictive failures/SMART alerts is still usable although it has a high chance of becoming unusable in the near future. More information can be found in "bdaclustercheck/bdacheckhw Fail on Oracle Big Data Appliance: ERROR: Wrong disk status:Online,Spun Up Yes alert (Doc ID 1580223.1)."

Summary

There are many BDA utilities and MegaCli64 commands to assist in determining whether a disk is failing, failed, or fine.

If a flashing amber LED is found and is determined to be ok per "Troubleshooting Flashing Amber LED on Oracle Big Data Appliance V2.0 for Non-Failure Status (Doc ID 1537798.1)" and/or if bdacheckcluster is run and a media count error is identified then the disk is fine.

If however such errors are accompanied by Predictive Failure Count: X and S.M.A.R.T alert: Yes then get the disk replaced.

Similarly if the MegaCli64 utilities, mount, lsscsi, etc. show the problems discussed in this document then file a Service Request with Oracle Support to investigate and get the disk replaced.

References

This solution has no attachment