Big Data Appliance X3-2 Hardware
 

PLA-Support>Eng Systems>BDA>Big Data Appliance>DB: BDA_EST
 

In this Document

Purpose

Troubleshooting Steps

What does a flashing amber LED on a disk drive mean

ILOM snapshot indication of a service fault

Indications that a flashing amber LED does not represent disk failure

Resetting the service LEDs back to "OFF"

Verify Flashing LEDs are reset

References

Applies to:

Purpose

This note provides a general approach for determining if a flashing amber LED on an Oracle Big Data Appliance (BDA) disk drive is indicating a disk failure. If analysis does not point to a disk failure steps to reset a flashing amber LED are given.

Troubleshooting Steps

What does a flashing amber LED on a disk drive mean

On the Oracle Big Data Appliance (BDA) a bda-monitor service runs in the background and turns on a service LED if any of the following is true for a disk:

1. It is failed.

2. It shows a predictive failure, where the predictive failure is based on the value of "Predictive Failure Count" as returned for the disk when using:

MegaCli64 -PDList -a0| grep -E 'Slot|Failure Count'

with  "Predictive Failure Count: 0" indicative of no predictive failures.  Anything non-zero triggers an alert.

Output for 12 healthy disks looks like:

 
3. It has failed a specific performance test 5 times in a row, where the performance test is based on:

/sbin/hdparm -t /dev/sd<X> (where X is the disk location) returning 100MB/second or more.

Using /dev/sdc as an example the output of hdparm (run as root or sudo) 5 times in a row below shows that this disk is performing in the expected range:

The performance test could fail if there is greater than normal load on the system. However this scenario would not indicate that the disk is showing performance issues. Hadoop Distributed File System (HDFS) is tolerant of disk failures and disk performance so replacing disks performing less than optimally should not be necessary.

ILOM snapshot indication of a service fault

Once an amber flashing LED is observed on a disk on the BDA check the file: ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out from the ILOM snapshot to confirm that a disk service fault is set.  Collect the ILOM snapshot on the server with the flashing amber LED(s):

1. According to the directions in: How to Collect a Snapshot from an x86 Platform Service Processor version (SP or ILOM) (Doc ID 1448069.1)

In this case uncompress the ILOM snapshot and examine ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out

or

2. Via the output from the "bdadiag snapshot" utility as documented in Oracle Big Data Appliance Diagnostic Information Collection with bdadiag V2.0 (Doc ID 1516469.1).

In this case uncompress the "bdadiag snapshot" output, cd to the ilom subdirectory, uncompress the ILOM snapshot, and examine ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out

ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out should show a service fault on the disk for which the flashing amber LED is observed on the BDA.  For example for the disk in slot 3, ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out should show service fault on DBP/HDD3:

A quick alternative to collecting the ILOM snapshot to verify a disk service fault is to issue (as root or sudo):

 
Check for disk service faults on all servers of a rack with:

Indications that a flashing amber LED does not represent disk failure

Once one or more disks with flashing amber LEDs are observed on the BDA, and the corresponding service fault confirmed via ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out,  there are several additional checks that can be made to determine if the observed LED represents a hard or predictive disk failure. 

The checks listed below can be used to determine that the disk looks to be in a healthy state inspite of the observd flashing amber LED on the BDA and corresponding service fault in ipmi/@usr@local@bin@ipmiint_sunoem_led_get.out.

1. bdacheckhw utility reports a healthy hardware profile.

A healthy hardware profile with functional disks looks like below.  For example in the case of a flashing amber LED for the disk in slot three, bdacheckhw reports that the disk is healthy:

 

2. megacli64-status.out reports healthy disks

Checking the megacli64-status.out file from the bdadiag output found after uncompressing the bdadiag file in bdadiag_<clustername>_<timestamp>/raid/megacli64-status.out reports healthy status.  The output looks likes:

 

3.  megacli64-PdList_long.out shows the disk or disks with the flashing amber light have a firmware state of Online, Spun Up and a foreign state of None.

Checking the megacli64-PdList_long.out file from the bdadiag output found after uncompressing the bdadiag file in bdadiag_<clustername>_<timestamp>/raid/megacli64-PdList_long.out reports healthy status on specific disks.  The output looks like below confirming a healthy status for the disk in slot 3. 

 

4. megacli64-LdPdInfo.out Virtual Drive output for the disk indicates no problems

Checking the megacli64-LdPdInfo.out file from the bdadiag output found after uncompressing the bdadiag file in bdadiag_<clustername>_<timestamp>/raid/megacli64-LdPdInfo.out reports output like below shows healthy state for the disk or disks with a flashing amber LED.  The example output below is for the disk in slot 3.

 

5. hdparm -t /dev/sd<X> indicates hard disk timings for device reads is currently inline with performance threshold. 

Recall once the LED is turned ON it remains ON therefore a performance issue could have occurred previously causing the LED to flash but this command verifies performance is now within acceptable standard.

For example for the case of the disk in slot 3:

 

6. It is possible to write a file to the disk and read it back successfully.

Verify that coping a file onto and off of the filesystem mount on the disk or disks with the flashing amber lights is successful also indicates that the disk is functioning properly.  A quick diagnostic test can be done as below.  This test is for the disk in slot 3.

If there is no other indication of a hard or predictive disk failure associated with the flashing amber light then, as a workaround, you can reset the service LED(s) back to the OFF state.

Resetting the service LEDs back to "OFF"

Reset the service LED or LEDs back to the "OFF" state by issuing the following commands as root or sudo:

1. Before doing the reset, re-verify that the disk or disks with the flashing LED does not indicate a hard or predictive failure.

Query the state of all disks on the node with:

# MegaCli64 -PDList -a0| grep -E 'Slot|state|Failure Count|flagged'| sed 's/Slot/\nSlot/'

Disks without a hard or predictive failure show:

Output for 12 healthy disks looks like:

 

2. If no failure, reset service LEDs to "OFF"

On a single server:

For all servers in a rack:

For all servers in a cluster:

 
Output for resetting the service LEDs to "OFF" for all servers in a rack looks like:

Verify Flashing LEDs are reset

Verify that service LED was reset by checking that the LED that was flashing is now "OFF" on the BDA,  i.e. the LED is solid green.

If you can not immediately check the state on the BDA verify that the service LED is reset to OFF by:

1. Capturing the ILOM snapshot as described in "ILOM snapshot indication of a service fault".  The ipmi/@usr@local@bin@ipmiint_sunoem_led_get.output should show all service LEDs OFF:

 
2. Querying the ON/OFF state of the service LED for a particular HDD with the command (as root or sudo) several times:

If OFF is returned all 6 times then the LED is OFF and not blinking. Otherwise, you might see a state of ON and OFF which indicates a blinking LED.

To verify that disk in slot 3:

 

If any of these verifications fail please file a support request (SR) via My Oracle Support to further investigate.

This solution has no attachment