Oracle Big Data Appliance Node Hangs during Reboot due to File System Check Error

Asset ID:	1-72-1597925.1
Update Date:	2016-10-11
Keywords:

Solution Type Problem Resolution Sure

Solution 1597925.1 : Oracle Big Data Appliance Node Hangs during Reboot due to File System Check Error

Applies to:

Big Data Appliance Hardware - Version All Versions and later
Linux x86-64

Symptoms

Rebooting a node from ssh session seems to take forever. Checking the status of the reboot in ILOM Web Remote Console indicates there is a File System Check error.

For Details about launching a Remote console refer to Doc ID 1550440.1
Doc ID 1550440.1 How to Power Cycle Oracle Big Data Appliance Node using ILOM when the node is NOT reachable using Ping/SSH

Reboot Error

Cause

Error occured while trying to mount a partition which is listed in /etc/fstab on a particular disk .

Executing MegaCli64 indicates either the disk is Offline or Unconfigured

For example

MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Sample Output

Firmware state: Unconfigured(good), Spun Up
OR
Firmware state: Offline

As the disk is currently NOT visible to the OS, mounting of the partition on that disk listed in /etc/fstab fails with errors.

Solution

To fix the issue, in the ILOM window enter system root password . Once the valid root password is provided , there will be a '(Repair filesystem) 1 #' prompt.

Depending on the state of the disk execute the needed command so that the disk is visible to OS .

1) When the disk is in Unconfigured(good) state , execute CfgLdAdd command to add the disk .

Sample Command

MegaCli64 CfgLdAdd r0[<eclosureid>:<slot-num>] a0

For example adding a disk in slot 10 with enclosure-id 20

(Repair filesystem) 1 # MegaCli64 CfgLdAdd r0[20:10] a0

Adapter 0: Created VD 0

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

2) When the disk is in Offline state , execute PDonline to bring the disk online.

Sample Command

MegaCli64 PDonline physdrv[<eclosureid>:<slot-num>] a0

For example for disk in slot 10 with enclosure-id 20

(Repair filesystem) 1 # MegaCli64 PDonline physdrv[20:10] a0

EnclId-20 SlotId-10 state changed to OnLine.

Exit Code: 0x00

3) Check the state of the disk is 'Online, Spun Up'

For example

MegaCli64 pdlist a0 | egrep "^Firm|^Slot|^Foreign"

Slot Number: 10
Firmware state: Online, Spun Up
Foreign State: None

4) Exit out of '(Repair filesystem) ' shell and system will be rebooted again.

Note:- If the disk cannot be configured to be Online , then maybe disk is BAD and needs replacement. In that case please open an Service Request to assist in resolving the issue.

References

<NOTE:1550440.1> - How to Power Cycle Oracle Big Data Appliance Node using ILOM when the node is NOT reachable using Ping/SSH

Attachments

This solution has no attachment