Big Data Appliance X3-2 In-Rack Expansion
 
Big Data Appliance X6-2 Hardware
 
Big Data Appliance X5-2 Starter Rack
 
Big Data Appliance X4-2 Starter Rack
 
Big Data Appliance X5-2 Hardware
 
Big Data Appliance Hardware
 
Big Data Appliance X5-2 In-Rack Expansion
 
Big Data Appliance X3-2 Full Rack
 
Big Data Appliance X4-2 In-Rack Expansion
 
Big Data Appliance X4-2 Full Rack
 
Big Data Appliance X4-2 Hardware
 
Big Data Appliance X3-2 Hardware
 
Big Data Appliance X5-2 Full Rack
 
Big Data Appliance X3-2 Starter Rack
 

PLA-Support>Eng Systems>BDA>Big Data Appliance>DB: BDA_EST
 

In this Document

Purpose

Scope

Details

Overview

About Disk Drive Identifiers

Disk Drive Identifiers

Standard Mount Points

Configuring an Operating System Disk

Partitioning the Operating System Disk

Repairing the RAID Arrays

Formatting the HDFS Partition of an Operating System Disk

Restoring the Swap Partition

Restoring the GRUB Master Boot Records and HBA Boot Order

Verifying the Disk Configuration

What If Firmware Warnings or Errors occur?

What If a Server Fails to Restart?

References

Applies to:

Purpose

The document will describe the steps for configuring a server's disk drive as an Operating System Disk for the /u01 and /dev/sda disk in Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.x/V4.x.

Scope

This document is to be used by anyone who is configuring the disk. If attempting the steps and further assistance is needed please log a service request to contact support for help.

Details

Overview

Failure of a disk is never catastrophic on Oracle Big Data Appliance. No user data should be lost. Data stored in HDFS or Oracle NoSQL Database is automatically replicated.

The following are the basic steps for replacing a server disk drive and configuring it as an Operating System Disk:

1. Replace the failed disk drive.

2. Perform the basic configuration steps for the new disk. If multiple disks are unconfigured, then configure them in order from the lowest to the highest slot number. Finish all the steps for one disk and then start with all the steps for the next.

3. Identify the dedicated function of the failed disk, either as an HDFS disk, an operating system disk, or an Oracle NoSQL Database disk.

4. Configure the disk for the operating system.

5. Verify that the configuration is correct. 

About Disk Drive Identifiers

The Oracle Big Data Appliance includes a disk enclosure cage that holds 12 disk drives and is controlled by the HBA (Host Bus Adapter). The drives in this enclosure are identified by slot numbers (0..11) and can have different purposes, for example the drives in slot 0 and 1 have a raid 1 OS and boot partitions. The drives can be dedicated to specific functions, as shown in Table 1.

Version 2 of the image introduces new device symbolic links in /dev/disk/by_hba_slot/. The links refer the physical location or slot number of a disk inside the disk enclosure cage. The links are of the form of s<n>p<m>, where n is the slot number and m is the partition number. For example in an unaltered system the /dev/disk/by_hba_slot/s0p1 corresponds to /dev/sda1, ..s0p4 to ..sda4, ..s1p1 to sdb1 etc, and disk /dev/sda itself corresponds to /dev/by-hba-slot/s0, ..sdb to ..s1 etc.

When a disk is hot swapped, the operating system cannot reuse the kernel device name. Instead, it allocates a new device name. For example if /dev/sda was hot swapped then the disk corresponding /dev/disk/by-hba-slot/s0 may link to /dev/sdn instead of /dev/sda. Therefore, the links in /dev/disk/by-hba-slot/ are automatically updated (as part of udev rules) when devices are added or removed. Hence we prefer to use the symbolic device links in /dev/disk/by-hba-slot in configuration and recovery procedures.

In the rest of the document we will refer to disk or partition by its symbolic name in /dev/disk/by-hba-slot/. Thus s0, s0p4 etc.

Disk Drive Identifiers

The following table (Table 1) shows the mappings between the RAID logical drives and the probable initial kernel device names, and the dedicated function of each drive in an Oracle Big Data Appliance server.
This information will be used in a later procedure of partioning the disk for it's appropriate function so please note which mapping is applicable for the disk drive that is being replaced.

Table 1 - Disk Drive Identifiers

Standard Mount Points

The following table (Table 2) shows the mappings between HDFS partitions and mount points. This information will be used in the later procedure so please note which mapping is applicable for the disk drive that is being replaced.

Table 2 - Mount Points

Configuring an Operating System Disk

The first two disks support the Linux operating system. These disks store a copy of the mirrored operating system, a swap partition, a mirrored boot partition, and an HDFS data partition. To configure an operating system disk, you must copy the partition table from the surviving disk, create an HDFS partition (ext4 file system), and add the software raid partitions and boot partitions for the operating system.

Complete these procedures after replacing logical drive 0 (/dev/sda). For logical drive 1 (/dev/sdb) follow procedure in "How to Configure a Server Disk After Disk Replacement as an Operating System Disk /u02 and /dev/sdb on Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.x/V4.x (Doc ID 1581373.1)."

If multiple disks are unconfigured, then configure them in order from the lowest to the highest slot number. Finish all the steps for one disk and then start with all the steps for the next.

1. Partitioning the Operating System Disk

2. Repairing the RAID Arrays

3. Formatting the HDFS Partition of an Operating System Disk

4. Restoring the Swap Partition

5. Restoring the GRUB Master Boot Records

Partitioning the Operating System Disk

To partition a logical drive:

1.   Complete the steps in the following note prior to running these steps - "Steps for Replacing a Disk Drive and Determining its Function on the Oracle Big Data Appliance V2.2.*/V2.3.1/V2.4.0/V2.5.0/V3.x/V4.x (Doc ID 1581331.1)."

2.   Confirm that there is no existing partition table on the new disk for /dev/disk/by-hba-slot/s0:   

You should see a message about a missing partition table.

OL6 Example which does not show a missing partition table:

OL5 Example which does not show a missing partition table:

3.  The partition table needs to be cleared in the following cases:

a) If the partition table displays a partition table, then clear it.

or

b) If the output shows "Error: msdos labels do not support devices that have more than 4294967295 sectors.", then clear it. For example:  

or

c) If the output shows "Error: Both the primary and backup GPT tables are corrupt.", then clear it. 

In any of the above, clear the partition table:

Example output clearing the partition table on /dev/disk/by-hba-slot/s0:

Note: You can use the "dd if=/dev/zero of=/dev/disk/by-hba-slot/s0 bs=1M count=100" command to restart an operating system disk configuration, if you make a mistake.

Example output reissuing the command from step 2 after clearing the partition table:

Also note in the case of the error: "Error: Both the primary and backup GPT tables are corrupt.", GPT may write backup tables to the end of the disk. In these  rare cases the end of the disk needs to be zeroed out as well.

If needed try:

# zero out last 100 MB of a disk
dd bs=512 if=/dev/zero of=/dev/sda count=2048 seek=$(('blockdev --getsz /dev/sda' - 2048))

Which comes from: http://unix.stackexchange.com/questions/13848/wipe-last-1mb-of-a-hard-drive

4.  Create the partition table on s0 using parted /dev/disk/by-hba-slot/s0 -s mklabel gpt print:

Example:

5.  To partition the OS disk make sure to have corresponding partitions of the surviving disk of the raid arrays. Do this by using CHS size formats. Print the CHS partition info of the surviving disk /dev/disk/by-hba-slot/s1.

List the Cylinder, Head, Sector (CHS) partition information of the surviving disk. Thus, if you are partitioning /dev/disk/by-hba-slot/s0, then enter /dev/disk/by-hba-slot/s1 in the following command:

OL6 Example output using /dev/disk/by-hba-slot/s1 since the surviving disk is /dev/disk/by-hba-slot/s1: 

OL5 Example output using /dev/disk/by-hba-slot/s1 since the surviving disk is /dev/disk/by-hba-slot/s1:

6. Create partitions 1 to 3 on /dev/disk/by-hba-slot/s0 by duplicating the partitions of the surviving disk /dev/sdb. Issue three commands in this format for /dev/disk/by-hba-slot/s0 for the disk you have replaced and file_system, start, and end with the correct values:

On OL6:

Use the file_system, start, and end addresses that you obtained in Step 5 instead of the addresses shown in the following example:   

 
Example output for /dev/disk/by-hba-slot/s0 with the values found for the surviving disk /dev/disk/by-hba-slot/s1 that were replaced for file_system, start, and end:

On OL5:

Use the file_system, start, and end addresses that you obtained in Step 5 instead of the addresses shown in the following example:   

 
Example output for /dev/disk/by-hba-slot/s0 with the values found for the surviving disk /dev/disk/by-hba-slot/s1 that were replaced for file_system, start, and end:

7.  Create primary partition 4 using the start address obtained in Step 5 and an end address of 100% for /dev/disk/by-hba-slot/s0 for the disk you have replaced:

On OL6:

# parted /dev/disk/by-hba-slot/s0 -s mkpart primary ext4 68082,213,35 100%

Example creating partition on /dev/disk/by-hba-slot/s0:

On OL5:

Example creating partition on /dev/disk/by-hba-slot/s0:

Partition 4 is an HDFS data partition so make it as big as possible. Other partitions please use exact chs info from surviving disk.The bda hardware check (bdacheckhw) checks for partition names and flags. Therefore we also need to clear the name and set the raid flags. Setting the name to empty can only be done in single command mode.

8.  Set the RAID flags for /dev/disk/by-hba-slot/s0 for the disk you have replaced:

Example setting the RAID flags on /dev/disk/by-hba-slot/s0:

9. For OL6 ONLY set the boot flag on the first partition: (Not for OL5)

 
10. For OL5 ONLY Clear the names, using parted in interactive mode for /dev/disk/by-hba-slot/s0 for the disk you have replaced:

Example output clearing the names on /dev/disk/by-hba-slot/s0:

Once in the shell then type the following to clear the names. Where parted is the parted prompt, don't type it:

 Example showing (parted) which is the parted prompt:

11. Check if the partitions are setup correctly.

OL6 Example output for /dev/disk/by-hba-slot/s0 -s unit chs print.  Note you must verify that the boot flag is set on OL6.

OL5 Example output for /dev/disk/by-hba-slot/s0

12. Complete the steps in the next section titled "Repairing the RAID Arrays."

Repairing the RAID Arrays

After partitioning the disks, repair the the two logical RAID arrays. There are two md arrays: /dev/md0 and /dev/md2

    /dev/md0 is made up of /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s1p1 and is mounted as /boot.

    /dev/md2 is made up of /dev/disk/by-hba-slot/s0p2 and /dev/disk/by-hba-slot/s1p2 and is mounted as / (root).

To repair the RAID arrays issue a series of mdadm commands in pairs as there are two arrays md0 and md2. For each partition first mark the partition failed, remove it, and add back in. 

 

1.  Assuming the faulty drives have been replaced issue the following commands to remove the partitions from the RAID arrays: 

Example:

 a) Mark the partitions as failed for /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2: 

     Example output marking the partitions as failed on /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2:

     You can ignore "No such device" messages in the mdadm commands.

 
b)  Remove the partitions from the RAID arrays for /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2: 

 Example output when removing the partitions on  /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2:

 
2.  Verify that the RAID arrays are degraded:  

Example output:

 
3.  Verify that the degraded file for each array is set to 1:  

 Example output: 

Example output:

4. Restore the partitions to the RAID arrays for /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2:   

Example output restoring the partitions on /dev/disk/by-hba-slot/s0p1 and /dev/disk/by-hba-slot/s0p2:

5.  Check that resynchronization is started, so that /dev/md[02] is in a state of recovery and not idle. Although you may see 'idle' if recovery goes too fast.

 And

 Example output:

6.  To verify that resynchronization is proceeding, you can monitor the mdstat file. A counter identifies the percentage complete.

 Example output which shows the percentage complete at 66.1%:

The following output shows that synchronization is complete:  

7.  View the contents of /etc/mdadm.conf:  

Example output:

8. Compare the output of the following command with the content of /etc/mdadm.conf from Step 7:  

 In this example output the content of the mdadm command and /etc/mdadm.conf are the same.  No changes /etc/mdadm.conf are required:

9.  If the UUIDs in the lines for /dev/md0 and /dev/md2 in the file are different from the output of the mdadm command, then use a text editor to replace them with the output of the above mdadm command.

     a. Open /etc/mdadm.conf in a text editor.
 
     b. Select from ARRAY to the end of the file, and delete the selected lines.
 
     c. Copy the output of the command into the file where you deleted the old lines.
 
     d. Save the modified file and exit.

10.  Complete the steps in the next section titled "Formatting the HDFS Partition of an Operating System Disk."

Formatting the HDFS Partition of an Operating System Disk

Partition 4 (sda4 or sdb4) on an operating system disk is used for HDFS. After you format the partition and set the correct label, HDFS rebalances the job load to use the partition if the disk space is needed.

To format the HDFS partition:

1.  Format the HDFS partition as an ext4 file system using /dev/disk/by-hba-slot/s0p4 for the disk that was replaced:

Example output formatting the HDFS partiton on /dev/disk/by-hba-slot/s0p4 as an ext4 file system:

Example showing command failing. 

If this command fails i.e. if "mkfs -t ext4 /dev/disk/by-hba-slot/s0p4" fails, then dismount the device and repeat the command.

Example output from formatting the HDFS Partition on /dev/disk/by-hba-slot/s0p4 after dismounting /u01:

2. Verify that the partition label /u01 for s0p4 is missing:

Example output when nothing is missing:

3. Dismount (only if you did not do so in step 1 above) the appropriate HDFS partition, /u01 for /dev/sda:

Example:

Once the HDFS partition is formatted as an ext4 file system then don't dismount the drive again because the device is already dismounted. Use mount -l to check if the device mounted.

Example showing /u01 is not mounted:

 
4.  Reset the partition label for /u01 and /dev/disk/by-hba-slot/s0p4:

(For OL6 use tune2fs.  For OL5 use tune4fs.)

On OL6:

On OL5:

Example on OL5 resetting the partition label on /u01 and /dev/disk/by-hba-slot/s0p4:

 
5.  Mount the HDFS partition for /u01 for the device you have partitioned:

Example mounting /u01:

You can check to see if the device is mounted:

The following shows /u01 is mounted:

 
6.  Complete the steps in the next section titled "Restoring the Swap Partition."

Restoring the Swap Partition

After formatting the HDFS partition, you can restore the swap partition.

To restore the swap partition:

1. Set the swap label with SWAP-sda3 and /dev/disk/by-hba-slot/s0p3:

Example OL6 output setting the swap label on sda3 and /dev/disk/by-hba-slot/s0p3:

Example OL5 output setting the swap label on sda3 and /dev/disk/by-hba-slot/s0p3:

 
2.  Verify that the swap partition is restored:

Example output verifying the swap partition is restored:  

 
3. Verifiy the replaced disk is listed in 'ls -l /dev/disk/by-label' output. If the replaced disk is listed then please skip to  "Restoring the GRUB Master Boot Records and HBA Boot Order." section. But if the replaced disk is NOT listed then continue to next step (4).

4. Trigger kernel device uevents to replay missing events at system coldplug.

a) For Linux OS 5, execute below command

b) For Linux OS 6, execute below command

Note:- With both commands --verbose option can be used to check what events are triggered

5. Verifiy the replaced disk is listed in 'ls -l /dev/disk/by-label' output.

6. Complete the steps in the next section titled "Restoring the GRUB Master Boot Records and HBA Boot Order."

Restoring the GRUB Master Boot Records and HBA Boot Order

After restoring the swap partition, you can restore the Grand Unified Bootloader (GRUB) master boot record and the HBA Boot Order.

To restore the GRUB boot record:

1. The device.map file maps the BIOS drives to operating system devices. The following is an example of a default device map file:

Unfortunately grub device map does not support symbolic links. Thus (hd0) is mapped to /dev/sda , (hd1) is mapped to /dev/sdb in the map file and not by /dev/disk/by-hba-slot.

For this reason one needs to edit /boot/grub/device.map such that hd0/hd1 resolves to the right kernel device name.

a) Execute below command to check which kernel device the drive is using in slot0

Sample output when device name is sda

If device name is ../../sda then jump to step2(Open GRUB) . But if device is mapped to some other name say ../../sdn then follow below steps to set hd0 to point to the new device name by following these steps ....

b) Make a copy of /boot/grub/device.map file

Sample output

c) Edit mydevice.map file to set hd0 point to the new device name

In this example /dev/sdn is the new device name in slot 0.

2. Open GRUB:

a) If device name in slot0 is /dev/sda then use default device.map file

OR

b) If device name in slot0 is not /dev/sda then use the newly created mydevice.map file

Example output when using device.map file:

3. Set the root device by entering hd0 for disk in slot0

Example setting the root device to hd0 for /dev/sda:  

 
4. Install grub by entering hd0 for disk in slot0

 Example installing grub on hd0 (/dev/sda):

 
5. Close the GRUB command-line interface:  

 Example output:

6.Verify that the boot drive in the HBA is set correctly. If it is set correctly skip the next step, Step 7 and go on to Step 8. If it is not set correctly perform the next step, Step 7.

On BDA V4.3 and higher:

a) Verify that the BootDrive VD:0 is set as the boot drive in the HBA. 

Example output when the BootDrive VD:0 is set as the boot drive in the HBA. In this case skip the next step, Step7 and go on to Step 8.

b) Example output when the BootDrive VD:0 is NOT set as the boot drive in the HBA. In this case follow the next step, Step7.

i. In this example the BootDrive VD:0 is not set (Continue to Step 7).

ii. You may also encounter the case where the BootDrive is set to VD:1.  In this case also continue to Step 7.

The standard default value for the boot drive in the HBA is VD0. If the boot drive in the HBA is set to VD1, set the boot drive to VD0 for consistency in Step 7.

On BDA V4.2 and lower:

Verify that logical drive L0 (letter L zero) is set as the boot drive in the HBA.

Example output when the logical drive L0 is set as the boot drive in the HBA. In this case skip the next step, Step7 and go on to Step 8.

Any other outout, continue to Step 7.  This includes the case when the BootDrive is not set and this includes the case when the BootDrive is set to VD:1. The standard default value for the boot drive in the HBA is VD0. If the boot drive in the HBA is set to VD1, therefore also set the boot drive to VD0 for consistency in Step 7.

7. Ensure that the Boot Drive is set correctly.  You only need to perform this step if the boot drive in the HBA is NOT set correctly as per the previous step.

On BDA V4.3 and higher:

If the 'MegaCli64 /c0 show bootdrive' command does not report that the boot drive is set in the HBA i.e. BootDrive VD:0 or that the BootDrive is VD:1 then issue the following command.

Example Output:

 Verify:

On BDA V4.2 and lower:

If the 'MegaCli64 -AdpBootDrive -get a0' command does not report logical L0 or "Boot Virtual Drive - #0 (target id - 0)" then issue the following command.

 Example output:

Verify:

8. Ensure the auto select boot drive feature is enabled.

Example output showing that the Auto Select Boot is already enabled:

9. Check the configuration. See the section titled "Verifying the Disk Configuration."

Verifying the Disk Configuration

To verify the disk configuration:

1. Check the software configuration as root user:  

Example successful output from running bdachecksw:

2. If there are errors, then redo the configuration steps as necessary to correct the problem.

If error like below occurs i.e replaced disk partition is listed at the end and all partitions are recognized then this error can be ignored and is caused due to Bug 17899101 in bdachecksw script.

Bug 17899101 is fixed in V2.4 release of BDA.

Patch 17924936 contains one-off patch for BUG 17899101 to V2.3.1 release of BDA.

Patch 17924887 contains one-off patch for BUG 17899101 to V2.2.1 release of BDA.

Refer to the Readme file for instructions on how to apply the patch.  Readme file also contains un-install instructions as needed.

What If Firmware Warnings or Errors occur?

If the bdacheckhw utility reports errors / warnings with regards to the HDD (Hard Disk Drive) Firmware Information indicating that the HDD firmware needs to be updated follow the instructions in "Firmware Usage and Upgrade Information for BDA Software Managed Components on Oracle Big Data Appliance V2 [ID 1542871.1]”.

What If a Server Fails to Restart?

The server may restart during the disk replacement procedures, either because you issued a reboot command or made an error in a MegaCli64 command. In most cases, the server restarts successfully, and you can continue working. However, in other cases, an error occurs so that you cannot reconnect using ssh. In this case, you must complete the reboot using Oracle ILOM.

To restart a server using Oracle ILOM:

1.  Use your browser to open a connection to the server using Oracle ILOM. For example:

    http://bda1node12-c.example.com
   

2.  Log in using your Oracle ILOM credentials.

3.  Select the Remote Control tab.

4.  Click the Launch Remote Console button.

5.  Enter Ctrl+d to continue rebooting.

6.  If the reboot fails, then enter the server root password at the prompt and attempt to fix the problem.

7.  After the server restarts successfully, open the Redirection menu and choose Quit to close the console window.

References

This solution has no attachment