Oracle Database Appliance Software
 
Linux OS
 
Oracle Database Appliance X3-2
 
Oracle Database Appliance
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Database Appliance>DB: ODA_EST
 
_Old GCS Categories>ST>Server>Engineered Systems>Oracle Database Appliance>Hardware
 

Applies to:

Goal

This article provides steps to replace failed / faulty system boot disk on Oracle Database Appliance Servers.

Solution

For X5-2

How to Replace a boot drive in an Oracle Database Appliance X5-2 Server node (Doc ID 1991446.1)  

 Important Note: There are two common scenarios for needing to replace a Boot Disk

For X4-2 and X3-2

NOTE: "oakcli add disk -local" feature was introduced starting in 12.1.2.3.0. for Bare Metal Installations (Only)

There are two system boot disks on each Database Appliance Server  

For ODA V1

ODA V1 - continued

Partitions on these two System Boot Disks are protected by RAID1 (mirroring) at OS level.

Following commands can be used to find the details about each RAID device

Following file shows current sync status of the RAID devices on the system

  /proc/mdstat

Properly working / synchronized RAID devices will have following status

If any system boot disk (for example /dev/sdb) is failed, then contents of /proc/mdstat look like as below

The following steps can be used to replace the failed system boot disk

 
Important Note: There are two common scenarios for needing to replace a Boot Disk

1. Remove the failed/faulty system boot disk partitions from the RAID group

IF the disk is not completely failed use the mdadm --manage -- FAIL command for the problem disk before removal>

Comment:
 This example is showing the failing and removal of BOTH system disks to confirm commands and outputs.
  (IF you lose the OS on both boot disks then this node would require redeployment/re-image)

Now, run cat /proc/mdstat, it should look something like this:

Now, you can use the remove command:

2. Replace the failed / faulty system boot disk

As system boot disks are hot pluggable, Identify and replace the FAILED system boot disk located at system back panel of the database appliance server. Generally, amber Service Required LED might be lit for the failed / faulty disk drive.

 3. Identify the new device name created for the replaced disk using following command*

*for V2  lsscsi -v | grep 600G

In this output, /dev/sdc is the new device name created for the replaced disk

4. Partition the new device (similar to the one which is running ok) using following steps
    Backup the partition table from properly running disk

Check and ensure that there is no partition table on replaced disk* (i.e /dev/sdc)

* V2 topology will be different

Partition the replaced disk i.e /dev/sdc using partition table backed up to /tmp/sda_part

Check and ensure that both are having similar partition boundaries

UPDATE [5/2015] - PERFORM THIS STEP BEFORE STEP 5 

5. Add replaced disk to the raid group

6. Check and ensure that sync is in progress

7. Update Grub MBR

On the replaced disk update the MBR stamp so it is identified as a bootable disk. Run the below commands to generate the /tmp/mbr.sh script.

     GRUB_SCRIPT="/tmp/mbr.sh"
     OS_DISK="sdc"
     echo "grub <<EOF" >>$GRUB_SCRIPT
     echo "device (hd0) /dev/$OS_DISK" >>$GRUB_SCRIPT
     echo "root (hd0,0)" >>$GRUB_SCRIPT
     echo "setup (hd0)" >>$GRUB_SCRIPT
     echo "EOF" >>$GRUB_SCRIPT  

NOTE: In the above OS_DISK needs to be the replaced disk.
     Run the /tmp/mbr.sh

8. Run a validation check

      --- "validate" will shows errors on the disk until the rebuild is 100% complete.

Now system boot disks are back to their normal state. 
 

9. Update the multipath.conf

Need to ensure that the device mapper disks are not created for the System Boot Disks and hence need to add it to the blacklist section of /etc/multipath.conf file.

If you encounter the following error  mdadm: Cannot open /dev/sdXX: Device or resource busy
Refer to pre- Step#5 on setting the blacklist section  Document:1483141.1 - mdadm: Cannot open /dev/sda1: Device or resource busy


Get the scsi id for the replaced disk

Add to the blacklist section of the multipath.conf

For latest x5-2 system we need add something like:

blacklist {
devnode "^asm/*"
devnode "ofsctl"
wwid *WD500BLHXSUWD-WXD1E71LUZT9*
wwid *WD500BLHXSUWD-WX71E71WY554*
}

in the black list in case it showing SATA_WDC_WD500BLHXSUWD-WXD1E71LUZT9_ in the wwid.

References

This solution has no attachment