Exadata X3-2 Hardware
 
Exadata X4-2 Hardware
 
Exadata Database Machine X2-2 Qtr Rack
 
Exadata X3-2 Half Rack
 
Exadata X4-2 Quarter Rack
 
Exadata Database Machine X2-2 Full Rack
 
Exadata X3-2 Full Rack
 
Exadata X4-2 Half Rack
 
Exadata Database Machine X2-8
 
Exadata X3-8 Hardware
 
Exadata Database Machine X2-2 Half Rack
 
Exadata Database Machine X2-2 Hardware
 
Exadata X4-2 Full Rack
 
Exadata X4-2 Eighth Rack
 
Exadata X3-2 Quarter Rack
 
Exadata Database Machine V2
 
Exadata X3-8b Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 
_Old GCS Categories>ST>Server>Engineered Systems>Exadata>Hardware
 

Canned Action Plan procedure to replace an Exadata Compute (Database) node hard disk drive (Predictive or Hard Failure). This covers Exadata disk alert HALRT-02007 and HALRT-02008.

Applies to:

Goal

Identify and replace a failed hard disk drive from an Exadata Compute (Database) node for hard or predictive failures (X4-2 and earlier).

Solution

DISPATCH INSTRUCTIONS:

The customer may choose to do the replacement themselves. In this case, the disk should be sent out using a parts-only dispatch.

The following information will be required prior to dispatch of a replacement:

• Type of Exadata (V2, X2-2, X2-8, X3-2, X3-8, X4-2)

• Type of database node (V2=x4170 / X2-2 = x4170m2 / X2-8 = x4800 or x4800m2 / X3-2 = x3-2 (formerly x4270m3 Server) / X3-8 = x2-8 (formerly x4800m2) server / X4-2 = x4-2)

• Name/location of database node

• Slot number of failed drive

• Image Version (output of "/opt/oracle.cellos/imageinfo -all")

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?: Linux megaraid familiarity

TIME ESTIMATE: 60 minutes

Complete time may be dependent on disk re-sync time.

TASK COMPLEXITY: 0

CRU-optional; default is FRU with Task Complexity: 2

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:
PROBLEM OVERVIEW: A hard disk in an Exadata V2/X2-2/X2-8 or X3-2/X3-8 or X4-2 compute (database (DB)) node needs replacing.  For X5-2 and later models, use Note 1967510.1.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

Hard disks in Exadata DB nodes are configured into RAID volumes and are hot swappable provided the failed hard disk has been offlined by LSI MegaRAID that manages the volume, and ZFS if running Solaris. The volume contains redundancy and should remain online, while in a degraded state.

A critical hard disk failure may be marked “critical” or “failed” depending on release version, and is referred to as critical hard failure in this document.
A predictive hard disk failure may be marked  “predictive failure” or “proactive failure” which may or may no include additional information on failure mode type.  There may be differences based on exact failure mode and release version, the process should be treated the same and is referred to as preditive failure in this document.

For a critical hard failure, the LED for the failed hard disk should have the "OK to Remove" blue LED illuminated/flashing and have the "Service Action Required" amber LED illuminated/flashing. This may trigger alarm HALRT-02007 - refer to Note 1113034.1.

For a predictive failure, the LED for the failed hard disk should have the “Service Action Required” amber LED illuminated/flashing.On certain image revisions, predictive failures may not yet be removed from the volume and may not have a fault LED on. This may trigger alarm HALRT-02008 - refer to Note 1113014.1.

The normal DB node volume arrangement depends on the OS installed and the current active image version. Use “/opt/oracle.cellos/imageinfo” to determine the current active image version, and “uname -s” to determine the OS type. The volumes expected are as follows:

V2/X2-2/X3-2/X4-2 Linux only, if dual-boot Solaris image partition has been reclaimed or was not present:

• 3-disk RAID 5 with 1 global hotspare on images 11.2.3.1.1 and earlier
• 4-disk RAID 5 on images 11.2.3.2.0 and later

X2-2/X3-2/X4-2 Linux and Solaris dual-boot, if other OS image partitions have not been reclaimed:

• 2-disk RAID 1 for Linux on images 11.2.2.3.2 and later.
• 2 single-disk RAID0 as 1 mirrored zpool for Solaris on images 11.2.2.3.2 and later. 

X2-2/X3-2/X4-2 Solaris only, if dual-boot Linux image partition has been reclaimed:

• 4 single-disk RAID0 volumes configured into 2 mirrored zpool's (rpool and data), on images 11.2.2.3.2 and later

X2-8 Linux only, if dual-boot Solaris image partition has been reclaimed:

• 7-disk RAID 5 with 1 global hotspare on images 11.2.3.1.1 and earlier
• 8-disk RAID 5 on images 11.2.3.2.0 and later

X2-8 Linux and Solaris dual-boot, if other OS image partitions have not been reclaimed:

• 3-disk RAID 5 with 1 global hotspare for Linux on images 11.2.3.1.0 and earlier
• 4 single-disk RAID0 volumes configured into 2 mirrored zpool's (rpool and data) on images 11.2.3.1.0 and earlier
• Solaris support on X2-8 was discontinued as of image 11.2.3.1.1

X3-8 Linux only:

• 8-disk RAID 5

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

1. Backup the volume and be familiar with the restore from bare metal procedure before replacing the disk. See Note 1084360.1 for details.

If the DB node was running 11.2.2.1.1 or 11.2.2.2.x images and was in a state of write-through caching mode at some stage (default is write-back), there is a possibility that the Linux file system is corrupt due to a disk controller firmware bug. When this is encountered the file system may have been operating normally however will go read-only when attempting to rebuild the corrupted blocks across to the hotspare disk. This may be unavoidable as the rebuild copy back from hotspare to replacement occurs automatically. This requires a bare metal restore to correct.  To check the status of the current cache policy, use the following command, the current should be WriteBack, not WriteThrough. The Linux example shows the state that is not desired, Solaris is showing the desired state:

Linux:

Solaris:

2. Identify the disk using the Amber fault and Blue OK-to-Remove LED states. The DB node server within the rack can be determined from the hostname usually, and the known default Exadata server numbering scheme counting server numbers up from 1 as the lower most DB node in the rack. The server's white Locate LED may be flashing as well. 

The slot number locations are labelled on the right hand front bezel ear on V2/X2-2 DB Nodes and on next to each disk slot on X3-2/X4-2 and X2-8/X3-8 DB nodes.

If still unsure on the slot location, use the following commands to identify the faulted disk:

a. Obtain the enclosure ID for the MegaRAID card:

Linux:

Solaris:

b. Identify the physical disk slot that is failed:

Linux:

  Solaris:

"Unconfigured(bad)" is the expected state for the faulted disk. In this example, it is located in physical slot 0, and it can be seen that the Hotspare in slot 3 has started rebuilding the volume.

If all disks show as Online or Hotspare, then the disk may be in predictive failure state but not yet gone offline. The failed disk can be identified using this additional information:

Linux:

Solaris:

In this example, the disk in slot 1 has reported itself as predictive failed several times but is still online. This disk should be considered the bad one. For more details refer to Note 1452325.1.

c. Use the locate function which turns the "Service Action Required" amber LED on flashing:

Linux:

Solaris:

where E# is the enclosure ID number identified in step a, and S# is the slot number of the disk identified in step b. In the example above, the command would be:

3. Verify the state of the RAID is optimal or rebuilding if there is a hotspare, or degraded if there is not, with the good disk(s) online before hot-swap removing the failed disk.

If the failed disk was the global hotspare, then this step should be skipped.

Linux (RAID5 Example):

 Linux (RAID1 Example):

4. On the drive you plan to remove, push the storage drive release button to open the latch.

5. Grasp the latch and pull the drive out of the drive slot (Caution: The latch is not an ejector. Do not bend it too far to the right. Doing so can damage the latch. Also, whenever you remove a storage drive, you should replace it with another storage drive or a filler panel, otherwise the server might overheat due to improper airflow.)

6. Wait three minutes for the system to acknowledge the disk has been removed.

7. Slide the new drive into the drive slot until it is fully seated.

8. Close the latch to lock the drive in place.

9. Verify the "OK/Activity" Green LED begins to flicker as the system recognizes the new drive.  The other two LEDs for the drive should no longer be illuminated. The server's locate and disk's service LED locate blinking function should automatically turn off.

If it does not, it can be manually turned off for the device using:

Linux:

Solaris:

where E# is the enclosure ID number identified in step 2a, and S# is the slot number of the disk identified in step 2b. In the example above, the command would be:

OBTAIN CUSTOMER ACCEPTANCE
 WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

1. Verify the disk is brought online into a volume by LSI MegaRAID. Until the disk is added into a volume, the OS will not be able to use the disk.

If the OS is Linux, depending on the volume arrangement and image version, the disk may automatically become the new hotspare disk, or it may stay in an Unconfigured(good) state until the hotspare rebuild has completed.  If it stays Unconfigured then the hotspare will copy back to rebuild on the new disk after the rebuild has completed. If it is a RAID1 then it should automatically come into the volume and start rebuilding.

If the OS is Solaris, it is a Solaris RAID0 volume, and may not come into a volume automatically and will be in state Unconfigured(good) until it is in a volume.

Use the following to verify the physical disk is in one of these expected states – Hotspare, Unconfigured(good), Copyback, or Online:

where E# is the enclosure ID number identified in step 2a of the replacement steps, and S# is the slot number of the disk replaced. In the example above, the command and output would be:

2 Verify the replacement disk has been added to the expected RAID volume.

If the OS is running Linux and the failed disk was originally the global hotspare, then the replacement should have become the hotspare automatically, identified in step 1, and this step should be skipped. If that did not occur automatically, then the new disk can be assigned as the hotspare with the following command:

where E# is the enclosure ID number identified in step 2a of the replacement steps, and S# is the slot number of the disk replaced.

If the OS is running Linux and the failed disk was part of a RAID volume, use the following MegaRAID command to verify the status of the RAID:

If it has already completed the copyback when checked, then it may already be in “Online” state. If it is in rebuilding or copyback state, you can use the following to verify progress to completion:

where E# is the enclosure ID number identified in step 2a of the replacement steps, and S# is the slot number of the disk in Rebuild state. This is typically the original Hotspare disk slot.

or

where E# is the enclosure ID number identified in step 2a of the replacement steps, and S# is the slot number of the disk in Copyback state. This is typically the replaced disk slot.

If the OS is running Solaris, the RAID0 MegaRAID volume may need to be recreated, if it was not done so automatically. In this example the rpool mirror disk in slot 3 was failed:

Use format to partition the disk with a full-disk Solaris label, single cylinder boot block on slice 8, and the rest of the disk as root partition on slice 0.

Re-attach the new disk to the zpool. Use -f option if this is a mounted root pool:

If the attach fails on X2-2 nodes, it may be because 540-7869 was substituted with 542-0388 and vice versa. These parts are not compatible and have different cylinder counts. <Note 1370699.1> provides a workaround for that.

If this was one of the 2 boot disks in the root pool, then re-enable booting:

Verify status of the zpool rebuilding:

PARTS NOTE:

Refer to the Exadata Database Machine Owner's Guide Appendix C for part information.

How to identify which Exadata disk FRU part number to order , based on image and vendor and mixed disk support status - Note 1416303.1
Oracle Exadata V2 - Full Components List (https://support.us.oracle.com/handbook_internal/Systems/Exadata_V2/component.disks.html)
Oracle Exadata X2-2 - Full Components List (https://support.us.oracle.com/handbook_internal/Systems/Exadata_X2_2/component.disks.html)
Oracle Exadata X2-8 - Full Components List (https://support.us.oracle.com/handbook_internal/Systems/Exadata_X2_8/component.disks.html)
Oracle Exadata X3-2 - Full Components List (https://support.us.oracle.com/handbook_internal/Systems/Exadata_X3_2/component.disks.html)
Oracle Exadata X3-8 - Full Components List (https://support.us.oracle.com/handbook_internal/Systems/Exadata_X3_8/component.disks.html)
Oracle Exadata X4-2 - Full Component List (https://mosemp.us.oracle.com/handbook_internal/Systems/Exadata_X4_2_Computenode/components.html#Disks)

REFERENCE INFORMATION:

• Exadata Database Machine Documentation:
  
  • Exadata Database Machine Owner's Guide is available on the Storage Server OS image in /opt/oracle/cell/doc/welcome.html
    
  • Exadata Database Machine Maintenance Guide - https://docs.oracle.com/cd/E80920_01/index.htm
• Sun Fire X4170, X4270, and X4275 Servers Service Manual (Servicing Customer-Replaceable Devices) https://docs.oracle.com/cd/E19477-01/820-5830-13/hotswap.html#50634786_83028
• Sun Fire X4270 M2 Server Product Library Documentation (includes Sun Fire X4270 M2 Server Service Manual) https://docs.oracle.com/cd/E19245-01/index.html
• Oracle Sun Server X3-2 Product Library Documentation (includes Sun Server X3-2 Service Manual) https://docs.oracle.com/cd/E22368_01/index.html
• Oracle Sun Server X4-2 Product Library Documentation (includes Sun Server X4-2 Service Manual) https://docs.oracle.com/cd/E36975_01/index.html
• HALRT-02007: Database node hard disk failure (Doc ID 1113034.1)
• HALRT-02008: Database node hard disk failure (Doc ID 1113014.1)
• Oracle Sun Database Machine V2 Diagnosability and Troubleshooting Best Practices (Doc ID 1071220.1)
• Oracle Sun Database Machine X2-2/X2-8 Diagnosability and Troubleshooting Best Practices (Doc ID 1274324.1)
  • Bare Metal Restore Procedure for Compute Nodes on an Exadata Environment (Doc ID 1084360.1)
  • Determining when Disks should be replaced on Oracle Exadata Database Machine (Doc ID 1452325.1)

Internal Only References:
  - INTERNAL Exadata Database Machine Hardware Current Product Issues - V2/X2-2/X2-8 (Doc ID 1360343.1)
  - INTERNAL Exadata Database Machine Hardware Current Product Issues - X3-2/X3-8 (Doc ID 1501450.1)
  - INTERNAL Exadata Database Machine Hardware Troubleshooting (Doc ID 1360360.1)

References

This solution has no attachment