Exadata X4-2 Hardware
 
SPARC SuperCluster T4-4 Full Rack
 
Exadata X3-2 Hardware
 
Oracle SuperCluster T5-8 Full Rack
 
Exadata Database Machine X2-2 Qtr Rack
 
Oracle SuperCluster M7 Hardware
 
Exadata SL6 Hardware
 
Zero Data Loss Recovery Appliance X6 Hardware
 
Exadata X7-2 Hardware
 
Exadata X5-2 Hardware
 
Exadata X6-2 Hardware
 
Exadata X6-8 Hardware
 
Oracle SuperCluster T5-8 Half Rack
 
Exadata X3-2 Half Rack
 
Exadata X5-2 Full Rack
 
Exadata X5-2 Eighth Rack
 
Exadata X4-2 Quarter Rack
 
Exadata Database Machine X2-2 Full Rack
 
Exadata Database Machine X2-8
 
Exadata X3-2 Full Rack
 
Exadata X4-2 Half Rack
 
Exadata X5-2 Quarter Rack
 
Exadata X4-8 Hardware
 
Exadata X3-8 Hardware
 
Zero Data Loss Recovery Appliance X7 Hardware
 
Zero Data Loss Recovery Appliance X5 Hardware
 
Exadata Database Machine X2-2 Half Rack
 
Oracle Exadata Storage Server Software
 
Zero Data Loss Recovery Appliance X4 Hardware
 
Exadata X5-2 Half Rack
 
Oracle SuperCluster M8 Hardware
 
Exadata Database Machine X2-2 Hardware
 
Exadata Cloud at Customer X6-2 Hardware
 
SPARC SuperCluster T4-4 Half Rack
 
Exadata X4-2 Full Rack
 
Exadata X4-2 Eighth Rack
 
SPARC SuperCluster T4-4
 
Oracle SuperCluster T5-8 Hardware
 
Exadata X3-8b Hardware
 
Exadata Database Machine V2
 
Exadata X3-2 Quarter Rack
 
Oracle SuperCluster M6-32 Hardware
 
Exadata X7-8 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 
_Old GCS Categories>ST>Server>Engineered Systems>Exadata>Hardware
 
_Old GCS Categories>Sun Microsystems>Specialized Systems>Database Systems
 

Applies to:

Goal

How to Replace a Hard Drive in an Exadata Storage Server (Cell) (Hard Failure)
(For Predictive Failure, please see Doc ID 1390836.1)

Solution

DISPATCH INSTRUCTIONS:
The following information will be required prior to dispatch of a replacement:

• Exadata Model (V2 or X2 or X3 etc.) / Exadata Storage Expansion Rack (X2 or X3 etc.)
• Type of storage cell/Node (V2= x4275 / X2 = x4270m2 / X3 = X3-2L / X4 = X4-2L / X5 = X5-2L / X6=X6-2L / X7 = X7-2L)
• Size of failed drive and part number
• Name/location of storage cell
• Slot number of failed drive
• Image Version (output of "/opt/oracle.cellos/imageinfo -all")

Special Instructions for Dispatch are required for this part.

For Attention of Dispatcher:

The parts required in this action plan may be available as spares owned by the customer, which they received with the Engineered System. (These are sometimes referred to as ride-along spares.)

If parts are not available to meet the customer preferred delivery time/planned end date, then request TAM or field manager to contact the customer, and ask if the customer has parts available, and would be prepared to use them.

If customer spare parts are used, inform the customer that Oracle will replenish the customer part stock as soon as we can. More details on this process can be found in GDMR procedure "Handling Where No Parts Available" step 2: https://ptp.oraclecorp.com/pls/apex/f?p=151:138:38504529393::::DN,BRNID,DP,P138_DLID:2,86687,4,9082,

WHAT SKILLS DOES THE ENGINEER NEED:
Have familiarity with the Exadata Storage Servers and replacing hard drives.

TIME ESTIMATE: 60 minutes

Complete process may take longer depending on re-balance time that may be required.

TASK COMPLEXITY: 0
CRU-optional; default is FRU with Task Complexity: 2

FIELD ENGINEER INSTRUCTIONS:
PROBLEM OVERVIEW:
Failed hard drive in Exadata Storage Server.

This document is specific to hard drives in "critical failure" state, also known as hard failure. There are situations where a drive will be flagged at first as a predictive failure which means the disk may still be in use.  In such cases, please reference Doc ID 1390836.1 for replacement steps.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:
It is expected that the Exadata Machine is up and running and the storage cell containing the failed drive is booted and available.

If there are multiple drives to be replaced within an Exadata machine (or between an Exadata interconnected with another Exadata or Expansion Cabinet), it is critical that only ONE DRIVE BE REPLACED AT A TIME to avoid the risk of data loss. Before replacing another disk in Exadata, ensure the re-balance operation has completed from the first replacement.

Before proceeding, confirm the part number of the part in hand (either from logistics or an on-site spare) matches the part number dispatched for replacement (especially important in cases where the customer has multiple racks of different drive types/sizes). 

600GB Special Handling (Exadata Critical Alert EX32): Exadata Storage Servers in V2, X2-2 and X2-8 systems with 600GB high performance disk drives must run version 11.2.2.4.1 (released December 2011) or higher, prior to completing this procedure to replace the disk, or the replacement will not be accepted by the Exadata Storage Server software. See Doc ID 2199949.1 for details.

8TB Special Handling: Exadata Storage Servers in X2-2 and later with 8TB disk drives must run version 12.1.2.3.6 or 12.2.1.1.2 (released July 2017) or higher, prior to completing this procedure to replace the disk if the replacement part is 7337414, or the replacement may not be accepted by the Exadata Storage Server software. If the image is earlier, then only use part 7301588. For Exadata/ZDLRA refer to Doc ID 2352138.1 for details.  For SuperCluster refer to Doc ID 2376948.1 for details.

It is expected that the customer's DBA has completed these steps prior to arriving to replace the disk. The following commands are provided as guidance in case the customer needs assistance checking the system prior to replacement. If the customer or FSE requires more assistance prior to the physical replacement of the device, EEST/TSC should be contacted.

1. Confirm the drive needing replacement based on the output provided  ("name" or "slotNumber" value) and LED status of drive.  For a hard failure, the LED for the failed drive should have the "Service Action Required" amber LED illuminated/flashing.  It should also have the "OK to Remove" blue LED illuminated/flashing, but may not depending on the nature of the failure mode and when it occurred.

For example, follow Doc ID 1113013.1 to determine the failed drive.

For "imageinfo" versions 11.2.3.1.x and earlier, use this syntax:

CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status=critical DETAIL
name: 28:5
deviceId: 21
diskType: HardDisk
...
luns: 0_5
makeModel: "SEAGATE ST360057SSUN600G"
physicalFirmware: 0705
physicalInterface: sas
physicalSerial: E07KZ8
physicalSize: 558.9109999993816G
slotNumber: 5
status: critical

For "imageinfo" versions 11.2.3.2.x and later, use this syntax:

CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status like failed DETAIL

In the output above, both the "name:" value (following the ":") and the "slotNumber" provide the slot of the physical device requiring replacement where the "status" field is "critical" status. In the above example, the slot is determined to be slot 5.  (slotNumber: 5 AND name: 28:5)

2. The Oracle ASM disks associated with the grid disks on the physical disk will be automatically dropped with FORCE option, and an ASM re-balance will start immediately to restore the data redundancy. Due to being "critical", there is no need to check that ASM is still re-balancing.

Validate the disk that was marked "critical" is no longer part of the ASM diskgroups:

a. Login to a database node with the username for the owner of Oracle Grid Infrastructure home. Typically this is the 'oracle' user.

edx2db01 login: oracle
Password:
Last login: Thu Jul 12 14:43:10 on ttyS0
[oracle@edx2db01 ~]$

b. Select the ASM instance for this DB node and connect to SQL Plus:

[oracle@edx2db01 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[oracle@edx2db01 ~]$ sqlplus ' / as sysasm'

SQL*Plus: Release 11.2.0.2.0 Production on Thu Jul 12 14:45:20 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>

 

In the above output the “1” of “+ASM1” refers to the DB node number. For example, DB node #3 the value would be +ASM3.

c. From the DB node, run the following query, using the name the celldisk associated with this physical disk, which is given in the Cell alert:

 This query should return no rows indicating the disk is no longer in the ASM diskgroup configuration. If this returns any other value, then contact the SR owner for further guidance.

Note: If you are not sure what the celldisk name is, or do not have the alert output available, from the CellCLI interface run "list alerthistory"

3. The Cell Management Server daemon monitors and takes action on replacement disks to automatically bring the new disk into the configuration.

a. Login to the cell server and enter the CellCLI interface

 b. Verify the status of the msStatusis running before replacing the disk:

CellCLI> list cell attributes cellsrvStatus,msStatus,rsStatus detail
cellsrvStatus: running
msStatus: running
rsStatus: running

4. If the failed disk is in slot 0 or slot 1, then the disk is a system disk which contains the running OS. The root volume should be verified as ready to remove.

a. Login as 'root' on the Storage Cell, and use 'df' to determine the md device name for "/" volume:

Verify the root volume is in 'State: clean,degraded' before hot replacing a system disk. If it is 'State: active' or 'State: clean' then the disk is not yet ready to be removed.
The faulted disk should show as 'removed'. If it shows 2 disks as 'active sync' or 'active' then the failed disk is not yet ready to be hot removed.
If the disk is hot removed when it is not ready, the OS may crash making the recovery more difficult. See Note 1524329.1 for steps to resolve this.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Confirm the drive needing replacement based on the output provided  ("name" or "slotNumber" value) and LED status of drive.

For a hard failure, the LED for the failed drive should have the "Service Action Required" amber LED illuminated/flashing, and may have the "OK to Remove" blue LED illuminated/flashing depending on the nature of the failure mode and when the failure occured. The cell server within the rack should also have its LOCATE white LED illuminated/flashing.

Perform the physical replacement of the disk following the directions from the service manual of the respective server (see REFERENCE INFORMATION below):

Slot locations for Exadata Storage Servers based on Sun Fire X4275 and Sun Fire X4270M2 servers:
View from the front:

Slot locations for Exadata Storage Servers based on Sun Server X3-2L/X4-2L and Oracle Server X5-2L/X6-2L/X7-2L servers:
View from the front:

1. On the drive you plan to remove, push the storage drive release button to open the latch.

2. Grasp the latch and pull the drive out of the drive slot (Caution: The latch is not an ejector. Do not bend it too far to the right. Doing so can damage the latch. Also, whenever you remove a storage drive, you should replace it with another storage drive or a filler panel, otherwise the server might overheat due to improper airflow.)

3. Wait three minutes for the MS daemon to recognize the removal of the old drive.

4. Slide the new drive into the drive slot until it is fully seated.

5. Close the latch to lock the drive in place.

6. Verify the "OK/Activity" Green LED begins to flicker as the system recognizes the new drive.  The other two LEDs for the drive should no longer be illuminated.

7. Wait three minutes for the MS daemon to start rebuilding the virtual drives before proceeding. Note: Do not run any controller commands in the service manual when replacing the disk.

8. The server's locate and disk's service LED locate blinking function should automatically turn off during the steps to return to an operational state.

OBTAIN CUSTOMER ACCEPTANCE
- WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

It is expected that the engineer stay on-site until the customer has given the approval to depart.  After the drive has been physically replaced, the system will automatically perform required actions to bring the drive active within ASM and mdadm if applicable.  The customer should check the status of the drive after replacement. The following commands are provided as guidance in case the customer needs assistance checking the status of the system following replacement. If the customer or the FSE requires more assistance following the physical replacement of the device, EEST/TSC should be contacted.

After replacing the physical disk on Exadata Storage Server, wait for three minutes before running any commands to query the device from the server.  CellCLI (examples below) should be the principle tool to query the drives.  If that is unsuccessful you can use "lsscsi" and "/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0" to verify the drive from an OS perspective.
 
1. When you replace a physical disk, the disk must be acknowledged by the RAID controller before the rest of the system can access it. Login to the cell server and enter the CellCLI interface, and run the following command, where <ID> is the "name" value provided in the action plan:

CellCLI> LIST PHYSICALDISK <ID> detail

CellCLI> list physicaldisk 28:5 detail
         name:                   28:5
         deviceId:               11
         diskType:               HardDisk
         ...
         makeModel:              "SEAGATE ST360057SSUN600G"
         physicalFirmware:       0A25
         physicalInsertTime:     2012-07-23T20:02:31-04:00
         physicalInterface:      sas
         physicalSerial:         E02LZ1
         physicalSize:           558.9109999993816G
         slotNumber:             5
         status:                 normal

The "status" field should report "normal". Note also that the physicalInsertTime should be current date and time, and not an earlier time. If they are not, then the old disk entries may still be present and the disk replacement did not complete successfully. If this is the case, refer to the SR owner for further assistance.using/substituting the "name" value provided in the action plan

2. The firmware of the drive will be automatically upgraded to match the other disks in the system when the new drive is inserted, if it is below the supported version of the current image. If it is above the minimum supported version then no action will be taken, and the newer firmware will remain. This can be validated by the following command:

CellCLI> alter cell validate configuration

3. After the physical disk is replaced, a lun should be automatically created, and the grid disks and cell disks that existed on the previous disk in that slot are automatically re-created on the new physical disk. If those grid disks were part of an Oracle ASM group, then they will be added back to the disk group and the data will be re-balanced on them, based on the disk group redundancy and asm_power_limit parameter values.

Grid disks and cell disks can be verified with the following CellCLI command, where the lun name is reported in the physicaldisk output from step 1 above ("0_5" in this example"):

CellCLI> list lun 0_5 detail
         name:                   0_5
         cellDisk:               CD_05_edx2cel02
         deviceName:             /dev/sdad
         diskType:               HardDisk
         id:                     0_5
         isSystemLun:            FALSE
         lunAutoCreate:          FALSE
         lunSize:                558.40625G
         lunUID:                 0_5
         physicalDrives:         28:5
         raidLevel:              0
         lunWriteCacheMode:      "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"
         status:                 normal

CellCLI> list celldisk where lun=0_5 detail
         name:                   CD_05_edx2cel02
         comment:
         creationTime:           2012-07-23T20:03:34-04:00
         ...
         lun:                    0_5
         raidLevel:              0
         size:                   558.40625G
         status:                 normal

CellCLI> list griddisk where celldisk=CD_05_edx2cel02 detail
         name:                   DATA_Q1_CD_05_edx2cel02
         availableTo:
         cellDisk:               CD_05_edx2cel02
         ...
         status:                 active

         name:                   DBFS_DG_CD_05_edx2cel02
         availableTo:
         cellDisk:               CD_05_edx2cel02
         ...
         status:                 active

         name:                   RECO_Q1_CD_05_edx2cel02
         availableTo:
         cellDisk:               CD_05_edx2cel02
         ...
         status:                 active

Status should be normal for the cell disks and active for the grid disks. All of the creation times should also match the insertion time of the replacement disk.  If they are not, then the old disk entries may still be present and the disk replacement did not complete successfully. If this is the case, refer to the SR owner for further assistance.

Note: The lun name attribute will also be shown in the original alert generated by the storage cell.

4. To confirm that the status of the re-balance, connect to the ASM instance on a database node, and validate the disks were added back to the ASM diskgroups and a re-balance is running:

SQL> set linesize 132
SQL> col path format a50
SQL> select group_number,path,header_status,mount_status,name from V$ASM_DISK where path like '%CD_05_edx2cel02';

GROUP_NUMBER PATH                                               HEADER_STATU MOUNT_S NAME
------------ -------------------------------------------------- ------------ ------- ------------------------------
           1 o/192.168.9.10/DATA_Q1_CD_05_edx2cel02       MEMBER       CACHED  DATA_Q1_CD_05_edx2CEL02
           2 o/192.168.9.10/DBFS_DG_CD_05_edx2cel02       MEMBER       CACHED  DBFS_DG_CD_05_edx2CEL02
           3 o/192.168.9.10/RECO_Q1_CD_05_edx2cel02       MEMBER       CACHED  RECO_Q1_CD_05_edx2CEL02


SQL> select * from gv$asm_operation;

   INST_ID GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK  EST_RATE
---------- ------------ ----- ---- ---------- ---------- ---------- ----------  ----------
EST_MINUTES ERROR_CODE
----------- --------------------------------------------
         2            3 REBAL WAIT         10


         1            3 REBAL RUN          10         10       1541       2422
      7298           0

An active re-balance operation can be identified by STATE=RUN. The column group_number and inst_id provide the diskgroup number of the diskgroup been re-balanced and the instance number where the operation is running.  The re-balance operation is complete when the above query returns "no rows selected".

Validate the expected number of griddisks per failgroup and diskgroup.  Normal deployment includes 12 griddisks for data, 12 for reco and 10 for dbfs_dg. (MODE_STATUS = ONLINE or MOUNT_STATUS=CACHED) (via SQL> )

SQL> select group_number,failgroup,mode_status,count(*) from v$asm_disk group by group_number,failgroup,mode_status;

 

The re-balance operation has completed when there are no "group_number" values of 0, and each disk group has count the same number of disks.

If the new griddisks were not automatically added back into the ASM diskgroup configuration, then locate the disks with group_number=0, and add them back in manually using "alter diskgroup <name> add disk <path> re-balance power 10;" command:

Repeat the prior queries to validate the re-balance has started and there are no longer any disks with "group_number" values of 0.

5. If the disk replaced was a system disk in slot 0 or 1, then the status of the OS volume should also be checked. Login as 'root' on the Storage cell and check the status using the same 'df' and 'mdadm' commands listed above:

While the system disk is rebuilding, the state will show as "active, degraded" or "active,degraded,recovering" with one indicating it is rebuilding and the 3rd being the 'faulty' disk. After rebuild has started, re-running this command will give a "Rebuild Status: X% complete" line in the output. When the system disk sync status is complete, the state should return to "clean" only with 2 devices. If an extra entry (faulty spare) is seen, this can be ignored - refer to Doc ID 2031054.1 for details.

If the status of any of the above checks (firmware, grid disk / cell disk creation, re-balance) is not successful or the disk fault LED does not turn off, re-engage Oracle Support to get the correct action plan to manually complete the required steps.


PARTS NOTE:

Refer to the Exadata Database Machine Owner's Guide Appendix C for part information.

REFERENCE INFORMATION:

• HALRT-02003: Data hard disk failure: (DocID 1113013.1)
• Exadata Database Machine Documentation: 
  • Exadata Database Machine Maintenance Guide is available on the Storage Server OS image in /opt/oracle/cell/doc/welcome.html
  • https://docs.oracle.com/cd/E80920_01/DBMMN/toc.htm
• Sun Fire X4170, X4270, and X4275 Servers Service Manual (Servicing Customer-Replaceable Devices) https://docs.oracle.com/cd/E19477-01/820-5830-13/hotswap.html#50634786_83028
• Sun Fire X4270 M2 Server Product Library Documentation (includes Sun Fire X4270 M2 Server Service Manual) https://docs.oracle.com/cd/E19245-01/index.html
• Sun Server X3-2L Documentation Library https://docs.oracle.com/cd/E23393_01/index.html
• Sun Server X4-2L Documentation Library https://docs.oracle.com/cd/E36974_01/index.html
• Oracle Server X5-2L Documentation Library https://docs.oracle.com/cd/E41033_01/html/E48325/index.html
• Oracle Server X6-2L Documentation Library https://docs.oracle.com/cd/E62172_01/html/E62184/index.html
• Oracle Server X7-2L Documentation Library https://docs.oracle.com/cd/E72463_01/html/E72479/index.html
• Oracle Sun Database Machine V2 Diagnosability and Troubleshooting Best Practices (Doc ID 1071220.1)
• Oracle Sun Database Machine X2-2/X2-8 Diagnosability and Troubleshooting Best Practices (Doc ID 1274324.1)
• Steps to manually create cell/grid disks on Exadata V2 if auto-create fails during disk replacement (Doc ID 1281395.1)

Internal Only References:
  - Replacing a physicaldisk on a storage cell , cellcli list physicaldisk reports two entries on same slot but LUN is not created (Doc ID 1352938.1)

Mirror partitions not resynched after replacing failed system drive (lun 0 or 1) (Doc ID 1316829.1)

References

This solution has no attachment