Oracle SuperCluster T5-8 Full Rack
 
Oracle SuperCluster M7 Hardware
 
Exadata SL6 Hardware
 
Exadata X6-2 Hardware
 
Exadata X6-8 Hardware
 
Oracle SuperCluster T5-8 Half Rack
 
Exadata X5-2 Hardware
 
Exadata X5-2 Eighth Rack
 
Exadata X5-2 Full Rack
 
Exadata X5-2 Quarter Rack
 
Exadata X4-8 Hardware
 
Exadata X5-2 Half Rack
 
Oracle SuperCluster T5-8 Hardware
 
Oracle SuperCluster M6-32 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

Applies to:

Goal

  Procedure for how to replace an NVMe drive in an Exadata Storage Cell without loss of data or Exadata service

Solution

The following information will be required prior to dispatch of a replacement:

    Name/location of storage cell
    Slot number of failed drive

Special Instructions for Dispatch are required for this part.

For Attention of Dispatcher:

The parts required in this action plan may be available as spares owned by the customer, which they received with the Engineered System. (These are sometimes referred to as ride-along spares.)

If parts are not available to meet the customer preferred delivery time/planned end date, then request TAM or field manager to contact the customer, and ask if the customer has parts available, and would be prepared to use them.

If customer spare parts are used, inform the customer that Oracle will replenish the customer part stock as soon as we can. More details on this process can be found in GDMR procedure "Handling Where No Parts Available" step 2: https://ptp.oraclecorp.com/pls/apex/f?p=151:138:38504529393::::DN,BRNID,DP,P138_DLID:2,86687,4,9082,

WHAT SKILLS DOES THE ENGINEER NEED:
Have familiarity with the Exadata Storage Servers and replacing hard drives.

TIME ESTIMATE: 60 minutes

Complete process may take longer depending on re-balance time that may be required.

TASK COMPLEXITY: 2


FIELD ENGINEER INSTRUCTIONS:

PROBLEM OVERVIEW:
Failed NVMe drive in Exadata Extreme Flash Storage Server.

NVMe drives are a combined controller and storage device and have very different failure modes compared to SAS devices. So the controller can report a Healthy Status and can also report failure code. If the controller believes the internal state of drive metadata could allow the drive to return incorrect data to the host, the drive will go into Disable Logical mode. This mode will shut down the drive storage device, but the controller will still be visible to the NVMe driver. This is also known as ASSERT or BAD_CONTEXT mode.


WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:
It is expected that the Exadata Machine is up and running and the storage cell containing the failed drive is booted and available.

If there are multiple drives to be replaced within an Exadata machine (or between an Exadata interconnected with another Exadata or Expansion Cabinet), it is critical that only ONE DRIVE BE REPLACED AT A TIME to avoid the risk of data loss. Before replacing another disk in Exadata, ensure the re-balance operation has completed from the first replacement.

Before proceeding, confirm the part number of the part in hand (either from logistics or an on-site spare) matches the part number dispatched for replacement.

It is expected that the customer's DBA has completed these steps prior to arriving to replace the disk. The following commands are provided as a guidance in case the customer needs assistance checking the system prior to replacement. If the customer or FSE requires more assistance prior to the physical replacement of the device, EEST/TSC should be contacted.


1. Confirm the drive needing replacement based on the output provided, the below shows nvmecli output from a drive that is in Disable Logical state (assert):

or check that the PCIe device is present using lspci | grep 0953 on X5 servers or lspci | grep 172X on X6 servers. Each NVMe device should appear once, there should be either 8 or 12 NVMe devices present depending on the customers configuration:

 
2. The Oracle ASM disks associated with the grid disks on the physical disk will be automatically dropped with FORCE option, and an ASM re-balance will start immediately to restore the data redundancy.

Validate the failed NVMe drive is no longer part of the ASM diskgroups:

   a) Login to a database node with the username for the owner of Oracle Grid Infrastructure home. Typically this is the 'oracle' user.

   b) Select the ASM instance for this DB node and connect to SQL Plus:

     
    In the above output the “1” of “+ASM1” refers to the DB node number. For example, DB node #3 the value would be +ASM3.

   c) From the DB node, run the following query, using the name the celldisk is associated with on this physical disk, which is given in the Cell alert, an example is below:

This query should return no rows indicating the disk is no longer in the ASM diskgroup configuration. If this returns any other value, then contact the SR owner for further guidance.

Note: If you are not sure what the celldisk name is, or do not have the alert output available, from the CellCLI interface run "list alerthistory"


3. The Cell Management Server daemon monitors and takes action on replacement disks to automatically bring the new disk into the configuration.

   a) Login to the cell server and enter the CellCLI interface

 b. Verify the status of the msStatus is running before replacing the disk:

4. If the failed NVMe drive is in slot 0 or slot 1, then the disk is a system disk which contains the running OS.  Verify the root volume is in 'clean' state before hot replacing a system disk. If it is 'active' and the disk is hot removed, the OS may crash making the recovery more difficult.

a. Login as 'root' on the Storage Cell, and use 'df' to determine the md device name for "/" volume:

b. Use 'mdadm' to determine the volume status:

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Confirm the drive needing replacement based on the output provided  ("name" or "slotNumber" value) and LED status of drive.  In order to remove the NVMe drive the PCIe hot-plug procedure MUST be followed. If the drive is removed without performing the hot-removal operation, system will crash and reset with a PCIe Surprise Link Down against the drive. There is a clear visual indication (Blue LED) when drive is safe to remove. If Blue LED is not lit, do not remove the drive.

Drives have both a physical slot location, and an instance in /dev, they may not be the same numerically. For example, physical slot 10, may be /dev/nvme7 depending on how many drives were populated at boot time. When preparing to gather data and logs from a drive, always check physical to logical mapping with the below command:

Name is the physical slot, deviceName is the /dev/nvme entry.

Slot locations for Extreme Flash Exadata Storage Cell:

View from the front:

8 Drive Configuration:
[Filler] [NVMe0] [Filler] [NVMe1] [Filler] [Filler] [Filler] [NVMe3] [Filler] [NVMe4] [Filler] [Filler] [Filler] [NVMe6] [Filler] [NVMe7] [Filler] [Filler] [Filler] [NVMe9] [Filler] [NVMe10] [Filler] [Filler]

1. To prepare a NVMe drive for removal, the following command MUST be run:

where NVME_# where # is the slot ID.

eg:

NOTE: The blue OK to Remove status indicator LED on the drive will light once a PCIe hot-remove operation has completed. Do not remove the drive until this LED indicator is lit, otherwise a system reset could occur.

2. On the drive you plan to remove, push the latch release button to open the drive latch

3. Grasp the latch and pull the drive out of the drive slot (Caution: whenever you remove a storage drive, you should replace it with another storage drive or a filler panel, otherwise the server might overheat due to improper airflow.)

4. Wait three minutes for the MS daemon to recognize the removal of the old drive

5. Slide the drive into the slot until the drive is fully seated

6. Close the drive latch to lock the drive in place

7. Drive should automatically power on when inserted. /var/log/messages will report a drive is present and identify the slot ID

8. Wait three minutes for the MS daemon to start rebuilding the virtual drives before proceeding


OBTAIN CUSTOMER ACCEPTANCE
- WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

It is expected that the engineer stay on-site until the customer has given the approval to depart.  The customer should check the status of the drive after replacement. The following commands are provided as guidance in case the customer needs assistance checking the status of the system following replacement. If the customer or the FSE requires more assistance following the physical replacement of the device, EEST/TSC should be contacted.

After replacing the NVMe drive on Exadata Storage Server, wait for three minutes before running any commands to query the device from the server.  CellCLI (examples below) should be the principle tool to query the drives.
 
1. Re-enable the NVMe drive after inserting it:

 
The "status" field should report "normal". Note also that the physicalInsertTime should be current date and time, and not an earlier time. If they are not, then the old disk entries may still be present and the disk replacement did not complete successfully. If this is the case, refer to the SR owner for further assistance.

2. The firmware of the drive will be automatically upgraded to match the other disks in the system when the new drive is inserted, if it is below the supported version of the current image. If it is above the minimum supported version then no action will be taken, and the newer firmware will remain. This can be validated by the following command:

3. After the drive is replaced, a lun should be automatically created, and the grid disks and cell disks that existed on the previous disk in that slot are automatically re-created on the new physical disk. If those grid disks were part of an Oracle ASM group, then they will be added back to the disk group and the data will be re-balanced on them, based on the disk group redundancy and asm_power_limit parameter values.

Grid disks and cell disks can be verified with the following CellCLI command, where the lun name is reported in the physicaldisk output from step 1 above ("0_7" in this example"):

Status should be normal for the cell disks and active for the grid disks. All of the creation times should also match the insertion time of the replacement disk.  If they are not, then the old disk entries may still be present and the disk replacement did not complete successfully. If this is the case, refer to the SR owner for further assistance.

Note: The lun name attribute will also be shown in the original alert generated by the storage cell.

4. To confirm that the status of the re-balance, connect to the ASM instance on a database node, and validate the disks were added back to the ASM diskgroups and a re-balance is running:

An active re-balance operation can be identified by STATE=RUN. The column group_number and inst_id provide the diskgroup number of the diskgroup been re-balanced and the instance number where the operation is running.  The re-balance operation is complete when the above query returns "no rows selected".

Validate the expected number of griddisks per failgroup and diskgroup.

The re-balance operation has completed when there are no "group_number" values of 0, and each disk group has count the same number of disks.

5. If the disk replaced was a system disk in slot 0 or 1, then the status of the OS volume should also be checked. Login as 'root' on the Storage cell and check the status using the same 'df' and 'mdadm' commands listed above:

While the system disk is rebuilding, the state will show as "active, degraded" or "active,degraded,recovering" with one indicating it is rebuilding and the 3rd being the 'faulty' disk. After rebuild has started, re-running this command will give a "Rebuild Status: X% complete" line in the output. When the system disk sync status is complete, the state should return to "clean" only with 2 devices.

If the status of any of the above checks (firmware, grid disk / cell disk creation, re-balance) is not successful, re-engage Oracle Support to get the correct action plan to manually complete the required steps.


PARTS NOTE:

Refer to the Exadata Database Machine Owner's Guide Appendix D for part information.

REFERENCE INFORMATION:

    Exadata Database Machine Documentation:
        Exadata Database Machine Owner's Guide is available on the Storage Server OS image in /opt/oracle/cell/doc/welcome.html
         http://amomv0115.us.oracle.com/archive/cd_ns/E13877_01/welcome.html
    Oracle Exadata Storage Server X5-2 Extreme Flash Service Manual
    Mirror partitions not resynced after replacing failed system drive (lun 0 or 1) (Doc ID 1316829.1)


Internal Only References:
  - Replacing a physicaldisk on a storage cell , cellcli list physicaldisk reports two entries on same slot but LUN is not created (Doc ID 1352938.1)
  - Exadata Documentation - http://amomv0115.us.oracle.com/archive/cd_ns/E50790_01/doc/doc.121/e51951/storage.htm#DBMMN21046

This solution has no attachment