Oracle SuperCluster T5-8 Full Rack
 
Oracle SuperCluster M7 Hardware
 
Exadata SL6 Hardware
 
Zero Data Loss Recovery Appliance X6 Hardware
 
Oracle SuperCluster T5-8 Half Rack
 
Exadata X5-2 Eighth Rack
 
Exadata X5-2 Hardware
 
Exadata X5-2 Full Rack
 
Exadata X6-8 Hardware
 
Exadata X6-2 Hardware
 
Exadata X4-8 Hardware
 
Exadata X5-2 Quarter Rack
 
Zero Data Loss Recovery Appliance X5 Hardware
 
Exadata X5-2 Half Rack
 
Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

Applies to:

Goal

 How to Replace an Exadata X5-2, X6-2 Storage Server Internal USB drive.

Solution

 DISPATCH INSTRUCTIONS
WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?: Exadata Trained

TIME ESTIMATE: 90 minutes

TASK COMPLEXITY: 3

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:
PROBLEM OVERVIEW: A faulty internal USB drive has been detected and needs to be replaced in an Exadata X5-2/X6-2 Storage cell.

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

The instructions below assume the customer DBA is available and working with the field engineer onsite to manage the host OS and DB/ASM services. They are provided here to allow the FE to have all the available steps needed when onsite, and can be done by the FE if the customer DBA wants or allows or needs help with their steps.

1. Locate the server in the rack being serviced.

Turn on the locate indicator light ‘on’ for easier identification of the server being repaired. If the server number has been identified then the Locate Button on the front panel may be pressed. To turn on remotely, use either of the following methods:

From a login to the CellCli:

 From a login to the server’s ILOM:

From a login to the server’s ‘root’ account:

2. Shutdown the node for which the USB stick requires replacement.

a. For Extended information on this section check MOS Note:
ID 1188080.1 Steps to shut down or reboot an Exadata storage cell without affecting ASM

This is also documented in the Exadata Database Maintenance Guide chapter 3 section titled "Maintaining Exadata Storage Servers" subsection "Shutting Down Exadata Storage Server" available on the customer's cell server image in the /opt/oracle/cell/doc directory.

Exadata Maintenance Guide Documentation is available internally here:

http://amomv0115.us.oracle.com/archive/cd_ns/E50790_01/doc/doc.121/e51951/storage.htm#DBMMN22021

In the following examples the SQL commands should be run by the Customers DBA prior to doing the hardware replacement. These should be done by the field engineer only if the customer directs them to, or is unable to do them. The cellcli commands will need to be run as root.

Note the following when powering off Exadata Storage Servers:

• Verify there are no other storage servers with disk faults. Shutting down a storage server while another disk is fails may result in the running database processes and Oracle ASM to crash if it loses both disks in the partner pair when this server’s disks go offline.
  
  
• Powering off one Exadata Storage Server with no disk faults in the rest of the rack will not affect running database processes or Oracle ASM.
  
  
• All database and Oracle Clusterware processes should be shut down prior to shutting down more than one Exadata Storage Server. Refer to the Exadata Owner’s Guide for details if this is necessary.

b. ASM drops a disk shortly after it/they are taken offline. Powering off or restarting Exadata Storage Servers can impact database performance if the storage server is offline for longer than the ASM disk repair timer to be restored. The default DISK_REPAIR_TIME attribute value of 3.6hrs should be adequate for replacing components, but may have been changed by the Customer. To check this parameter, have the Customer log into ASM and perform the following query:

As long as the value is large enough to comfortably replace the components being replaced, then there is no need to change it.

c. Check if ASM will be OK if the grid disks go OFFLINE.

If one or more disks return asmdeactivationoutcome='No', then wait for some time and repeat step #2. Once all disks return asmdeactivationoutcome='Yes', proceed to the next step.

d. Run cellcli command to Inactivate all grid disks on the cell that needs to be powered down for maintenance. (this could take up to 10 minutes or longer)

 e. Execute the command below and the output should show asmmodestatus='UNUSED' or 'OFFLINE' and asmdeactivationoutcome=Yes for all griddisks once the disks are offline and inactive in ASM.

f.Before shutting down make a note of the active image version

In this example it is 12.1.2.2.0 ,this will be required later if replacing the USB in an extreme flash storage cell.

g.For Extreme Flash systems ONLY which use NVME drives in place of hard-disk and the image is 12.1.2.2.0 or 12.1.2.2.1 it will be necessary to make the NVME devices bootable before shutting down.This step is not required for High capacity storage cells,for High capacity proceed to step "h"

Due to bug 22620662 EXTREME FLASH CELLS FAIL TO BOOT FROM THE NVME SYSTEM DISKS

This step is not necessary if the extreme flash image version is 12.1.2.1.x or 12.1.2.3.0 or higher.

i) For 12.1.2.2.0 or 12.1.2.2.1 ONLY type the following:

 This will create the file device.map which will be used and will be kept for reference if required.

ii).Make the NVME devices bootable.Type the following:

This is an example of the expected output:

If the above command fails with "failed Error 22t: No such partition" then continue to shutdown the cell as the device can still be replaced and made bootable using the diagnostics iso image.See later instructions in this action plan.

h. Once all disks are offline and inactive, the customer may shutdown the Cell using the following command:

When powering off Exadata Storage Servers, all storage services are automatically stopped.

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

1. Slide out the server for maintenance. Do not remove any cables prior to sliding the server forward, or the loose cable ends will jam in the cable management arms. Take care to ensure the cables and Cable Management Arm is moving properly. Refer to Note 1444683.1 for CMA handling training.

2. Remove the AC power cords prior to removing the server’s top cover.

For High Capacity Storage cell with a single internal USB drive follow "High Capacity configuration".
For Extreme Flash storage cell with two internal USB drives follow the later section "Extreme Flash Configuration"

High Capacity Configuration

1. Remove and replace the USB thumb drive from the internal USB port.Make a note of which slot the USB drive is inserted into ,there are two USB slots.

   On Exadata Storage Server based on a Oracle Server X5-2L, the internal USB ports are located near the handle on the Rear I/O daughter board located between PCIe slots 3 and 4.

2. Replace the server’s top cover and re-attach the AC power cords. ILOM will take up to 2 minutes to boot.

3. Slide the server back into the rack.

4. After ILOM has booted, power on the server by pressing the power button, and then connect to the server’s console. 

 To connect to the console through ILOM:

- From the ILOM CLI:

5. From the console and monitor the system booting. The server should boot from the primary hard disk. This will be mentioned in the Exadata splash screen.

6. After the Storage Server is booted, login as ‘root’ user.

7. Run the following to copy the recovery image and configuration data to the new USB stick:

8. Set the next boot to forcibly stop at the BIOS setup menu:

 9. Reboot the server with the following command:

 10. Monitor the system booting again. The system should go automatically into the BIOS Setup screen.

11. Once the BIOS Setup screen is displayed on the console, use the arrow keys to navigate to the Boot screen .Check the "Legacy Boot Option Priority" list . Set the "USB:USBIN0:ORACLE SSM PMAP" to be the first boot device , followed by “PCI RAID Adapter” followed by the onboard network PXE devices. Press “Esc” to exit the ‘Boot Order Device Priority’ screen

Refer to X5 Series Servers Administration Guide for details.

12. Navigate to the Exit screen and select “Save Changes and Exit”

13. The server will boot . This time it should load the Exadata splash screen (grub) from the USB stick and indicate as such.

Extreme Flash (EF) Configuration

With the introduction of image 12.1.2.2.0 and higher only one USB drive /dev/sda " USB:USBIN0:ORACLE SSM PMAP"  (this is the lower USB slot ), is used for the grub boot-loader .USBIN1 is no longer mirrored or contains the grub boot loader.In addition it is now possible with 12.1.2.2.0 and higher to boot from the NVME devices.

EF Replacement when image is 12.1.2.1.x

1. Remove and replace the USB thumb drive from the internal USB port.There are two USB slots , drive /dev/sda is "USB:USBIN0:ORACLE SSM PMAP" this is the lower USB slot .Drive /dev/sdb is "USB:USBIN1:ORACLE SSM PMAP" this is the upper slot.

    On Exadata Storage Server based on a Oracle Server X5-2L, the internal USB ports are located near the handle on the Rear I/O daughter board located between PCIe slots 3 and 4.

2. Replace the server’s top cover and re-attach the AC power cords. ILOM will take up to 2 minutes to boot.

3. Slide the server back into the rack.

4. After ILOM has booted,login to the ILOM and connect to the console.

 To connect to the console through ILOM:

- From the ILOM CLI:

 5. From the console monitor the system booting.It will boot from the good USB drive.

6. After the Storage Server is booted, login as ‘root’ user,this will be the first session.Then with a second login session ssh to the storage cell as user root,this will be the second session.

The cell will now automatically rebuild the new USB drive ,this is performed by the actions of the "checkdeveachboot" validation script. To view the activity of the rebuild use the second login session and issue the following command.

Note, you may need to wait at least 5 minutes after login before this log file is created. Once it exists then use the following to watch the rebuild activity.

The activity will be started when  events such as the following are displayed in the log file

The script will show errors such as the following ,this is because the device files /dev/sdX1 and /dev/sdX2 ( where X= a or b ) have not yet been created.These will be created later by the script.

This example shows /dev/sdb1 missing when first checked.

Approximately 10 minutes after the validation script has started it will appear to have stopped ,the following will be displayed in the log file.The script will then sleep for at least 10 minutes before recommencing and again rebuilding the new device.

Further warnings are displayed

The script will eventually complete with the following event, this event will be displayed twice before the script halts.

At this point the mirror of the two USB devices, /dev/md4 will be recovering ,this will take approximately 35 minutes. Progress of the recovery can be viewed by examing the file /proc/mdstat by using the first console session

The recovery can also be viewed with mdadm

Wait for the recovery to complete ,it will show the following

7.Check the correct device files and label have been created

- ensure both drives are seen.

- ensure all device files exist

 Partition /dev/sdb1 may fail due to bug 20765279 ,it may show the following.

There are now two options for recovering the label on the drive .

a). Ensure the recovery of the mirror /dev/md4 has completed ,then reboot the cell .This action will recover the missing label.

When the cell has completed the reboot,login and again check the labels using the e2label command,all should now be correct.

or if the customer does not wish to reboot.

b).Follow this procedure

EF Replacement when using image 12.1.2.2.0 or higher

1. Remove and replace the USB thumb drive from the internal USB port. There are two USB slots , drive /dev/sda is "USB:USBIN0:ORACLE SSM PMAP" this is the lower USB slot .Drive /dev/sdb is "USB:USBIN1:ORACLE SSM PMAP" this is the upper slot.

USBIN1 is not used for this version.

On Exadata Storage Server based on a Oracle Server X5-2L, the internal USB ports are located near the handle on the Rear I/O daughter board located between PCIe slots 3 and 4.

 2. Replace the server’s top cover and re-attach the AC power cords. ILOM will take up to 2 minutes to boot.

3. Slide the server back into the rack.

 4. After ILOM has booted,login to the ILOM and connect to the console.

 To connect to the console through ILOM:

 Now power on the storage cell.As soon as the first BIOS display is visible press cntrl-p to select the boot pop-up menu .Eventually the boot menu will be displayed.

Select one of the NVME devices from the display to boot from ,see example below.

The cell will now boot.

If the cell fails to boot from the NVME device  , or it was not possible to make the NVME devices bootable prior to shutdown then refer to document "Exadata Extreme Flash storage cell fails to boot from NVME when using image 12.1.2.2.0 or 12.1.2.2.1 (Doc ID 2108452.1)"

6. After the Storage Server is booted, login as ‘root’ user.

7. Run the following to copy the recovery image and configuration data to the new USB stick:

It may be required to stop the MS service to run this command.

Remember to re-enable this once the make_cellboot_usb has completed.

8. Set the next boot to forcibly stop at the BIOS setup menu:

9. Reboot the server with the following command:

10. Monitor the system booting again. The system should go automatically into the BIOS Setup screen.

11. Once the BIOS Setup screen is displayed on the console, use the arrow keys to navigate to the Boot screen .Check the "Legacy Boot Option Priority" list . Set the "USB:USBIN0:ORACLE SSM PMAP" to be the first boot device , followed by “PCIE6:NVMe0:INTEL SSDPE2ME016T4S” then "PCIE6:NVMe1:INTEL SSDPE2ME016T4S" and finally "PXE:NET0:IBA XE Slot 3A00 v2320" . Press “Esc” to exit the ‘Boot Order Device Priority’ screen

Refer to X5 Series Servers Administration Guide for details.

12. Navigate to the Exit screen and select “Save Changes and Exit”

13. The server will boot . This time it should load the Exadata splash screen (grub) from the USB stick and indicate as such.

OBTAIN CUSTOMER ACCEPTANCE
WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

The following steps should be done by the customer's administrator to return the disks to service:

1. Activate the grid disks:

# cellcli

 2. Issue the command below and all disks should show 'active':

 3. Verify all grid disks have been successfully put online using the following command. Wait until 'asmmodestatus' is in status 'ONLINE' for all grid disks. The following is an example of the output early in the activation process.

It has been found on very rare occasions that the above procedure has not worked with image 12.1.2.1.0 and 12.1.2.1.1 , if this happens please refer to the doc and contact Exadata software support for assistance.

Exadata Storage Software 12.1.2.1.0 and 12.1.2.1.1 System Disk Replacement Issues (Doc ID 2003674.1)

PARTS NOTE:

7090170 - 8GB USB Stick

REFERENCE INFORMATION:

Oracle ILOM 3.2 documentation library - https://docs.oracle.com/cd/E37444_01/index.html

References

This solution has no attachment