Exadata X3-2 Hardware
 
Exadata X4-2 Hardware
 
Exadata X5-8 Hardware
 
Exadata X5-2 Full Rack
 
Exadata X5-2 Eighth Rack
 
Exadata X4-2 Quarter Rack
 
Exadata X5-2 Hardware
 
Exadata X3-2 Half Rack
 
Exadata X3-2 Full Rack
 
Exadata X5-2 Quarter Rack
 
Exadata X4-8 Hardware
 
Exadata X4-2 Half Rack
 
Exadata X3-8 Hardware
 
Exadata X5-2 Half Rack
 
Exadata X3-2 Eighth Rack
 
Exadata X4-2 Full Rack
 
Exadata X4-2 Eighth Rack
 
Exadata X3-8b Hardware
 
Exadata X3-2 Quarter Rack
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Disk cage replacement is a FRU procedure.

Applies to:

Goal

 How to remove and replace the Disk Cage on Exadata X3-2, X4-2, and X5-2 Storage Server to support 8TB disk upgrades.

Solution

CAP PROBLEM OVERVIEW: Disk cage upgrade procedure.

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE ENGINEER NEED:
Oracle Exadata X3-2, X4-2, and X5-2 Storage Server Training.

TIME ESTIMATE: 60 minutes, for X3/X4 up to 90 minutes

TASK COMPLEXITY: 2-FRU

FIELD ENGINEER INSTRUCTIONS:

PROBLEM OVERVIEW:

To support an 8TB disk drive upgrade an Exadata X3-2, X4-2, and X5-2 Storage CELL must have the disk cage replaced to support the physically larger 8TB drives.  Any X5-2 Storage CELL shipped with 8TB drives from the factory already has the correct disk cage present.  Only X5-2 Storage CELLs shipped with 4TB disk drives or smaller require the upgrade.  All X3-2 and X4-2 Storage CELLs require the upgraded disk cage.

Ensure the customer has the correct Upgrade Kit for the type of Storage CELL:

High Capacity Upgrade Kit - Twelve 8TB HDDs plus disk cage for X5 racks (Marketing #7113574)
High Capacity Upgrade Kit - Twelve 8TB HDDs plus disk cage for X3 and X4 racks (Marketing #7113575)

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

The Storage Cell containing the disk cage is required to be powered off prior to cage replacement.

It is expected that the customer's DBA has completed these steps prior to arriving to replace the cage. The following commands are provided as guidance in case the customer needs assistance checking the status of the system prior to replacement.  If the customer or the FSE requires more assistance prior to the physical replacement of the cage, EEST/TSC should be contacted.

1. Locate the server in the rack being serviced.  The cell server within the rack can be determined from the hostname usually, and the known default Exadata server numbering scheme. Exadata Storage Servers are identified by a number 1 through 18, where 1 is the lowest most Storage Server in the rack installed in RU2, counting up to the top of the rack.

Turn on the locate indicator light for easier identification of the server being repaired. If the server number has been identified then the Locate Button on the front panel may be pressed. To turn on remotely, use either of the following methods:

From a login to the CellCli on Exadata Storage Servers:

From a login to the server’s ILOM:

From a login to the server’s ‘root’ account:

2. Shutdown the node for which the disk cage requires replacement.

   a) For Extended information on this section check MOS Note: ID 1188080.1 Steps to shut down or reboot an Exadata storage cell without affecting ASM

This is also documented in the Exadata Maintenance Guide section titled "Maintaining Exadata Storage Servers" subsection "Shutting Down Exadata Storage Server" available on the customer's cell server image in the /opt/oracle/cell/doc directory or at https://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/storage.htm#DBMMN21129.


In the following examples the SQL commands should be run by the customers DBA prior to doing the hardware replacement. These should be done by the field engineer only if the customer directs them to, or is unable to do them. The cellcli commands will need to be run as root.

Note the following when powering off Exadata Storage Servers:

• Verify there are no other storage servers with disk faults. Shutting down a storage server while another disk is failed may result in the running database processes and Oracle ASM to crash if it loses both disks in the partner pair when this server’s disks go offline.

• Powering off one Exadata Storage Server with no disk faults in the rest of the rack will not affect running database processes or Oracle ASM.

   b) ASM drops a disk shortly after they are taken offline. Powering off or restarting Exadata Storage Servers can impact database performance if the storage server is offline for longer than the ASM disk repair timer to be restored. The default DISK_REPAIR_TIME attribute value of 3.6hrs may not be adequate for replacing components if something breaks, we recommend changing to 24hrs to give time to order and receive replacement parts if something breaks. To check this parameter, have the Customer log into ASM and perform the following query:

As long as the value is large enough to comfortably replace the components being replaced, then there is no need to change it.

   c) Check if ASM will be OK if the grid disks go OFFLINE. 

If one or more disks return asmdeactivationoutcome='No', then wait for some time and repeat this command. Once all disks return asmdeactivationoutcome='Yes', proceed to the next step.

   d) Run cellcli command to Inactivate all grid disks on the cell that needs to be powered down for maintenance. (this could take up to 10 minutes or longer)

   e) Execute the command below and the output should show asmmodestatus='UNUSED' or 'OFFLINE' and asmdeactivationoutcome=Yes for all griddisks once the disks are offline and inactive in ASM.

   f) Once all disks are offline and inactive, the customer may shutdown the Cell using the following command:

When powering off Exadata Storage Servers, all storage services are automatically stopped.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

Remove the existing Disk Cage:

1. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server into the maintenance position.
3. Attach an antistatic wrist strap to your wrist, and then to a metal area on the chassis.
4. Remove the server top cover.

2. Remove all the disk drives from the front of the server.

3. Remove the disk cage mounting screws to loosen the cage, making it easier to remove the cables.  Depending on your server type, do one of the following:

1. If your server is X5-2L, perform the following procedures:
   1. Remove the server fan assembly door.
   1. Remove the two screws on each side of the chassis and the three screws on top of the chassis.
   2. Slide the fan assembly door toward the rear of the server, and lift and remove the door from the chassis.
   2. Remove the four remaining No. 2 Phillips screws (two on each side of the chassis) that secure the disk cage assembly to the server chassis.
2. If your server is X4-2L or X3-2L, perform the following procedures:
1. Open the top cover fan assembly door.
2. Remove the six No. 2 Phillips screws (three on each side of the chassis) that secure the disk cage assembly to the server chassis.

4. Remove the fan modules from the server.

5. Disconnect all cables from the front disk backplane.

1. If your server is X4-2L or X3-2L, perform the following procedures:
1. Remove the mid-wall from the chassis.
1. Using a No. 2 Phillips screwdriver, remove the two screws on each side of the chassis that secure the mid-wall to the chassis.
2. Using a No. 2 Phillips screwdriver, loosen the four green captive screws that secure the mid-wall to the bottom of the chassis.
3. Lift the mid-wall out of the chassis.
2. This allows for easier access to the cables.
   
3. Use caution when disconnecting the signal cable between the disk backplane and motherboard, the cable is quite fragile. If upgrading more than 3 CELLs you may want to have a spare cable when performing the procedure.

6. Disconnect the left LED indicator module cable and the right LED/USB indicator module cable from the motherboard.

7. Slide the disk cage assembly forward, and then gently lift the disk cage assembly from the chassis.  Place the disk cage on an antistatic mat.

8. Remove the disk backplane from the disk cage.  Depending on your server type, do one of the following:

1. If your server is X5-2L, perform the following procedures:
1. Using a No. 2 Phillips screwdriver, loosen the right-side spring-mounted screw that secures the storage drive backplane to the disk cage.
   
2. Lift the storage drive backplane up to release it from the standoff hooks and off of the disk cage.
   
3. Place the storage drive backplane on an antistatic mat until it's ready to be installed in the new disk cage.
   
2. If your server is X4-2L or X3-2L, perform the following procedures:
1. Using a No. 2 Phillips screwdriver, loosen the two screws that secure the storage drive backplane to the disk cage.
2. Slide the backplane toward the front of the cage to release it from the three mushroom-shaped standoffs, and lift it off of the disk cage.
   
3. Place the storage drive backplane on an antistatic mat until it's ready to be installed in the new disk cage.
   
4. Repeat 1 through 3, to remove the 2nd and 3rd drive backplanes.  For the 2nd and 3rd disk backplane you may need to use a wrench (or multi-tool) to loosen the two screws that secure the storage drive backplane to the disk cage, as the "mounting bracket" of the 1st disk backplane makes access to the screws of the 2nd disk backplane difficult.
   

Install the new disk cage:

1. Before beginning the process of installing the new cage you must transfer serial number label identity from the old cage to the new cage.  To do this you must carefully peel the serial number labels from the RFID tag, and the top left front corner of the old disk cage face.  The top label can be seen when looking down over the top.  Once the labels are removed, carefully re-affix them to the same locations on the new disk cage.

2. Install the disk backplane from the old disk cage into the new disk cage. Depending on your server type, do one of the following:

1. If your server is X5-2L, perform the following procedures:
1. Lower the storage drive backplane onto the disk cage, and position it to engage the standoff hooks.
   
2. Using a No. 2 Phillips screwdriver, install and tighten the right-side spring-mounted screw to secure the storage drive backplane to the disk cage.
   
2. If your server is X4-2L or X3-2L, perform the following procedures:
1. Lower the storage drive backplane onto the disk cage, and position it to engage the three mushroom standoffs.
2. Using a No. 2 Phillips screwdriver, install and tighten the two screws to secure the storage drive backplane to the disk cage.
   
3. Repeat 1 and 2, to install the 2nd and 3rd drive backplanes.  For the 2nd and 3rd disk backplane you may need to use a wrench (or multi-tool) to tighten the two screws that secure the storage drive backplane to the disk cage, as the "mounting bracket" of the 1st disk backplane makes access to the screws of the 2nd disk backplane difficult.
   

3. Gently lift the disk cage assembly and set it into the server chassis. 

4. Reconnect the left LED indicator module cable and the right LED/USB indicator module cable to the motherboard.

5. Reconnect all cables to the front disk backplane.

1. If your server is X4-2L or X3-2L, perform the following procedures:
1. Install the mid-wall into the chassis.
1. Lift and place the mid-wall into the chassis.
2. Using a No. 2 Phillips screwdriver, tighten the four green captive screws that secure the mid-wall to the bottom of the chassis.
3. Using a No. 2 Phillips screwdriver, insert and tighten the two screws on each side of the chassis that secure the mid-wall to the chassis.
2. Use caution in reconnecting the signal cable between the disk backplane and motherboard, the cable is quite fragile. If upgrading more than 3 CELLs you may want to have a spare when performing the procedure.

6. Secure the cage to the chassis. Depending on your server type, do one of the following:

1. If your server is X5-2L, perform the following procedures:
   
   1. Install the four No. 2 Phillips screws (two on each side of the chassis) that secure the disk cage assembly to the server chassis.
   2. Install the server fan assembly door.
   1. Place the fan assembly door on the chassis and slightly over the fan assembly.
   2. Slide the fan assembly door forward and under the lip of the forward top cover until it latches into place.
   3. Use a No. 2 Phillips screwdriver to install and secure the fan assembly door.
   1. Install and tighten the two screws on each side of the chassis and the three screws on top of the chassis.
2. If your server is X4-2L or X3-2L, perform the following procedures:
1. Install the six No. 2 Phillips screws (three on each side of the chassis) that secure the disk cage assembly to the server chassis.

7. Install the fan modules in the server, and close the top cover fan assembly door.

8. Either re-install all the original HDDs back into the server, or upgrade to the new 8TB HDDs.

9. Return the server to operation:

1. Install the server top cover.
2. Reconnect the power cords to the power supplies, and power on the server.
3. Once the ILOM has booted you will see a slow blink on the green LED for the server. Power on the server by pressing the power button on the front of the unit.
   
   

Server Services Startup Validation:

1. As the system boots the hardware/firmware profile will be checked, and either a green "Passed" will be displayed, or a red "Warning" that the check does not match if the firmware on any components is different from what the image expects.  If the check passes, then the firmware is correct, and the boot will continue up to the OS login prompt. If the check fails, then the firmware will automatically be updated, and a subsequent reboot will occur.  You should not see this unless there were components replaced previously that were not updated.

2. After the OS is up, login as root and verify all the expected disk devices are present:

The following command should show 12 disks:

If the device count is not correct check also that the LSI controller has the correct Virtual Drives configured and in Optimal state, physically Online and spun up, with no Foreign configuration. There should be Virtual Drives 0 to 11, and the physical slots 0 to 11 should be allocated to 1 each (not necessarily the same 0:0 1:1 etc. mapping).

If this is not correct, then there is a problem with the disk volumes that may need additional assistance to correct. The server should be re-opened and the device connections and boards checked to be sure they are secure and well seated BEFORE the following CellCLI commands are issued.

3. Once the hardware is verified as up and running, the Customer's DBA will need to activate the grid disks:

Issue the command below and all disks should show 'active':

4. Verify all grid disks have been successfully put online using the following command. Wait until asmmodestatus is ONLINE for all grid disks and no longer SYNCING. The following is an example of the output early in the activation process.

Notice in the above example that the grid disks are still in the 'SYNCING' process. Oracle ASM synchronization is only complete when ALL grid disks show asmmodestatus=ONLINE. This process can take some time depending on how busy the machine is, and has been while this individual server was down for repair.

OBTAIN CUSTOMER ACCEPTANCE

- WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO

TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

- Verify that HW Components and SW Components are returned to properly functioning state with server up and all ASM disks online on Storage Servers.

REFERENCE INFORMATION:

Oracle Server X5-2L Documentation http://docs.oracle.com/cd/E41033_01/html/E48325/index.html

Sun Server X4-2L Documentation http://docs.oracle.com/cd/E36974_01/index.html

Sun Server X3-2L (formerly X4270M3) Documentation http://docs.oracle.com/cd/E23393_01/index.html

This solution has no attachment