Exadata X5-8 Hardware
 
Exadata X6-8 Hardware
 
Exadata X4-8 Hardware
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Exadata internal only for Oracle support engineers use and approved HW partners

Applies to:

Goal

How to Replace an Exadata X4-8, X5-8, X6-8 Compute Node Internal RAID HBA Card.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:
Exadata X4-8/X5-8/X6-8 Training

TIME ESTIMATE: 90 Minutes

TASK COMPLEXITY: 3

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Exadata X5-8/X6-8 Compute Node HBA ESM (SuperCap) or ESM Cable needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

- The server that contains the faulty HBA should have its services offline and system powered off.

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

The instructions below assume the customer DBA is available and working with the field engineer onsite to manage the host OS and
DB/ASM services. They are provided here to allow the FE to have all the available steps needed when onsite, and can be done by the
FE if the customer DBA wants or allows or needs help with their steps.

Step A. Pre-Steps to shutdown the node for servicing:

1. For Extended information on this section, check MOS Note:
ID 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.

For a documentation reference, in the Exadata Maintenance Guide, use the section of chapter 1 "General Maintenance Information"

titled "Non-Emergency Power Procedures" section "Powering Off Oracle Exadata Rack" sub-section "Powering off Database Servers" available on the customer's cell
server image in the /opt/oracle/cell/doc directory, or here:
https://docs.oracle.com/cd/E80920_01/DBMMN/exadata-general-maintenance.htm#DBMMN20984

It is highly recommended to make and verify a backup of all disk partitions prior to RAID HBA replacement.

If running OVM then go to section "For Compute Node running OVM" - for non-OVM proceed as follows:

 
Shutdown crs


i. As root user do the following to stop crs and disable autostart of crs on reboot:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.

In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

ii. Validate CRS is down cleanly. There should be no processes running.
  

For Compute Node running OVM proceed as follows:

If there are any concerns engage EEST engineer.

The customer should perform the following:

(a) See what user domains are running (record result )

Connect to the management domain (domain zero, or dom0).

  This is an example with just two domains and the management domain Domain-0

connect to each domain  using the command

where domainname would be dm01db01vm01 or dm01db01vm02 if using the above examples.

Shut down any instances of crs ,refer to the example above in previous section "shutdown crs" in all user domains

Press CTRL+] to disconnect from the console.

(b)Shutdown all user domains from dom0

(c) See what user domains are running (should be only Domain-0)

(d) Disable user domains from auto starting during dom0 boot after HBA has been replaced. 

3. Revert all the RAID disk volumes to WriteThrough mode to ensure all data in the RAID cache memory is flushed to disk and not lost
when replacement of the HBA occurs. Set all logical volumes cache policy to WriteThrough cache mode:

Verify the current cache policy for all logical volumes is now WriteThrough :

4. The customer can now shutdown the server operating system:

5. Disconnect the AC power cords from the server. This is a cold-service item.

Step B. Physical RAID Card replacement

Reference links for Service Manual:

Sun Server X4-8 Service Manual
https://docs.oracle.com/cd/E40591_01/html/E40317/index.html

Oracle Server X5-8 Service Manual
https://docs.oracle.com/cd/E56301_01/html/E56311/index.html

Sun Server X4-8 System Module (SMOD) Overview
https://docs.oracle.com/cd/E40591_01/html/E40317/gnsfb.html#XFESMgnzll

Sun Server X4-8 SAS Host Bus Adapter (HBA) Card Overview
https://docs.oracle.com/cd/E40591_01/html/E40317/gnsfb.html#XFESMgnshr

The server requires one internal HBA for the externally-accessible SAS (or SATA) SMOD server storage drives. The HBA is installed in a riser slot on the underside of the SMOD motherboard and connected to the backplanes by two mini-SAS4I connector cables. The card is accessible by turning over the SMOD.

Remove the old HBA PCI Card

1. Prepare the server for cold service.

2. Access the rear of the server.

3. Label and disconnect all cables from the SMOD.

4. Disengage the SMOD from the server midplane.

    A. To Unlock the SMOD handles, squeeze together the release latches on the end of both handles.  The unlocking action is accompanied by an audible click.

    B. To disengage the SMOD from the server midplane, simultaneously, rotate both handles downward toward their fully open positions.  This action disengages the connectors on the SMOD from the connectors on the server midplane.

5. To remove the SMOD, grab it by it's sides and slide it out of the server.

6. Set the SMOD on a flat surface with the front side facing toward you.

7. Close the SMOD handles.

8. Rotate the SMOD 180 degrees so the back (connector) side is facing toward you.

9. Turn the SMOD upside down and locate the HBA.  The HBA is located inside the SMOD and is accessible through the upward-facing opening on the right side.

10. To unlock the HBA card, rotate the green release handle out and then downward.  The handle is on the right (vertical) side of the SMOD.  To access it, you might need to push it outward from the inside.

11. To disconnect the HBA card from its connector on the SMOD motherboard, pull the card toward you.  Two cables with mini SAS4I connectors connect the HBA to the storage drive backplane.

12. Carefully lift the card up, and pull it slightly away from the SMOD until the cable connectors are visible.

13. Disconnect the cables from the card.

14. Remove the HBA card from the SMOD and place it on an anti-static mat.

Install the new HBA PCI Card into the SMOD

1. Position the HBA card with the component side facing upward and the edge connector pointed away from you.

2. With the SMOD positioned upside down and the back (connector) side facing you, orient the HBA card in the opening on the right.

3. Ensure that the two backplane cables are routed under the support bracket on the bottom of the SMOD.

4. Connect the cables to the connectors on the HBA card.  Note the cable on the right side should connect to the adapter connector closest to the HBA PCI slot connector.  The cable on the left should connect to the connector closest to the top of the card or outside edge.

5. Lower the card and align its edge connector with the PCIe slot on the SMOD motherboard.

6. To install the HBA card, push it into the slot.

7. To lock the card, rotate the green locking lever upward.

8. Rotate the lever inward so it tucks inside the SMOD frame and does not extend outward.

Install the SMOD into the Server

9. Ensure that the handles on the SMOD are in their fully open position.

    A. To unlock the SMOD handles, squeeze together the release latches on the end of both handles.  The unlocking action is accompanied by an audible click.

    B. To open, rotate both handles downward until they are at a 90 degree angle to the SMOD (fully-open position).

10. Orient the SMOD with the handles facing away from the server and the connectors facing toward the open slot in the server.  The handles should be at the bottom front and the disks should be at the top front.

11. Align the SMOD in the slot.

12. Slide the SMOD into the slot until it stops.  This leaves the SMOD protruding slightly from the back of the server. Do not attempt to push the SMOD inward beyond this point.

13. To install the SMOD, simultaneously rotate both handles upward until they lock into place.  This action draws the SMOD inward engaging the SMOD connectors with the connectors on the server midplane.

14. Connect the cables to the SMOD.

Power on :

1. Re-attach the AC Power cords to the server.
2. Once the ILOM has booted you will see a slow blink on the green LED for the server. Power on the server by pressing the power
button on the front of the unit.

Server Services Startup Validation:

DB Node Startup:

1. As the system boots the hardware/firmware profile will be checked, and either a green "Passed" will be displayed, or a red "Warning"
that the check does not match if the firmware on the HBA is different from what the image expects.
If the check passes, then the firmware is correct, continue to step 2.

If the check fails, then:
a) Login as root at the OS login prompt.
b) Run the following to update the RAID HBA to the correct supported firmware for the image:

c) After the firmware updates, the server will reboot again. The disk volumes should remain intact and boot up to the OS again.

2. After the OS is up, login as root and validate the physical and logical volumes are seen properly from the new RAID HBA in the OS,
for the configuration that it should be for the DB node and that the supercap is seen:

3. Set all logical drives cache policy to WriteBack cache mode:

Verify the current cache policy for all logical drives is now using WriteBack cache mode:

4. CRS services should now be started.

"DB Node Startup Verification" - for compute node NOT running OVM ,for OVM refer to next section.

Startup CRS and re-enable autostart of crs. After the OS is up, the Customer DBA should validate that CRS is running. As root execute:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.
In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.
Example output when all is online is:

2. Validate that instances are running:

It should return a record for the ASM instance and a record for each database.

For Compute Node running OVM

If the customer requires assistance please ask them to contact EEST engineer or parent case owner.

Once the compute node has booted ,re-enable user domains to autostart during Domain-0 boot.

Startup all user domains that are marked for auto start

See what user domains are running (compare against result from previously collected data)

if any not auto-started then Startup a single user domain

 Check that crs has started in user domains ,refer to previous section "DB Node Startup Verification"

4. Verify also the InfiniBand links are up at 40Gbps as the cables were disconnected:

OBTAIN CUSTOMER ACCEPTANCE
WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

- Verify that HW Components and SW Components are returned to properly functioning state with server up and database services
operating on DB Servers

PARTS NOTE:

REFERENCE INFORMATION:

1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.

Service Manual's:

Sun Server X4-8 Service Manual
https://docs.oracle.com/cd/E40591_01/html/E40317/index.html

Oracle Server X5-8 Service Manual
https://docs.oracle.com/cd/E56301_01/html/E56311/index.html

Sun Server X4-8 System Module (SMOD) Overview
https://docs.oracle.com/cd/E40591_01/html/E40317/gnsfb.html#XFESMgnzll

Sun Server X4-8 SAS Host Bus Adapter (HBA) Card Overview
https://docs.oracle.com/cd/E40591_01/html/E40317/gnsfb.html#XFESMgnshr

This solution has no attachment