Zero Data Loss Recovery Appliance X6 Hardware
 
Exadata X5-2 Hardware
 
Exadata X5-2 Eighth Rack
 
Exadata X5-2 Full Rack
 
Exadata X6-2 Hardware
 
Exadata X5-2 Quarter Rack
 
Zero Data Loss Recovery Appliance X5 Hardware
 
Exadata X5-2 Half Rack
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

In this Document

Goal

Solution

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: FRU replacement on Engineered system

Applies to:

Goal

 How to replace a faulty Infiniband card in Exadata X5-2/X6-2 Compute Node

Solution

 DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:

Exadata Server Training.

TIME ESTIMATE: 60 minutes.

TASK COMPLEXITY: 3-FRU

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY?:

The server that contains the faulty Infiniband HCA card should have its services offline and system powered off.

WHAT ACTION DOES THE FIELD ENGINEER/ADMINISTRATOR NEED TO TAKE?:

The instructions below assume the customer DBA is available and working with the field engineer onsite to manage the host OS and
DB/ASM services. They are provided here to allow the FE to have all the available steps needed when onsite, and can be done by the
FE if the customer DBA wants or allows or needs help with their steps.

Step A. Pre-Steps to shutdown the node for servicing:

1. For Extended information on this section, check MOS Note:
ID 1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.

For a documentation reference, in the Exadata Maintenance Guide, use the section of chapter 1 "General Maintenance Information"
titled "Non-Emergency Power Procedures" section "Powering Off Oracle Exadata Rack" sub-section "Powering off Database Servers" available on the customer's
cell server image in the /opt/oracle/cell/doc directory, or internal to Oracle here:
http://amomv0115.us.oracle.com/archive/cd_ns/E50790_01/doc/doc.121/e51951/general.htm#DBMMN21014

2.Check if there are any non-default Infiniband partitions.

 - Refer to document Updating IB partitions after replacing an Infiniband HCA in any nodes within IB network - steps to do after replacing HCA (Doc ID 1985159.1) - follows steps 1 and 2 from the document.

if running OVM then go to section "For Compute Node running OVM" - for non-OVM proceed as follows:

Shutdown crs

i. As root user do the following to stop crs and disable autostart of crs on reboot:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.
In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.

ii. Validate CRS is down cleanly. There should be no processes running.

For Compute Node running OVM proceed as follows:

If there are any concerns engage EEST engineer.

The customer should perform the following:

(a) See what user domains are running (record result )
Connect to the management domain (domain zero, or dom0).
This is an example with just two domains and the management domain Domain-0

connect to each domain using the command

where domainname would be dm01db01vm01 or dm01db01vm02 if using the above examples.

Shut down any instances of crs ,refer to the example above in previous section "shutdown crs" in all user domains

Press CTRL+] to disconnect from the console.

(b)Shutdown all user domains from dom0

(c) See what user domains are running (should be only Domain-0)

(d) Disable user domains from auto starting during dom0 boot after motherboard has been replaced.

The customer can now shutdown the server operating system:

Reference links for Service Manual:

X5-2 DB’s: ( http://docs.oracle.com/cd/E41059_01/html/E48312/napsm.gnriy.html#scrolltoc )

 The field engineer can now slide out the server for maintenance. Do not remove any cables prior to sliding the server forward, or the
loose cable ends will jam in the cable management arms (CMA). Ensure all customer-added data network cables are properly dressed
into the CMA Take care to ensure the cables and CMA is moving properly.
Remember to disconnect the power cords before opening the top of the server.

Locate and Remove the PCIe card.

(a) There are three external PCIe slots in the system. The external PCIe slots are numbered 1, 2, and 3 from left to right when you
view the server from the rear. The Infiniband card is always installed in PCIe slot 3.

(b) Locate the Infiniband card in PCIe slot 3 and unplug the two cables from the PCIe card making note of their locations so that they
can be re-installed in the same configuration (label if needed).

(c) lift the green-tabbed latch on the rear of the server's chassis next to the PCIe slot to release the PCIe card's rear bracket.

(d) To release the riser from the motherboard connector, lift the green-tabbed lease
lever on the PCIe riser to the open position.

(e) Slide the plastic PCIe card retainer, which is mounted on the side of the chassis,
toward the front of the server to release the card(s) installed in the riser .

(f) Grasp the riser with both hands and remove it from the server.

(g) Remove the Infiniband card from the PCIe riser. Hold the riser in one hand and use your other hand to carefully pull the PCIe
card connector out of the riser.

(h) Disconnect the rear bracket that is attached to the Infiniband card from the rear of the PCIe riser.

Replace the PCIe card.

(a) Insert the rear bracket that is attached to the Infiniband card into the PCIe riser.

(c) Hold the riser in one hand and use your other hand to carefully insert the PCIe card connector into the Riser.

(d) Install the PCIe riser with the installed PCIe cards into the server.

(e) Raise the PCIe riser release lever (marked with a green tab) to the open (up) position
Making sure to replace the riser into the same position from which it was removed (PCIe slot 3), gently press the riser into the
motherboard connector until it seats and press the green-tabbed, riser release lever to the closed (down) position.

(f) Close the green-tabbed latch on the rear of the server's chassis next to the applicable PCIe slot to secure the PCIe card's rear
bracket to the server's chassis.

(g) Reconnect the Infiniband cables to the PCIe card that were unplugged during the removal procedure making sure to connect them in the
same configuration as when they were disconnected.

Server Services Startup Validation:

DB Node Startup:

OBTAIN CUSTOMER ACCEPTANCE

The system administrator should verify the system is functioning correctly. Some suggested actions they can take to verify are:

1. On the host that the card was replaced run:

Ensure both Port 1 & Port2:
State is "Active"
Physical state: "LinkUp"
Rate: "40"

ii) refer to document Updating IB partitions after replacing an Infiniband HCA in any nodes within IB network - steps to do after replacing HCA (Doc ID 1985159.1) - follow steps 3 ,4 and 5 from the document.

2. Run Verify Infiniband topology (example of fully-operational system):

Most important at this stage are lines #2 & #3 of the output below which
have been highlighted.

3. Ping other nodes over the Infiniband subnet

CRS services should now be started.

"DB Node Startup Verification" - for compute node NOT running OVM ,for OVM refer to next section.

Startup CRS and re-enable autostart of crs. After the OS is up, the Customer DBA should validate that CRS is running. As root execute:

where GI_HOME environment variable is typically set to “/u01/app/11.2.0/grid” but will depend on the customer's environment.
In the above output the “1” of “+ASM1” refers to the DB node number. For example, Db node #3 the value would be +ASM3.
Example output when all is online is:

2. Validate that instances are running:

It should return a record for the ASM instance and a record for each database.

For Compute Node running OVM

If the customer requires assistance please ask them to contact EEST engineer or parent case owner.

Once the compute node has booted ,re-enable user domains to autostart during Domain-0 boot.

Startup all user domains that are marked for auto start

See what user domains are running (compare against result from previously collected data)

if any not auto-started then Startup a single user domain

 Check that crs has started in user domains ,refer to previous section "DB Node Startup Verification"

PARTS NOTE: 7092757

REFERENCE INFORMATION:

1093890.1 Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration.

Service Manual's:
X5-2 DB’s: ( Service Manual's:
X5-2 DB’s: ( http://docs.oracle.com/cd/E41059_01/html/E48312/napsm.html#scrolltoc )

This solution has no attachment