Oracle Server X7-2L
 

PLA-Support>Sun Systems>Sun_Other>Sun Collections>SN-OTH: x64-CAP VCAP
 

Applies to:

Goal

How to Replace an Oracle Server X7-2L PCIe NVMe Switch Card and/or Cable.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:
Oracle Server X7-2L Training

TIME ESTIMATE: 30 minutes

TASK COMPLEXITY: 1

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Oracle Server X7-2L PCIe NVMe Switch Card and/or Cable needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

If the system is still up and functioning, customer should perform an orderly and graceful shutdown of applications and OS.  Then power off the server and remove the AC power cords from the system.

A data backup is not a prerequisite but is a wise precaution.

WHAT ACTION DOES THE ENGINEER NEED TO TAKE:

1. Prepare the server for service.

1. Power off the server and disconnect the power cords from the power supplies.
2. Extend the server to the maintenance position.
3. Attach an anti-static wrist strap to your wrist, and then to a metal area on the chassis.

2. Remove the server top cover

1. If the release button latch on the server top cover is in the locked position, use a Torx T10 screwdriver to turn the release button latch clockwise to the unlocked position.
2. Lift up on the release button on top of the server cover. Lifting the release button causes the server cover to slide toward the rear of the chassis for easy removal.
3. Lift up and remove the top cover

3. Remove the air baffle

1. Release the latch for the host bus adapter (HBA) super capacitor tray and rotate the tray in to the upright position.
2. Remove the air baffle by lifting the baffle up and out of the server.

4. Remove the fan modules

1. Using your forefinger and thumb, lift the fan module straight up and out of the chassis and set it aside on an antistatic mat.

5. Remove the fan tray

1. Using a Torx T25 screwdriver, loosen the three spring-mounted screws that secure the fan tray to the server chassis.
2. Lift the fan tray from the server.

6. Remove the NVMe cables

1. Disconnect the NVMe cables from the disk backplane. Press each latch, and then pull out the cables to disengage them from the backplane connectors.
2. Disconnect the NVMe cables from the Oracle PCIe NVMe switch cards in PCIe slots 3, 4, 8, and 9. Press each latch, and then pull out the cables to disengage them from the HBA connectors.

7. Remove and Install the NVMe switch card

    Note: Skip this step for the replacement of NVMe cables

1. Rotate the PCIe card locking mechanism in to an upright position, and then lift up on the PCIe card to disengage it from the motherboard connectors.
2. Unpack the replacement PCIe card, and place it on an antistatic mat.
3. Insert the PCIe card into the correct slot, and rotate the PCIe locking mechanism downward to secure the PCIe card in place.

8. Installing the NVMe cables between the disk backplane and NVMe switch card

1. Route the NVMe cable bundle for Oracle PCIe NVMe switch cards in PCIe slots 3 and 4 through the cable trough in the middle of the chassis.
2. Route the NVMe cable bundle for Oracle PCIe NVMe switch cards in PCIe slots 8 and 9 through the cable trough along the left side of the chassis.
3. Reconnect the NVMe cables to the Oracle PCIe NVMe switch cards. Plug each cable into its connector until you hear an audible click. To ensure proper NVMe cable connections
4. Reconnect the NVMe cables to the disk backplane. Plug each cable into its NVMe connector until you hear an audible click.  
   

   Use the following table to ensure proper NVMe cable connections.

   PCIe Slot # and Cable P/N	HBA Connector	Disk Backplane Connector	Physical Storage Drive Slots
   3 (P/N:7345518 short cable)	1	D	2
   	2	E	6
   	3	F	10
   4 (P/N:7345517 long cable)	1	A	3
   	2	B	7
   	3	C	11
   8 (P/N:7345517 long cable)	1	A	1
   	2	B	5
   	3	C	9
   9 (P/N:7345518 short cable)	1	D	0
   	2	E	4
   	3	F	8

9. Install the fan tray

1. Lower the fan tray into the server.
2. Using a Torx T25 screwdriver, tighten the three spring-mounted screws that secure the fan tray to the server chassis

10. Install the fan modules

1. Press down on the fan module to fully seat the fan module.
2. Verify that the fan module status indicator (LED) on the replacement fan module is not illuminated.

11. Install the air baffle

1. Install the air baffle by placing it into the server and lowering it to its inserted position.
2. Rotate the HBA super capacitor tray down and in to its closed position. You will hear an audible click when the tray is properly secured to the air baffle.

12. Install the server top cover

1. Place the top cover on the chassis. Set the cover down so that it hangs over the back of the server by about 1 inch (25 mm) and the side latches align with the cutouts in the chassis.
2. Check both sides of the chassis to ensure that the top cover is fully down and flush with the chassis. If the cover is not fully down and flush with the chassis, slide the cover towards the back of the chassis to position the cover in the correct position.
3. Gently slide the cover toward the front of the chassis until it latches into place with an audible click. As you slide the cover toward the front of the server, the release button on the top of the server automatically rotates downward to the closed position. Latch the top cover by pushing down on the button until it is flush with the cover and you hear an audible click. An audible click indicates that the cover is latched.
4. Use a Torx T10 screwdriver to turn the release button latch counter-clockwise to the locked position.

13. Return the Server to operation:

1. Remove any anti-static measures that were used.
2. Return the server to it's normal operating position within the rack.
3. Re-install the AC power cords and any data cables that were removed but not yet re-attached.
4. Power on the server. Verify that the Power/OK indicator led lights steady on.

14. Verify NVMe Cable Connections

1. To enter the BIOS Setup Utility, press the F2 key (Ctrl+E from a serial connection) when prompted and while the BIOS is running the power-on self-tests (POST).
2. Open a terminal window. Using SSH (secure shell), log in to the Oracle ILOM command-line interface (CLI) on the server SP.
3. To access the Oracle ILOM Diagnostic Shell, type the start /sp/diag/shell command, and then type Enter.
4. Type y to confirm that you want to start the Oracle ILOM Diagnostic Shell. The diag> prompt appears.
5. To run the NVMe cable connection tests, type the hwdiag io nvme_test command, and then type Enter.
   -> start /SP/diag/shell
   Are you sure you want to start /SP/diag/shell (y/n)? y
   diag> hwdiag io nvme_test
   HWdiag (Diag Shell) - Build Number 120817 (Aug 21 2017, 00:15:43)
   Current Date/Time: Oct 04 2017, 17:19:34
   Checking NVME drive fru contents...
   checking fru on drive NVMe 0 OK
   checking fru on drive NVMe 1 OK
   checking fru on drive NVMe 2 OK
   checking fru on drive NVMe 3 OK
   checking fru on drive NVMe 4 OK
   checking fru on drive NVMe 5 OK
   checking fru on drive NVMe 6 OK
   checking fru on drive NVMe 7 OK
   checking fru on drive NVMe 8 OK
   checking fru on drive NVMe 9 OK
   checking fru on drive NVMe 10 OK
   checking fru on drive NVMe 11 OK
   NVME drives fru check: PASSED
   
   Checking NVME drive pcie links...
   checking pcie link on drive NVMe 0 OK
   checking pcie link on drive NVMe 1 OK
   checking pcie link on drive NVMe 2 OK
   checking pcie link on drive NVMe 3 OK
   checking pcie link on drive NVMe 4 OK
   checking pcie link on drive NVMe 5 OK
   checking pcie link on drive NVMe 6 OK
   checking pcie link on drive NVMe 7 OK
   checking pcie link on drive NVMe 8 OK
   checking pcie link on drive NVMe 9 OK
   checking pcie link on drive NVMe 10 OK
   checking pcie link on drive NVMe 11 OK
   NVME drives pcie link check: PASSED
   
   Checking NVME drive DSN...
   checking DSN on drive NVMe 0 OK
   checking DSN on drive NVMe 1 OK
   checking DSN on drive NVMe 2 OK
   checking DSN on drive NVMe 3 OK
   checking DSN on drive NVMe 4 OK
   checking DSN on drive NVMe 5 OK
   checking DSN on drive NVMe 6 OK
   checking DSN on drive NVMe 7 OK
   checking DSN on drive NVMe 8 OK
   checking DSN on drive NVMe 9 OK
   checking DSN on drive NVMe 10 OK
   checking DSN on drive NVMe 11 OK
   NVME drives DSN check: PASSED
   
   Checking NVME cabling...
   Cables associated with Switch Card 0 in PCIe Slot 3 verified
   Cables associated with Switch Card 1 in PCIe Slot 4 verified
   Cables associated with Switch Card 2 in PCIe Slot 8 verified
   Cables associated with Switch Card 3 in PCIe Slot 9 verified
   NVME cable check: PASSED
   NVME test PASSED
   diag>
   
   - If a cable is crossed (plugged into the incorrect port), then the NVMe drive DSN (Drive Serial Number) test should report two drives where the DSN does not match. The cabling for those drives should be inspected to ensure that it is correct.
   - If multiple NVMe drive PCIe links are not active, or multiple NVMe drive DSNs are not OK, then check the Port 2 connection on the NVMe switch card. If Port 2 is incorrectly cabled, it could affect all three drives that are connected to that NVMe switch card.
6. Once issues are resolved, power cycle the system by removing the power cords from the power supplies, waiting 30 seconds, and then reattaching the power cords.
7. Retest the system by repeating from the above step a.) Once all NVMe cable connection tests have passed, the server operating system can be booted.

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTION DOES THE CUSTOMER NEED TO TAKE TO RETURN THE SYSTEM TO AN OPERATIONAL STATE:

Boot up system and verify full functionality

REFERENCE INFORMATION:

Oracle Server X7-2L Service Manual

Oracle System Handbook - Oracle Server X7-2L

This solution has no attachment