How to Replace a SPARC T7-4 PCIe Card and/or PCIe Card Carrier [VCAP]

Asset ID:	1-71-1966601.1
Update Date:	2017-11-15
Keywords:

Solution Type Technical Instruction Sure

Solution 1966601.1 : How to Replace a SPARC T7-4 PCIe Card and/or PCIe Card Carrier [VCAP]

Applies to:

SPARC T7-4 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

How to Replace a SPARC T7-4 PCIe Card and/or PCIe Card Carrier

Solution

*******************************************************************************
To report errors or request improvements on this procedure, please add a comment on Doc ID: 1966601.1

*******************************************************************************

ESD Caution:

Circuit boards and drives contain electronic components that are extremely sensitive to static electricity. Ordinary amounts of static electricity from clothing or the work environment can destroy the components located on these boards. Do not touch the components along their connector edges.
Use a Antistatic Wrist strap. Attach one end of the strap to your wrist and the other end to the chassis, depending on what type of strap you use, with the adhesive end or the metal plug.
Use an Antistatic Mat. Place ESD-sensitive components such as motherboards, memory, and other PCBs on an antistatic mat.

Contamination Caution:

Dust particles of packaging material are number one cause of datacenter contamination. Make sure to remove all packaging material, up to the ESD safe packaging material, while still being outside the datacenter.

DISPATCH INSTRUCTIONS

WHAT SKILLS ARE REQUIRED?: No special skills required, Customer Replaceable Unit (CRU) procedure

Time Estimate: 30 minutes

Task Complexity: 0

REMOVAL/REPLACEMENT INSTRUCTIONS:

PROBLEM OVERVIEW: SPARC T7-4 PCIe Card and/or PCIe Card Carrier Replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

DAMAGE ALERT: Perform a visual inspection of the replacement part to make sure that there are no damaged components, connectors, bent pins, damaged packages during shipping, etc. If the part is damaged, don't install it into the system, order a new part. Handle with caution and package carefully the return part to avoid any damages during shipping.

Note - PCIe Expansion Card(s)/Carrier(s) is/are hot-service components that can be replaced at any time if the card(s) is/are not in use.

Note for OVM (LDOM) - To remove a PCIe card that is assigned to an I/O domain, first remove the device from the I/O domain. Then, add the device to the root domain before you physically remove the device from the system. These steps enable you to avoid a configuration that is unsupported by the Direct I/O or SR-IOV feature. For more information about making hardware changes to an I/O domain, refer to the Oracle VM for SPARC documentation. Also, removing the PCI card dynamically (DR) is not supported and 'cfgadm' will not work on Physical I/O devices bound to LDOM configuration, reference the OVM Administration Guide.

If the system is still up and functioning, the customer should perform an orderly and graceful shutdown of applications and OS. Then power off the server and remove the AC power cords from the system. A data backup is not a prerequisite but is a wise precaution.
NOTE: For ALL scenarios where an AC power down or AC power cycle is required for a T7-x server, please always use the steps in doc 1571054.1 prior to physically removing AC power cables from the server.

WHAT ACTIONS ARE REQUIRED?:

Note - The SPARC T7-4 server supports single-wide and double-wide card carriers. The removal steps are the same for both carrier types.

Note - Removing PCIe card carriers while the server is at the OpenBoot prompt is not supported. The server must be either powered off, or booted into the Oracle Solaris OS.

Replace a PCI Expansion Card/Carrier

1. Attach an anti-static wrist strap.

2. Locate the PCIe expansion card/carrier that you want to replace at the rear of the server.

3. Determine if you are removing a card carrier from a running server. Take one of these actions to bring the PCIe card offline:

a. If you are removing a PCIe expansion card/carrier from a server that is running (that is, if you are hot-plugging the PCIe expansion card/carrier), go to Step 4.
b. If you are removing a PCIe expansion card/carrier from a powered-down server, go to Step 5.

4. Take the PCIe card offline. Do one of the following:

a. Press the ATTN button. The PCIe card carrier's Power OK LED should should go off, indicating that the PCIe expansion card/carrier is ready to be removed. Go to Step 5.
b. Use the Oracle Solaris hotplug command:

1. List all devices in the device tree, including PCIe expansion cards:

# hotplug list -cv

2. This command lists dynamically reconfigurable hardware resources and shows their operational status. In this case, look for the status of the PCIe Expansion card/carrier you plan to remove. This information is listed in the State column.

For Example:

# hotplug list -cv

Connection State Description
___________________________________________________
PCIE1 EMPTY PCIe-Native
...
PCIE7 ENABLED PCIe-Native

Device Usage
_________________________________________________

SUNW,qlc@0 -
fp -
disk -
fp@0,0 -
SUNW,qlc@0,1 -
fp -
disk -
fp@0,0 -
PCIE13 EMPTY PCIe-Native
PCIE15 EMPTY PCIe-Native

3. Bring the PCIe expansion card/carrier offline:

For Example:

# hotplug disable PCIE7

4. Verify that the PCIe expansion card/carrier status is PRESENT

For example:

# hotplug list -cv PCIE7

Connection State Description
______________________________________________________________________________
PCIE7 PRESENT PCIe-Native

5. Verify that the card's green Power OK LED is off

b. Use the cfgadm command to bring the PCIe expansion card/carrier offline:

1. Use the cfgadm -al command to list all devices in the device tree, including PCIe expansion cards:

For Example:

# cfgadm -al

2. This command lists dynamically reconfigurable hardware resources and shows their operational status. In this case, look for the status of the PCIe expansion card/carrier you plan to remove. This information is listed in the Occupant column.

For Example:

Ap_id Type Receptacle Occupant    Condition
PCI-EM0                     sas/hp      connected    configured      ok
PCI-EM1                     sas/hp      connected    configured      ok
...

3. Bring the PCIe expansion card/carrier offline:

For Example:

# cfgadm -c disconnect Ap-id
where Ap-id is the ID of the card that you want to remove.

4. Verify that the card's green Power OK LED is off.

5. Disconnect any transceivers, if applicable, and any cables connected to the PCIe expansion card.

Tip - Label the cables to ensure proper connection to the replacement PCIe expansion card.

6. Pull the PCIe expansion card carrier handle down to disengage the carrier from the PCIe expansion card cage.

7. Remove the PCIe expansion card carrier from the server.

8. Remove a PCIe expansion card from the carrier:

a. Unlatch and open the top cover of the PCIe expansion card carrier.
b. Carefully, remove the PCIe expansion card from the carrier.

9. Install a PCIe expansion card into the PCIe expansion card carrier:

a. Unlatch and swing open the top of the PCIe espansion card carrier.
b. Insert the PCIe expansion card into the PCIe expansion card carrier until the bottom connector is firmly seated in the PCIe expansion card carrier's connector.

Caution - Do not twist or turn the PCIe expansion card as you insert it into the PCIe expansion card carrier. Ensure that the PCIe expansion card's connector is fully seated in the PCIe expansion card carrier's slot, and that the notch in the PCIe expansion card's rear bulkhead is seated around the PCIe expansion card carrier's alignment tab.

c. Close and latch the PCIe expansion card carrier's top cover.

10. Install a PCIe expansion card carrier into the PCIe expansion card cage.

Note - Installing PCIe expansion card carriers while the server is at the OpenBoot prompt is not supported. The server must either be powered off, or booted into the Oracle Solaris OS.

11. Insert the PCIe expansion card carrier into the open slot in the PCIe expansion card cage.

12. Close the latch to lock the PCIe expansion card carrier in place.

13. Reconnect all of the cables and any transceivers, if applicable, to the PCIe expansion card.

14. Determine if you are installing a PCIe card carrier into a running server.

a. If you are installing a PCIe card carrier into a server that is running (that is, if you are hot-plugging the PCIe card carrier), go to Step 15.
b. If you are installing a PCIe card carrier into a powered-down server, poweron the server, then verify the PCIe card.

15. Take the PCIe card online. Do one of the following:

a. Press the ATTN button. The PCIe card carrier's Power OK LED should light, indicating that the PCIe card is online.
b. Use the Oracle Solaris hotplug command:

For example:

# hotplug enable PCIE7

c. Use the Oracle Solaris cfgadm command:

# cfgadm -c connect Ap-id
where Ap_id is the ID of the card that you want to connect.

16. Verify the PCIe expansion cards installation/functionality

a. Verify that the Fault LED is not lit on the PCIe card.
b. Verify that the System Service Required LEDs on the front panel and rear I/O module are not lit.
c. Verify that the System PCIe Fault LED on the front panel is not lit.
d. Verify that the green Power LED is lit on the PCIe card carrier that you installed.
e. At the Oracle Solaris prompt, assure that the PCIe card is connected and online.

For Example:

# hotplug list -cv
Connection State Description
______________________________________________________________________________
PCIE1 EMPTY PCIe-Native
...
PCIE7 ENABLED PCIe-Native

Device Usage
___________________________________________________________________________
SUNW,qlc@0 -
fp -
disk -
fp@0,0 -
SUNW,qlc@0,1 -
fp -
disk -
fp@0,0 -
PCIE13 EMPTY PCIe-Native
PCIE15 EMPTY PCIe-Native

OBTAIN CUSTOMER ACCEPTANCE

WHAT ACTIONS ARE REQUIRED TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

Boot system and monitor boot sequence for errors. Test functionality of system:

1. Run the Solaris "fmadm faulty" and SP/ILOM "show faulty" command to verify that the fault has been cleared.
2. Perform one of the following tasks based on your verification results:
   * If the previous steps did not clear the fault, refer to doc 1004229.1 for information about the tools and methods you can use to diagnose and clear
     component faults.
   * If the previous steps indicate that no faults have been detected, the component has been replaced successfully. No further action is required
3. Restart software applications per applicable administration guides to resume system operation.

PARTS NOTE:
https://support.oracle.com/handbook_private/Systems/SPARC_T7_4/components.html#Ethernet

REFERENCE INFORMATION:
SPARC T7-4 Service Manual: http://docs.oracle.com/cd/E54990_01/html/E54994/index.html

See also: Oracle Integrated Lights Out Manager (ILOM) 3.2.1 Documentation Collection: http://docs.oracle.com/cd/E37444_01/

Save

References

<NOTE:1571054.1> - Performing an AC power cycle on the T3/T4/T5/S7/T7/T8 Servers

Attachments

This solution has no attachment