How to Replace an Oracle Server X7-8 Dual PCIe Card Carrier (DPCC) and/or PCIe Card

Asset ID:	1-71-2347740.1
Update Date:	2018-05-10
Keywords:

Solution Type Technical Instruction Sure

Solution 2347740.1 : How to Replace an Oracle Server X7-8 Dual PCIe Card Carrier (DPCC) and/or PCIe Card

Applies to:

Oracle Server X7-8 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

How to Replace an Oracle Server X7-8 Dual PCIe Card Carrier (DPCC) and/or PCIe Card.

Solution

DISPATCH INSTRUCTIONS

WHAT SKILLS DOES THE FIELD ENGINEER/ADMINISTRATOR NEED?:
No special skills required, Customer Replaceable Unit (CRU) procedure

TIME ESTIMATE: 30 minutes

TASK COMPLEXITY: 0

FIELD ENGINEER/ADMINISTRATOR INSTRUCTIONS:

PROBLEM OVERVIEW: An Oracle Server X7-8 Dual PCIe Card Carrier (DPCC) and/or PCIe Card needs replacement

WHAT STATE SHOULD THE SYSTEM BE IN TO BE READY TO PERFORM THE RESOLUTION ACTIVITY? :

System should be powered down.

WHAT ACTIONS ARE REQUIRED?:

Reference Doc:

Remove a DPCC:
https://docs.oracle.com/cd/E71925_01/html/E71936/gnsen.html#scrolltoc

Note: The DPCCs can be removed with the server OS running, but the hot-pluggability of the PCIe cards is OS and driver dependent which is outside the scope of this replacement action plan.

How to Remove a DPCC

1. Prepare the server for hot service.
See Preparing the Server for Component Replacement.

Note - This procedure can also be completed as a cold service procedure.

2. Identify the DPCC to be removed.

3. Use a stylus to press one or both ATTN buttons on the DPCC front.
The ATTN buttons alert the system to a request to remove a PCIe card. When the system has acknowledged the request, the server takes the device offline and lights indicators for each slot. When the indicators are lit, you can safely remove the component.

Note - If only a single PCIe card is present, press only the corresponding ATTN button.

4. To unlock the DPCC lever, lift the release latch and pull the lever downward, away from the server.
This action disengages the PCIe card IO connectors from the connectors on the back of the CMODs.

5. To remove the DPCC, slide it out of the server.

Remove a PCIe Card

1. Orient the DPCC so that the hinge is to the left.

2. To open the top of the DPCC, lift the release latch at the non-hinged end of the lid and rotate the lid upward and to the left.

3. To remove the card, pull it straight up and out of its connector.

4. Repeat steps 3 for the second PCIe card in the DPCC if replacing the DPCC.

Install a PCIe Card

1. Identify the DPCC PCIe slot.

2. Orient the DPCC so that the hinge is to the left.

3. Ensure that the DPCC top cover is open. The top of the DPCC is hinged at one end. To open, lift the tab on the non-hinged end.

4. Orient the PCIe card with the edge (or bus) connector facing downward and the IO (or cable connector) facing to the left. This puts the component side of the card facing away from you.

5. To install the card, align the edge connector with the slot in the DPCC and push the card downward into the slot.

6. To close the top of the DPCC, rotate it to the right ensuring the clip on the edge of the top is secured over the unhinged edge of the DPCC.

Caution - Pinch point. Keep fingers away from the underside of the top when closing it.

Install the DPCC

1. Ensure the top of the DPCC is closed and secured and the lever on the front of the DPCC is in its fully open position.

2. Align the DPCC with the vacant slot. The connector (back) side of the DPCC faces inward toward the server.

3. Slide the DPCC into the slot until it stops. This leaves the DPCC protruding slightly from the back of the server. Do not attempt to push the DPCC inward beyond this point.

4. Rotate the lever on the DPCC upward until it locks into place. This action draws the DPCC inward engaging the connectors in the DPCC with the connectors on the server midplane.

Caution - Pinch point. Keep fingers away from the backside of the lever when closing it

5. Use a stylus to press both ATTN buttons on the front of the DPCC.
The buttons alert the system to a request to bring the devices online. When the system acknowledges the request, it lights the OK indicators on the DPCC.

Note - If only a single PCIe card is present, press only the corresponding ATTN button. If you are doing cold service, this step is not necessary.

6. Verify that the green OK indicators on the front of the DPCC are on steady.

7. Prepare server for operation, power on server and start OS.

WHAT ACTIONS ARE REQUIRED TO RETURN THE SYSTEM TO AN OPERATIONAL STATE?:

How to verify the DPCC is working properly

Log in to the ILOM CLI.

Enter the following command to check status is normal for DPCC:

-> show /SYS/DPCC0

/SYS/DPCC0
    Targets:
        PCIE2
        PRSNT

    Properties:
        type = PCIE Module

    Commands:
        cd
        show

->

To check the status of a PCIe card use these commands:

-> show /SYS/DPCC0/PCIE2

/SYS/DPCC0/PCIE2
Targets:
PRSNT
P_ENABLE

Properties:
type = PCIe Hot Plug Carrier
fault_state = OK
clear_fault_action = (none)

    Commands:
        cd
        set
        show

->

-> show /System/PCI_Devices/Add-on/Device_2

/System/PCI_Devices/Add-on/Device_2
    Targets:

    Properties:
        part_number = 7107091
        description = Sun Flash Accelerator F80 800GB eMLC PCI-E 2.0 Low
                      Profile Adapter
        location = PCIE2 (PCIe Slot 2)
        pci_vendor_id = 0x1000
        pci_device_id = 0x007e
        pci_subvendor_id = 0x108e
        pci_subdevice_id = 0x050a

    Commands:
        cd
        show

->

Use the fault management shell to clear the fault for a specific component:

For example:

faultmgmtsp> fmadm acquit /SYS/DPCC0/PCIE2

or you can use the UUID

faultmgmtsp> fmadm acquit UUID

Check if any error output from event log

Example

-> show /SP/logs/event/list

Check if any faulted parts exist.

-> start /SP/faultmgmt/shell
Are you sure you want to start /SP/faultmgmt/shell (y/n)? y

Example

faultmgmtsp> fmadm faulty
No faults found

Check FMA and OS information to verify that no errors existed

Example

# fmadm faulty -a
STATE RESOURCE / UUID
-------- ----------------------------------------------------------------------

# prtdiag -v
System Configuration: Sun Server X7-8
BIOS Configuration: American Megatrends Inc. 29011300 08/26/2010
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)

==== Processor Sockets ====================================

Version Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU E7-8895 v2 @ 2.80GHz CPU 1
Intel(R) Xeon(R) CPU E7-8895 v2 @ 2.80GHz CPU 2
Intel(R) Xeon(R) CPU E7-8895 v2 @ 2.80GHz CPU 3
Intel(R) Xeon(R) CPU E7-8895 v2 @ 2.80GHz CPU 4
.
==== Upgradeable Slots ====================================

ID Status    Type             Description
--- --------- ---------------- ----------------------------
1   in use    PCI Express Gen3 x16 /SYS/DPCC0/PCIe1
2   in use    PCI Express Gen3 x86 /SYS/DPCC0/PCIe2
3   available PCI Express Gen3 x16 /SYS/DPCC1/PCIe3
4   in use    PCI Express Gen3 x86 /SYS/DPCC1/PCIe4
5   available PCI Express Gen3 x16 /SYS/DPCC2/PCIe5
6   in use    PCI Express Gen3 x86 /SYS/DPCC2/PCIe6
7   available PCI Express Gen3 x16 /SYS/DPCC3/PCIe7
8   in use    PCI Express Gen3 x86 /SYS/DPCC3/PCIe8
.
.

Check /var/adm/messages file to verify that no errors existed

Example

bash-3.00# pwd
/var/adm

# grep -i warning messages
# grep -i error messages
# grep -i fail messages

PARTS NOTE:

REFERENCE INFORMATION:
Oracle Server X7-8 Service Manual
https://docs.oracle.com/cd/E71925_01/html/E71936/index.html

How to Shutdown and Startup Exadata X5 (and later) compute nodes and storage cells when performing hardware maintenance (includes Supercluster X5 (and later) storage cells) (Doc ID 1982342.1)

References

<NOTE:1381773.1> - How to clear FMA logs on the ILOM or Solaris on x86 platforms

Attachments

This solution has no attachment