SPARC M5-32
 
SPARC M6-32
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: Mx-32
 

In this Document

Goal

Solution

PCIe Device Paths

References

Applies to:

Goal

The intent of this document is to provide the reader with an overview of how the SPARC M5-32 and M6-32 servers device paths are arranged.

More details about the SPARC M5-32 and M6-32 servers IO architecture are available in the Administration Guide .

Solution

A Physical Domain is composed of one to four DCUs (DCU0, DCU1, DCU2, DCU3).
The server contains up to four I/O units (IOUs) numbered IOU0, IOU1, IOU2, and IOU3. One per DCU.
Each IOU supports up to 16 PCIe slots, 8 10GBASE-T ports on 4 EMS modules, and 8 HDDs/SSDs.

PCIe I/O paths are routed from CMP root complexes, through PCIe switches, to a destination PCIe slot and/or EMS card. Each CMP provides two root complexes allowing it to attach to two PCIe switches. Each PCIe switch connects to two root complexes.
There are a total of 64 root complexes in this server (16 per DCU). These root complexes are named pci_0 to pci_63.

When all of the components are present and operational, 2 CMPs connect to each PCIe switch and that switch is partitioned into two virtual switches with independent traffic channels to maximize the I/O bandwidth.

Click on the following illustration to display the default PCIe I/O fabrics of a fully populated DCU0. Each PCIe and EMS slot is routed through the PCIe I/O switch fabric to its root complex. All four CMUs (CMU0–CMU3) are installed: rKVMS is supported with redundant paths, all PCIe and EMS slots are accessible, and CMUs are redundant.

Click on the following illustration to display a half-populated DCU0, which contains only CMU0 and CMU3. Each PCIe and EMS slot is routed through the PCIe I/O switch fabric to its root complex. Some of the PCIe and EMS slots are routed to different root complexes when compared to a fully-populated DCU.
Only CMU0 and CMU3 are installed: rKVMS is supported with no redundancy. If CMP0 fails, the rKVMS connection is lost. All PCIe and EMS slots are accessible, but if one CMU fails, half of the slots will be disabled.

The 2 illustrations above are applicable to the other DCUs.
Just replace the DCU, CMU and root complexe numbering as following :

Reminder about CMU/CMP considerations :

• A Domain Configuration Unit (DCU) is grouping 2 to 4 CMUs for non-bounded domain (expandable = true).
  • All four CMUs (CMU0–CMU3) are installed: rKVMS is supported with redundant paths, all PCIe and EMS slots are accessible, and CMUs are redundant.
  • Only CMU0 and CMU1 are installed: rKVMS is supported with redundant paths. If any CMP fails, the rKVMS connection remains. PCIe slots 9–16 and EMS 3 and 4 are inaccessible.
  • Only CMU0 and CMU3 are installed: rKVMS is supported with no redundancy. If CMP0 fails, the rKVMS connection is lost. All PCIe and EMS slots are accessible, but if one CMU fails, half of the slots will be disabled.
  • Only CMU1 and CMU2 are installed: rKVMS is supported with no redundancy. If CMU1 fails, the rKVMS connection is lost. All PCIe and EMS slots are accessible, but if one CMU fails, half of the slots will be disabled.
  • Only CMU2 and CMU3 are installed: rKVMS is not supported. PCIe slots 1–8 and EMS 1 and 2 are inaccessible. 
  • Configurations with only CMU0/CMU2 or only CMU1/CMU3 are not supported.
• A bounded domain (expandable = false) can have a minimum of one CMU.

Note about PCIe device root complex failover/reconfiguration : When a CMP or CMU fails, or when a CMU is removed, the primary root complexes on that CMP or CMU will no longer be available. Depending on how you set the Oracle ILOM ioreconfigure property on the PDomain host (/Servers/PDomains/PDomain_x/HOST/ ioreconfigure), the PCIe or EMS slots will either be rerouted to their secondary (alternate) root complexes or the slots will no longer be available.

If you set the ioreconfigure property to false, the PCIe I/O fabric paths will not be recreated when a CMU or CMP fails or is removed and the PDomain is restarted.
As a result, the primary paths to the associated PCIe or EMS slots are no longer available and no secondary (alternate) paths exist.
Setting ioreconfigure to false has no impact to the original, pre-failure physical device tree regardless of what CMUs or CMPs are missing.

If you set the ioreconfigure property to true, the PCIe I/O fabric paths will be recreated when a CMU or CMP fails or is removed and the PDomain is restarted.
As a result, the primary paths to the associated PCIe or EMS slots are no longer available but the PCIe or EMS slots can be reached using the secondary (alternate) root complexes.
Reconfiguring the I/O paths will change the PCIe addresses and external addresses associated with all devices including any affected boot devices.
Setting ioreconfigure to true will impact the original, pre-failure boot environment physical device tree if there are any failed or missing CMUs or CMPs since the PDomain was last restarted.

For more details about ioreconfigure, please refer to the SPARC M5-32 and SPARC M6-32 Servers Administration Guide.

From the ILOM shell :

• /SYS/CMU[0-7] (Contains root complexes)
• /SYS/IOU[0-3] represents the IOU
• /SYS/IOU[0-3]/IOB[0-1] (Contains PCIe switches)
• /SYS/IOU[0-3]/EMS[1-4] represents the EMS module
• /SYS/IOU[0-3]/EMS[1-4]/CARD (BaseIO card in EMS slot)
• /SYS/IOU[0-3]/PCIE[1-16] represents the PCIe slot
• /SYS/IOU[0-3]/PCIE[1-16]/CAR/CARD (user add-in card in LP-carrier in PCIE slot)
• /SYS/SPP[0-3] (Contains KVMS targets: PCIe switch, PCIE-to-USB bridge, SP emulates USB and Video devices)

The root complex names to device mapping as well as receptable status can be found via the 'ldm list-devices -a' or 'ldm list-io -l' output from the control domain on Solaris side.

Each EMS card provides access to :
- 4 HDD/SDD drives
- 2 10GbE interfaces

This illustration shows the SAS paths from the EMS modules to the drives.

Note that each SSD receives SAS path from one EMS module as following (primary paths); paths A in the illustration above :

Note that each HDD receives SAS paths from two different EMS modules as following (primary paths); paths A and B in the illustration above :

For alternate path (fully populated configuration and ioreconfigure), refer to the table below to determine the alternate root complexe and path.

For example, in a fully populated configuration, HDD2 in DCU0 is accessible via EMS2 and EMS4 (primary path) :

/pci@600/pci@1/pci@0/pci@0/pci@0/pci@4

pci@500/pci@1/pci@0/pci@0/pci@0/pci@4

but also from the alternate path (if primary is no longer available)

/pci@380/pci@1/pci@0/pci@0/pci@0/pci@4

/pci@480/pci@1/pci@0/pci@0/pci@0/pci@4

Also note that the EMS port used to access an HDD is different from one EMS to another EMS. Which means that the  target / PhyNum (as reported in the probe-scsi-all output) to access this disk is different. This can be summarized as following :

Using the same example as before, HDD2 in DCU0 is accessible via

/pci@600/pci@1/pci@0/pci@0/pci@0/pci@4/scsi@0/disk@b,0

pci@500/pci@1/pci@0/pci@0/pci@0/pci@4/scsi@0/disk@9,0

but also from the alternate path (if primary is no longer available)

/pci@380/pci@1/pci@0/pci@0/pci@0/pci@4/scsi@0/disk@9,0

/pci@480/pci@1/pci@0/pci@0/pci@0/pci@4/scsi@0/disk@b,0

Example from a DCU0 with /SYS/CMU3/CMP0 disabled, HDD2 is accessible from EMS2 (primary path) and EMS4 (alternate path) :

See SPARC M5-32 and M6-32 Servers: How to locate internal disks (Doc ID 1633233.1)

The list of root complexes and paths (primary and alternate) for each fully populated DCU can be summarized as following :

The list of root complexes and paths for each half populated DCU can be summarized as following :

A more detailed list of DCUs, NAC names, pci slots, root complexes, primary paths, CMU/CMP is available :

• Fully populated configuration,
• half populated configuration.

Note : Depending on which SPP is selected as the PDomain SPP when Solaris is installed in the control domain, the numbering and instance number for some interfaces may be different.

The following commands are available to list the devices paths, NAC name and respective information :

• OBP
  • show-devs
  • show-nets
• Solaris
  • prtpicl
  • prtdiag
  • fmtopo
  • cfgadm/hotplug
  • ldm list-devices -a io / ldm list-io -l
  • the /etc/path_to_inst  file

Example of ldm output from the same domain:

PCIe Device Paths

In ILOM, each end of a PCIE_LINK is represented in the NAC namespace because the PCIE_LINK represents a physical port on a chip. But both ends of the link reflect the same point in the PCIe fabric (i.e. the PCIe devPath).

BaseIO Card:

SPP KVMS & UsbEthernet targets:

This solution has no attachment