SPARC M7-16
 
SPARC M8-8
 
SPARC M7-8
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: M7
 
Tools>Primary Use>Configuration
 

The intent of this document is to provide the reader with an overview of how the SPARC M8 and SPARC M7 Series Servers device paths are organized.
More details about the SPARC M8-8, SPARC M7-8, and M7-16 servers IO Architecture are available in the SPARC M8 and SPARC M7 Series Servers Administration Guide.

In this Document

Purpose

Details

Applies to:

Purpose

The intent of this document is to provide the reader with an overview of how the SPARC M8 and SPARC M7 Series Servers device paths are organized.
More details about the SPARC M8-8, SPARC M7-8, and M7-16 servers IO Architecture are available in the SPARC M8 and SPARC M7 Series Servers Administration Guide.

Details

The SPARC M7 and SPARC M8 generation servers have an IO architecture that is significantly different from the previous generations of T and M series servers.
One of the largest contributors to the new architecture is that the processors no longer directly host the PCI Express fabric.

The figure on the right shows a simplified block diagram of a M7 CMIOU (CPU Memory and IO Unit) board. The IO Hub (IOH) is now external to the CPU.

The IO path starts at the M7 Processor, which is connected to the IOH chip via two bi-directional sixteen lane (x16) IO links (IL). The IOH then converts these IL links into five Gen3 PCIe links. They are shown as RC0 to RC4 in the diagram. Root complexes RC0, RC1 and RC3 connect to x16 PCIe slots that support PCIe card hot plug via a carrier.  Within a singe M7 CMIOU board there are three PCIe x16 slots. RC4 is split into two single lane (x1) links to connect to the Service Processor (SP) module and to a USB controller.

The PCIe card carrier slots are connected to the IOH root complex ports as follows:

RC3 connects to PCIE1/CAR slot
RC0 connects to PCIE2/CAR slot
RC1 connects to PCIE3/CAR slot

The device path enumeration of the root complex starts at pci@300 and increments by 0x1 (see tables below).

The IOH ASIC eliminates the need for 3rd party PCIE Switches.  PCIE slots are directly connected to root complex ports on the ASIC.  The direct connections result in low latency for PCIe transactions.

The M8 CMIOU diagram below shows it slightly modifies the IO connections to the IOH. The SP connection is still routed via root complex RC4 (shared on the M7 CMIOU), but the eUSB device is now connected to RC2 (unused on the M7 CMIOU).  This permits root domain assignment of the eUSB device separate from the SP pathway.

Root Complex: The Root Complex (RC) is the first element of a PCIe device path (eg /pci@300). In the SPARC M8 and SPARC M7 Series Servers, the RC functionality has been moved out the processor and onto the IOH chip.  Each CMIOU has one I/O controller chip.

Root Port: A root complex can have one or more Root Ports. Each Root Port is independent from any other in that fabric. However, any device under the same Root Complex shares that same fabric.  In other words, a Root Port connects to its own tree of PCIe devices, but they are completely independent of other devices to other Root Ports out of the same Root Complex.  The Root Port will always be the second term in the PCIe device path. The first Root port always be pci@1 and second will be pci@2 and so on.

The follow table lists the number of available PCIe slots per PDOmain on the M8 and M7 series servers

The following table lists the DCU and CMIOU components in the servers

The diagram below shows a high level of the IO layout for the SPARC M7-8 server with two PDomains.  The M8-8 server is largely identical with the exception that the eUSB device is connected via RC2 instead of sharing RC4 for SP communication.  For the SPARC M7-16 server, the diagram represents one half of the IO layout

Each Root Complex has a Solaris device path, an ILOM target path, and a bus name that is unique.  The bus name is used for IO assignment within Oracle VM server for SPARC.

The tables below the naming convention of device paths for boards CMIOU0 to CMIOU15 on both the M8 and M7 series servers

For CMIOU0

 * example for M7, M8 routes disk device via RC2

/pci@304/pci@1 [/SYS/SP0/SPM0]  ENABLED
/pci@304/pci@2/usb@0/storage@1/disk@0,0

For CMIOU1

For CMIOU2

For CMIOU3

For CMIOU4

For CMIOU5

For CMIOU6

For CMIOU7

For CMIOU8

For CMIOU9

For CMIOU10

For CMIOU11

For CMIOU12

For CMIOU13

For CMIOU14

For CMIOU15

Another consequence of moving the IOH to a new discrete chip that there is NO "IO Reconfiguration"

The previous generations of multi-socket M series systems supported a feature called "IO Reconfiguration" where at boot time the host firmware could reconfigure the IO hardware in case a CMT node completely failed, rerouting the connections to the onboard devices and slots so that they could be reached by the surviving CMT node(s).  This often resulted in unintended device path changes that caused downtime and confusion for both customers and Oracle Support engineers.

Since the PCIE fabric is now hosted by IOH chip, the fabric can remain unchanged on the M8 and M7 series servers if a CMT node is added or removed.  There is no longer a need for the "pci device paths" to change based on what processors are present.  The "pci device path" name for each root complex is controlled by the host firmware (ILOM).

Example from one of PDomain0 on a dual-host SPARC M7-8

Example of ldm output from the same domain:

 snippet of prtdiag output from the same domain

Example of How to check what kind of HBAs are installed in a PDomain from the active ILOM (service processor)

 Example How to check what kind of HBAs are installed in a PDomain from Solaris
  

This solution has no attachment