Sun SPARC[TM] Enterprise M4000 and M5000 Server Device Paths

Asset ID:	1-71-1002807.1
Update Date:	2018-04-17
Keywords:

Solution Type Technical Instruction Sure

Solution 1002807.1 : Sun SPARC[TM] Enterprise M4000 and M5000 Server Device Paths

Applies to:

Sun SPARC Enterprise M4000 Server - Version Not Applicable and later
Sun SPARC Enterprise M5000 Server - Version Not Applicable and later
All Platforms

Goal

The intent of this document is to provide the reader with an overview of how the Sun SPARC(R) Enterprise M4000 and M5000 system device paths are arranged.

Solution

There are two attributes that impact how the Sun SPARC Enterprise M4000 and Sun SPARC Enterprise M5000 device paths are arranged. The attributes are the I/O architecture and the assignment of the Logical System Board (LSB). When allocating resources to a domain through the Domain Control List (DCL), OPL's access control list, these resources are also assigned to a Logical System Board (LSB).

Example LSB assignment:

XSCF> showboards -va
XSB R DID(LSB) Assignment Pwr Conn Conf Test    Fault    COD
---- - -------- ----------- ---- ---- ---- ------- -------- ----
00-0   00(00)   Assigned    y    y    y    Passed Normal   n
01-0   01(00)   Assigned    y    y    y    Passed Normal   n
01-1   01(01)   Assigned    y    y    y    Passed Normal   n
01-2   01(02)   Assigned    y    y    y    Passed Normal   n
01-3   00(01)   Assigned    y    y    y    Passed Normal   n
XSCF>

More information on LSB's can be found in Document 1011446.1 Sun SPARC Enterprise Mx000 Servers Logical System Board (LSB) and Sun SPARC Enterprise M3000/M4000/M5000/8000/M9000 Server Administration Guide.

The following is a graphical representation of the Sun SPARC Enterprise M4000 and Sun SPARC Enterprise M5000 IOU0 device path layout:

|----- /pci@[LSB1]0,600000/pci@0 (Base Path)

|-- ../pci@8/pci@0/scsi@1/disk0 IOU0 HD0

|-- ../pci@8/pci@0/scsi@1/disk1 IOU0 HD1

|-- ../pci@8/pci@0/scsi@1/disk3 DVD (IOU0 only)

|-- ../pci@8/pci@0/scsi@1/tape DAT (IOU0 only)

|-- ../pci@8/pci@0/network@2 1st IOU network port (Solaris bge instance)

|-- ../pci@8/pci@0/network@2,1 2nd IOU network port (Solaris bge instance)

|-- ../pci@8/pci@0,1 Slot 0 (PCI-X)

|-- ../pci@9 Slot 1 (PCIe)

|----- /pci@[LSB1]1,700000 Slot 2 (PCIe)

|----- /pci@[LSB2]2,600000 Slot 3 (PCIe)

|----- /pci@[LSB2]3,700000 Slot 4 (PCIe)

Note: When a Physical System Board (PSB) is in UniMode, [LSB1] it is the same as [LSB2]. When the PSB is placed in QuadMode, [LSB1] it is NOT the same as [LSB2] when the resources are in the same domain. It is possible however that when in QuadMode, [LSB1] could be the same as [LSB2] when the resources are in different domains.

The Sun SPARC Enterprise M5000 IOU1 device path layout is very similar to the IOU0 device path with the exception that there is no DVD and DAT connected to IOU1. The following is a graphical representation of the Sun SPARC Enterprise M5000 IOU1 device path layout:

|----- /pci@[LSB3]0,600000/pci@0 (Base Path)

|-- ../pci@8/pci@0/scsi@1/disk0 IOU1 HD2

|-- ../pci@8/pci@0/scsi@1/disk1 IOU1 HD3

|-- ../pci@8/pci@0/network@2 1st IOU network port (Solaris bge instance)

|-- ../pci@8/pci@0/network@2,1 2nd IOU network port (Solaris bge instance)

|-- ../pci@8/pci@0,1 Slot 0 (PCI-X)

|-- ../pci@9 Slot 1 (PCIe)

|----- /pci@[LSB3]1,700000 Slot 2 (PCIe)

|----- /pci@[LSB4]2,600000 Slot 3 (PCIe)

|----- /pci@[LSB4]3,700000 Slot 4 (PCIe)

Note: Just as with IOU0, when a Physical System Board (PSB) is in UniMode, the IOU1 [LSB3] is the same as [LSB4]. When the PSB is placed in QuadMode, IOU1 [LSB3] is NOT the same as [LSB4] when the resources are in the same domain. It is possible however that when in QuadMode, IOU1 [LSB3] could be the same as [LSB4] when the resources are in different domains.

The following table represents the same device path information presented in a table format along with the UniMode and QuadMode assignments:

Sun SPARC Enterprise M4000 and Sun SPARC Enterprise M5000 IOU0 Table:

Device	Type	UniMode	QuadMode	Device Path
IOU0 HD0	SAS/SATA	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/scsi@1/disk0
IOU0 HD1	SAS/SATA	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/scsi@1/disk1
CD/DVD	SCSI	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/scsi@1/disk3
DAT	SCSI	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/scsi@1/tape
net0	10/100/1000 HDX/FDX	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/network@2
net1	10/100/1000 HDX/FDX	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0/network@2,1
IOU Slot 0	PCI-X	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@8/pci@0,1
IOU Slot 1	PCIe	XSB XX-0	XSB XX-0	/pci@[LSB1]0,600000/pci@0/pci@9
IOU Slot 2	PCIe	XSB XX-0	XSB XX-0	/pci@[LSB1]1,700000
IOU Slot 3	PCIe	XSB XX-0	XSB XX-1	/pci@[LSB2]2,600000
IOU Slot 4	PCIe	XSB XX-0	XSB XX-1	/pci@[LSB2]3,700000

Sun SPARC Enterprise M5000 IOU1 Table:

Device	Type	UniMode	QuadMode	Device Path
IOU1 HD2	SAS/SATA	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@8/pci@0/scsi@1/disk0
IOU1 HD3	SAS/SATA	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@8/pci@0/scsi@1/disk1
net0	10/100/1000 HDX/FDX	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@8/pci@0/network@2
net1	10/100/1000 HDX/FDX	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@8/pci@0/network@2,1
IOU Slot 0	PCI-X	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@8/pci@0,1
IOU Slot 1	PCIe	XSB XX-0	XSB XX-0	/pci@[LSB3]0,600000/pci@0/pci@9
IOU Slot 2	PCIe	XSB XX-0	XSB XX-0	/pci@[LSB3]1,700000
IOU Slot 3	PCIe	XSB XX-0	XSB XX-1	/pci@[LSB4]2,600000
IOU Slot 4	PCIe	XSB XX-0	XSB XX-1	/pci@[LSB4]3,700000

A significant piece of the understanding the complete device path is knowing which LSB the XSB resource has been assigned. Allowing a XSB resource to be part of a domain through the DCL at the same time assigning the resource to a LSB has the following impact on device path numbering (ie. the [LSB1] and [LSB2] values of the device path):

LSB Assignment	LSB1-4 Value	LSB Assignment	LSB1-4 Value
0	No Value	8	8
1	1	9	9
2	2	10	a
3	3	11	b
4	4	12	c
5	5	13	d
6	6	14	e
7	7	15	f

A quick method to look up the location of a PCI device is to cross reference the device path to the output of "fmtopo -V".

# /usr/sbin/prtdiag -v
System Configuration: Oracle Corporation sun4u SPARC Enterprise M5000 Server
System clock frequency: 1012 MHz
Memory size: 131072 Megabytes
....
00 PCIe 2       3, fc40, 10df       2, 0, 0 okay     8,    8 SUNW,emlxs-pciex10df,fc40      LPe12002-S
    /pci@2,600000/SUNW,emlxs@0

00 PCIe 2       3, fc40, 10df       2, 0, 1 okay     8,    8 SUNW,emlxs-pciex10df,fc40      LPe12002-S
    /pci@2,600000/SUNW,emlxs@0,1

# /usr/lib/fm/fmd/fmtopo -V
hc://:product-id=SUNW,SPARC-Enterprise:server-id=kaz-cust:chassis-id=BDF123522F/chassis=0/ioboard=0/hostbridge=1/pciexrc=0/pciexbus=2/pciexdev=0/pciexfn=0
group: protocol                       version: 1   stability: Private/Private
    resource          fmri      hc://:product-id=SUNW,SPARC-Enterprise:server-id=kaz-cust:chassis-id=BDF123522F/chassis=0/ioboard=0/hostbridge=1/pciexrc=0/pciexbus=2/pciexdev=0/pciexfn=0
    label             string    iou#0-pci#3
    FRU               fmri      hc:///component=iou#0-pci#3
    ASRU              fmri      dev:////pci@2,600000/SUNW,emlxs@0
.....
hc://:product-id=SUNW,SPARC-Enterprise:server-id=kaz-cust:chassis-id=BDF123522F/chassis=0/ioboard=0/hostbridge=1/pciexrc=0/pciexbus=2/pciexdev=0/pciexfn=1
group: protocol                       version: 1   stability: Private/Private
    resource          fmri      hc://:product-id=SUNW,SPARC-Enterprise:server-id=kaz-cust:chassis-id=BDF123522F/chassis=0/ioboard=0/hostbridge=1/pciexrc=0/pciexbus=2/pciexdev=0/pciexfn=1
    label             string    iou#0-pci#3
    FRU               fmri      hc:///component=iou#0-pci#3
    ASRU              fmri      dev:////pci@2,600000/SUNW,emlxs@0,1

For more information on Sun SPARC Enterprise M8000/M9000 device paths, see Document 1004116.1 Sun SPARC Enterprise M8000 and M9000 Device Paths.
For more information on External I/O Expansion Unit device paths, see Document 1005332.1 Sun SPARC External I/O Expansion Unit Device Paths.

Actual device paths may be seen using some of the following methods:
OBP: show-devs command
Solaris: /etc/path_to_inst file
Solaris: prtpicl command
Solaris: prtdiag -v command

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - M Series Servers

Internal References

Sun SPARC Enterprise Mx000 Servers - Deep Dive

OPL Tools IO Layout

References

<NOTE:1005332.1> - Sun SPARC(R) External I/O Expansion Unit Device Paths
<NOTE:1011446.1> - Sun SPARC Enterprise M4000/M5000/M8000/M9000 Servers : Logical System Board (LSB)
<NOTE:1019730.1> - Sun SPARC(R) Enterprise M3000 and M3000-E (SPARC VII+) Server Device Paths
http://docs.oracle.com/cd/E19855-01/E21618/index.html
<NOTE:1004116.1> - Sun SPARC(R) Enterprise M8000 and M9000 Device Paths

Attachments

This solution has no attachment