SPARC M5-32
 
SPARC M6-32
 

PLA-Support>Sun Systems>SPARC>Enterprise>SN-SPARC: Mx-32
 

In this Document

Purpose

Details

Role of interconnect

Identify the Interconnect path

Configuration rules

Empty DCU (no CMU installed)

The root complexes used for the usbecm interface

Additional Information

Special case : Golden SPP failover

References

Applies to:

Purpose

This document provides details about the Interconnect, FMA Fault Proxying and LDOM on the SPARC M5-32 and M6-32 Servers.

Details

Role of interconnect

The Interconnect provides an internal communication interface between the Host/Pdom and the service processor.
On M5-32/M6-32, it's actually an interface between the Host (control/primary domain) and the Pdomain-SPP (aka Golden SPP) of the Host.

For M5-32/M6-32, this IP address can be used to connect the Pdomain-SPP. Not the SP like on the Tseries servers.

The Interconnect is using an Ethernet-Over-USB interface (usbecm).

It can be controlled/managed :
- from the SP :
    /Servers/PDomains/PDomain_x/SP/network/interconnect
- from the Solaris Host
    using the 'ilomconfig' command from Solaris or OHMP.

Note : with Solaris 11.1, ilomconfig is bundled with Solaris. The Oracle Hardware Management Pack is not required to configure the Interconnect.

See :

• Oracle ILOM Administrator’s Guide for Configuration and Maintenance Firmware Release 3.2.1 (here)
• SPARC M5-32 and SPARC M6-32 Servers Administration Guide (here)

The Fault Management Architecture is using the Interconnect to proxy the faults diagnosed on the SP/SPP to the Host and to proxy the faults diagnosed on the Solaris Host to the SP/SPP. This is known as FMA Fault Proxying; keeping the FMA faults in sync between the host and the Active SP.

When the interconnect is not available then the FMA Fault Proxying cannot work properly.

Note : when the FMA Fault Proxying mechanism does not work, FMA on SP and Solaris Host still works but the faults diagnosed are no longer proxyed to the other side.

The interconnect is enabled and configured by default and should always be enabled from both the SP and the Host.

Interconnect is using the usbecm interface. On top of it, an IP address is available.
The usbecm interface is using the rootcomplex/bus from CMUx/CMP0 where x is the lowest numbered CMU in the DCU (CMU0 in DCU#0, CMU4 in DCU#1 etc...). So the usbecm interface is using the following rootcomplexes/bus depending on the PDom configuration and Pdomain-SPP role selection :
- DCU#0 - SPP0 : pci_1,
- DCU#1 - SPP1 : pci_17,
- DCU#2 - SPP2 : pci_33,
- DCU#3 - SPP3 : pci_49
See SPARC M5-32 and M6-32 Servers: Device Paths (Doc ID 1540545.1).

When LDOMs are configured, if the usbecm connection is not available to the control domain, FMA Fault Proxying does not work. See below.

Identify the Interconnect path 

• From the SP

It's a communication path to the Pdomain-SPP.

Ex :

• From Solaris

The available physical paths

The IP configuration can be checked as following :

So the Interconnect is using by default the IP address 169.254.182.77 to communicate to the Pdomain-SPP via the IP address 169.254.182.76.

On a multi-DCUs Pdom, any of the SPP can be selected as Pdomain-SPP and this role may change after a stop/start operation.

On a single-DCU Pdom, there is obviously only one SPP available and selected as Pdomain-SPP.

It's possible to identify the path used by the usbecm interface.

On this dual DCU (DCU1 + DCU2) Pdom, SPP2 is the Pdomain-SPP.

And the usbecm interface is using the path to SPP2

As described in SPARC M5-32 and M6-32 Servers: Device Paths (Doc ID 1540545.1), /pci@b40/pci@1/pci@0/pci@1/pci@0/pci@0 is the path to SPP2.

If at some point the Golden SPP role switch to SPP1, the interconnect will use usbecm2 (/pci@740/pci@1/pci@0/pci@1/pci@0/pci@0/usb@0).

Configuration rules

Empty DCU (no CMU installed)

• should not be assigned to a Host (/Servers/PDomains/PDomain_1/HOST dcus_assigned),
• the SPP for the empty DCU might become the Pdomain-SPP at some point,
• because there is no CMU in the empty DCU, the usbecm interface cannot use any root complex,
• this may result in a non-functioning interconnect and so no FMA Fault Proxying.

The root complexes used for the usbecm interface

• must be owned by the control/primary domain (pci_1, pci_17, pci_33, pci_49),
• when the root complexes used for the usbecm interface is assigned to a non-primary domain then FMA Fault Proxying does not work,
• this also applies to multi-DCUs PDom because the Pdomain-SPP may change at some point so the root complexes need to be available on the control domain even after a Pdomain-SPP role change,
• as long as the root complex is owned by the primary domain, it's possible to assign the PCIe endpoint to a non-primary domain; ie. PCIE2 (full configuration) or PCIE1/PCIE2 (half configuration). The card installed in the slot requires to support DIO.
  
  • References :
    
    • Oracle®VMServer for SPARC 3.1 Administration Guide / Creating an I/O Domain by Assigning PCIe EndpointDevices / Howto Create an I/O Domain by Assigning a PCIe EndpointDevice
    • Oracle VM Server for SPARC PCIe Direct I/O and SR-IOV Features (Doc ID 1325454.1)
• unrelated to FMA Fault Proxying, but listed in the context of required root complex ownership -- when the root complexes used for the usbecm interface is assigned to a non-primary domain, storage redirection used for rcdrom boot does not work
  • the rKVMS device path used for storage redirection is also used for the interconnect path described in this document
  • no rcdrom device will be listed nor available in OBP if the required root complexes are not assigned to the control domain

Additional Information

• When the SPARC M5-32 and M6-32 Servers is running SysFW 9.1.1.a or 9.1.1.b, the FMA Fault Proxying may not be running properly because of 17768292 and so because the ip-transport module is not loaded (fmstat -T). The SysFW should be upgraded to the latest version. When properly loaded, you should see

• During the boot sequence, the following warning may be reported

which means that "the driver failed to enumerate the device connected  on port  <number>  of hub. If enumeration fails, disconnect and re-connect."; see man hubd(7D).

You can use the commands above to confirm that the path is used for the Eth-o-USB interface and that the interconnect and FMA Fault proxying are properly configured.

You can also check the /var/adm/messages file and/or host console logs to confirm that, despite the warning, the re-connect was successful

The impact is only on the FMA Fault Proxying.

The FMA ereport forwarding is not impacted; ie. any error detected on the Solaris side and requiring an SP diagnosis to produce a fault (GM/FERG) is still forwarded to the SP via the SPP as it's using a different channel than the Interconnect.

Special case : Golden SPP failover

When a Host is composed of multiple DCUs, one SPP is selected as Pdomain-SPP (Golden SPP). The Golden SPP then hosts the interconnect communication with the Host.
At some point, the Golden SPP role may switch from an SPP to another for several reason.
Golden SPP switch is then reported in the event logs from the Active SP as following :

If the host is up when the Golden SPP failover occurs, from the Solaris /var/adm/messages file on the control domain

A Golden SPP role switch can occur :

1. when restarting (stop/start) the host
2. at any time while host is up&running

1- when restarting (stop/start) the host

If a Golden SPP role switch occurs while restarting the host (ie. Golden SPP changed between the stop and the start operations), Solaris and the respective svc:/network/ilomconfig-interconnect:default will properly initialized the usbecm instance and the interconnect communication path.
No extra step should be required in this case.
The commands as described above can be used to confirmed that it works properly.



2- at any time while host is up&running

If a Golden SPP switch occurs while the host is running Solaris, the interconnect communication may not be working anymore.
The former usbecm interface will be down and removed

The new usbecm interface will be brought back up

And the new interface is reported via dladm.

But the interface is not yet fully functional and must be configured manually

The commands as described above can be used to confirmed that it works properly.

At this point, you can restart ('svcadm restart' or 'svcadm disable/enable') the svc:/network/ilomconfig-interconnect:default service.
You can also disable/enable the interconnect ('ilomconfig disable|enable interconnect').
This should not be required.


Corner case :

If, for some reason, the usbecm interface fails to be registered

All the usbecm instances are reported as removed.

Then restarting the svc:/network/ilomconfig-interconnect:default service will more likely report

As will do, disable/enable interconnect

At this point, it should be possible to manually configure the network interface (ipadm delete-addr/create-addr) as described above.
And FMA Fault Proxying should be able to communicate properly with the SPP.

But it may be recommended to reboot the primary domain so the usbecm interface is properly registered again.


Note that when the interconnect communication is re-establish, you may see a replay for some faults from the SP; ie. SP re-proxying some faults to the host. Fault replay (vs new fault) can be confirmed by checking the UUID (fmdump) and diag-time (fmdump -Vu UUID).

Note : at some point, it will be possible to manually switch the Golden SPP role via the Host initiate_sp_failover property.

set /Hostx initiate_sp_failover=true

After performing a manual Golden SPP failover, the same procedure as described in "2- at any time while host is up&running" should be implemented.

This solution has no attachment