On SPARC T5-2, removing bus with usbecm device (pci_1 / pci@340) from primary/control domain can impact FMA and ldmd processing

Asset ID:	1-72-1948940.1
Update Date:	2017-11-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1948940.1 : On SPARC T5-2, removing bus with usbecm device (pci_1 / pci@340) from primary/control domain can impact FMA and ldmd processing

Applies to:

Solaris Operating System - Version 10 3/05 and later
SPARC T5-2 - Version All Versions and later
Oracle Solaris on SPARC (64-bit)

Symptoms

If a SPARC T5-2 is configured with multiple root domains, and bus pci_1 (pci@340) is removed from the control domain, the control domain will not have access to the usbecm device. The logical domain daemon uses this device to access the proxy to receive CPU and memory FMA events from the Service Processor. This note is specific to the SPARC T5-2 system, but similar considerations apply to SPARC M5-32 and M6-32 servers. Please see DocID 1942045.1 for details on those servers. Note that this pertains to T5-2 servers with two CPU sockets. The T5-2 single socket server has two buses (pci@340/pci_1, pci@300/pci_0) which are both needed by the control domain, so a separate root domain cannot be created.

Changes

This situation occurs if the bus containing usbecm device is removed from the control domain. The available buses, and the bus used for this device, should be determined before making bus assignments by issuing the following commands. The first command lists the names of the buses and their pseudonyms. All of the commands shown here are run from the control domain.

# ldm list-devices -a io
IO
    DEVICE                           PSEUDONYM        BOUND   OPTIONS
    pci@340                          pci_1            yes
    pci@300                          pci_0            yes
    pci@3c0                          pci_3            yes
    pci@380                          pci_2            yes

The next command shows the current bus assignments.

# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
pci_1                                     BUS    pci_1    primary
pci_0                                     BUS    pci_0    primary
pci_3                                     BUS    pci_3    primary
pci_2                                     BUS    pci_2    primary
...snip...

The next two commands determine which bus is used for the usbecm device.

# ls -l /dev | grep usbecm
lrwxrwxrwx 1 root root 32 Nov 21 10:15 usbecm -> ../devices/pseudo/clone@0:usbecm
lrwxrwxrwx 1 root root 73 Nov 21 10:15 usbecm0 -> ../devices/pci@340/pci@1/pci@0/pci@3/usb@0/hub@5/communications@2:usbecm0

# grep usbecm /etc/path_to_inst
"/pci@340/pci@1/pci@0/pci@3/usb@0/hub@5/communications@2" 0 "usbecm"

The output from these commands indicates that the usbecm device is on bus pci@340, which has the pseudonym pci_1.

Cause

The usbecm USB network device is used for communication between the Service Processor (SP) and Solaris running in the control domain. This device is on pci_1 (pci@340) (CM0/IOS0). Removing this bus from the control domain removes the communication path.

Solution

Use the above commands to determine which bus has the usbecm device. On a T5-2 that will be bus pci_1. Ensure that root complex assignment leaves pci_1 (pci@340) with the control domain when designing bus configurations.

References

<NOTE:1942045.1> - IO faults proxying in LDOM environment

Attachments

This solution has no attachment