Description

When multiple LDOMs are configured to share VCPU resources on a single SCC (SPARC Cache Clusters are comprised of 32 VCPUs), instructions on those VCPs will execute at only 25% of expected performance.

Occurrence

This issue can occur on the following platforms:

SPARC Platform

• SPARC T7-1
• SPARC T7-2
• SPARC T7-4
• SPARC M7-8
• SPARC M7-16
• SPARC S7-2
• SPARC S7-2L
• SPARC Netra S7-2

when any of the above systems are running any currently released System Firmware version, and with the system Power Management policy set to 'performance'.

To check the Power Management setting, use the following ILOM command:

      --> show /Servers/PDomains/PDomain_0/SP/powermgmt policy

For the M7 platform:

      --> show Servers/PDomains/PDomain_<domain ID>/SP/powermgmt/policy

Symptoms

If the described issue occurs, the system will experience longer than expected execution times.

To check a system for affected VCPUs, a performance test can be run.

For example, the following shell script will allow detection of slower VCPUs:

      #!/bin/sh

      CMD="dd if=/dev/random of=/dev/null bs=8 count=51200"
      CPUS=`psrinfo | cut -f1`
      for i in $CPUS ; do
      echo $i "\c"
      pbind -b $i $$ >/dev/null
      sh -c "time -p $CMD" 2>&1 | grep real
      done

The output on a relatively unloaded server should look similar to the following:

      0 real 0.2
      1 real 0.2
      2 real 0.3
      3 real 0.4
      4 real 0.2
      ...

Affected VCPUs will show significantly higher values, as in the following example:

      32 real 0.9

Workaround

 This issue can be avoided or worked around using any one of the following:

1) Assign VCPUs to guest domains in multiple of 32 (i.e: 32, 64, 96, etc).

Or:

2) Disable Power Management

      To disable Power Management, use the following ILOM command:

      --> set /SP/powermgmt/ policy=disabled

      or for M7 systems:

      --> set Servers/PDomains/PDomain_<affected domain ID>/SP/powermgmt/policy=disabled

Note: A power cycle is required once the policy is set to disabled for this change to take effect.

Or:

3) Toggle Power Management to Disabled and then back to the desired setting

Note: The above workaround (3) will need to be repeated if the guest domain is rebooted.

Resolution

A final resolution is pending completion.

History

13-Jul-2017: Document released, status is Workaround
21-Jul-2017: Minor correction to command in "Occurrence" and script quotes in "Symptoms" - no change in content
02-Oct-2017: Updated to add Bug 26718819
28-Nov-2017: Added "Note" to Option 2 in Workaround

The tracking bug for this issue is now Bug 26718819 - please read both Bugs for info

Questions regarding any portion of this document must be submitted to
sunalertpublication_us_grp@oracle.com and copy the
contributor/responsible engineer listed below.

Internal Contributor/Submitter: Grant.Gredvig@oracle.com
Internal Eng Responsible Engineer: Grant.Gredvig@oracle.com
Oracle Knowledge Analyst: david.mariotto@oracle.com
Internal Eng Business Unit Group: ServerOS
Internal Escalation ID: 3-14094430651 3-14748429891 3-15087414131 3-15160868741 3-15071948141
Internal Pending Patches: TBD

Workaround Notes (Justin):

Disabling power management provides a completely automated solution until we can identify a permanent fix.

To implement the workaround on a system where guests have already been configured we need to shut down any running domains, then save the configuration before cycling the platform with power management set to disabled;

[Solaris - Guest Domain(s

<halt Solaris on the gdom(s)>

[Solaris - Control Domain]

ldm list-constraints -x <gdom> > <gdom>.xml // this is a precaution in case we need to recreate the domain
ldm stop <gdom>
ldm unbind <gdom>
ldm add-spconfig <config_name> // we are reseting /SYS so will load the last active configuration
<halt control domain>

[SP]

set /SP/powermgmt policy=disabled
stop /SYS
start /SYS

[Solaris - Control Domain]

<boot Solaris on the cdom>
ldm bind <gdom>
ldm start <gdom>

[Solaris - Guest Domain(s)]

<boot Solaris on the gdom(s)>
Test performance

References

This solution has no attachment