Description

SPARC Solaris sun4v domains with certain firmware versions (as listed below) may panic after 1101 days of uptime. 

Occurrence

This "1101 days of uptime" panic was caused by a change introduced in Hypervisor version 1.12 which was only released on the following systems:

SPARC Platform

• SPARC T4 Servers with firmware version 8.4.0 through 8.9.5
• SPARC T5/M5/M6 Servers with firmware version 9.0 through 9.6.6
• SPARC T7/M7 Servers with firmware version 9.7.1 through 9.7.3
• SPARC S7 Servers with firmware version 9.7.2 through 9.7.4
• Netra SPARC S7 Servers with firmware version 9.7.2 through 9.7.3

Notes:

1. The Solaris command 'uname -i' can be used to display the machine implementation.

2. To determine the firmware version installed on the server, use the following ILOM command:

      -> show /HOST sysfw_version

      /HOST
      Properties:
      sysfw_version = Sun System Firmware 9.5.4.b 2016/03/24 21:39

Symptoms

If the described issue occurs, the domain will panic. Various types of panic might be evident in the HOST console log, including "panic: send_mondo_set: timeout". For example:

      send mondo timeout [retries: 0xb0acc] cpuids: 0xfa
      panic: failed to stop cpu250
      panic[cpu228]/thread=3018ced3180: send_mondo_set: timeout

The ILOM event log (-> show /SP/logs/event/list) may have sufficient history to check the HOST uptime.
Look for Host "Powered on" or "HV started".

Example (event log):

      225 Thu Jan 26 14:17:22 2017 System Log minor
      Host: Solaris panicking      <======================panic date/time
      <snip>
      199 Tue Jan 21 13:57:43 2014 System Log minor
      Host: Host started
      198 Tue Jan 21 13:57:39 2014 System Log minor
      Host: HV started      <======================HV start date/time
      197 Tue Jan 21 13:49:00 2014 System Log minor
      Host: Powered On

The Solaris GNU date command can be used to easily calculate a date 1101 days earlier from the panic date.

Example:

      % /usr/gnu/bin/date -d "Jan 26 2017 - 1101 days"
      Tuesday, January 21, 2014 12:00:00 AM PST

Note: Many date calculators are available via Internet search that display the duration between two dates. If the period of run time from HOST start to panic time is 1101 days, then Bug 23193383 has likely been manifested. Oracle

In addition, Oracle Support can analyze snapshot HOST status logs for confirmation.

Workaround

The HOST must be stopped and restarted before 1101 days of uptime to prevent domain panic. Domain reboot will not suffice. The HOST (or HOSTs, for multi-domain servers) must be stopped (powered off) and then restarted.

Resolution

This issue is addressed on the following platforms:

SPARC Platform

• T4 Servers with firmware version 8.9.8 (as delivered in patch 152477-04) or later
• T5 Servers with firmware version 9.6.7.a (as delivered in patch 25389439) or later
• M5/M6 Servers with firmware version 9.6.7.a (as delivered in patch 25389444) or later
• T7 Servers with firmware version 9.7.4 (as delivered in patch 25373796) or later
• S7 Servers with firmware version 9.7.5.b (as delivered in patch 25790079) or later
• Netra S7 Servers with firmware version 9.7.4 (as delivered in patch 25373804) or later
• M7 Servers with firmware version 9.7.4 (as delivered in patch 27185996) or later

Note: Loading new system firmware requires the HOST to be stopped and restarted to deploy firmware fixes resident in HOST hardware. See the relevant server Administration Guide for details regarding firmware upgrade.

Patches

<Patch:152477-04>, <Patch:25389439>, <Patch:25389444>, <Patch:25373796>,
<Patch:25790079>, <Patch:25373804>, <Patch:27185996>

History

05-Jan-2018: Document released, status is Resolved
16-Jan-2018: Updated "Occurrence" for clarification to servers affected

This is software problem. Various periodic operations scheduled by hypervisor
may manifest failure in indeterministic ways, including panic.

Questions regarding any portion of this document should be
addressed to sunalertpublication_us_grp@oracle.com and
copy the submitter/responsible engineer listed below.

Internal Contributor/Submitter: david.lafko@oracle.com
Internal Eng Responsible Engineer: greg.onufer@oracle.com
Oracle Knowledge Analyst: david.mariotto@oracle.com
Internal Eng Business Unit Group: Systems Server OS
Internal Escalation ID: 3-16183317981
Internal Pending Patches: None
Internal Resolution Patches: 152477-04, 25389439, 25389444, 25373796, 25790079, 25373804, 27185996

References

This solution has no attachment