ILOM restart due to memory leak and causes devfsadmd, ldc, and usb related error or warning messages in Solaris

Asset ID:	1-72-1626240.1
Update Date:	2017-10-18
Keywords:

Solution Type Problem Resolution Sure

Solution 1626240.1 : ILOM restart due to memory leak and causes devfsadmd, ldc, and usb related error or warning messages in Solaris

Applies to:

SPARC T4-1
Oracle Database - Enterprise Edition - Version 11.2.0.2 to 11.2.0.2 [Release 11.2]
Information in this document applies to any platform.

Symptoms

On systems with old System Firmware, you may see the following devfsadmd, ldc, and usb related error or warning logs in Solaris /var/adm/messages :

devfsadmd[431]: failed to lookup dev name for /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@1/disk@0,0
ldc: [ID 717221 kern.warning] WARNING: ldc_open: (0xb) channel rx queue unconf failed
genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/communications@3 (usbecm0) removed

For example,

Feb 14 00:32:09 my-server mac: [ID 736570 kern.info] NOTICE: usbecm0 unregistered
Feb 14 00:32:09 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/communications@3 (usbecm0) removed
Feb 14 00:32:10 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4 (usb_mid4) removed
Feb 14 00:32:10 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3 (hubd2) removed
Feb 14 00:32:10 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@2 (scsa2usb1) removed
Feb 14 00:32:10 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@2/disk@0,0 (sd4) removed
Feb 14 00:32:12 my-server ldc: [ID 717221 kern.warning] WARNING: ldc_open: (0xb) channel rx queue unconf failed
Feb 14 00:32:14 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4/keyboard@0 (hid2) offline
Feb 14 00:32:14 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4/mouse@1 (hid0) offline

:
Feb 14 00:32:48 my-server scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@1/disk@0,0 (sd3):
Feb 14 00:32:48 my-server Command failed to complete...Device is gone
Feb 14 00:32:50 my-server devfsadmd[431]: [ID 937045 daemon.error] failed to lookup dev name for /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/storage@1/disk@0,0

And a few minutes later the devices get back online with the following messages.

Feb 14 00:34:44 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4/keyboard@0 (hid2) online
Feb 14 00:34:44 my-server usba: [ID 912658 kern.info] USB 1.10 interface (usbif46b,ff10.config1.1) operating at low speed (USB 1.x) on USB 2.0 external hub: mouse@1, hid0 at bus address 3
Feb 14 00:34:44 my-server usba: [ID 349649 kern.info] American Megatrends Inc. Virtual Keyboard and Mouse
Feb 14 00:34:44 my-server genunix: [ID 936769 kern.info] hid0 is /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4/mouse@1
Feb 14 00:34:44 my-server usba: [ID 723738 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2 (ehci0): Low speed end point's poll interval of 1 ms is below threshold. Rounding up to 8 ms
Feb 14 00:34:44 my-server last message repeated 2 times
Feb 14 00:34:44 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/device@4/mouse@1 (hid0) online
Feb 14 00:34:55 my-server mac: [ID 469746 kern.info] NOTICE: usbecm0 registered
Feb 14 00:34:55 my-server usba: [ID 912658 kern.info] USB 2.0 device (usb430,a4a2) operating at hi speed (USB 2.x) on USB 2.0 external hub: communications@3, usbecm0 at bus address 6
Feb 14 00:34:55 my-server usba: [ID 349649 kern.info] SunMicro Virtual Eth Device
Feb 14 00:34:55 my-server genunix: [ID 936769 kern.info] usbecm0 is /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/communications@3
Feb 14 00:34:55 my-server genunix: [ID 408114 kern.info] /pci@400/pci@2/pci@0/pci@f/pci@0/usb@0,2/hub@2/hub@3/communications@3 (usbecm0) online

In ILOM SEL (System Event Log), ILOM reports that a "Warm start" had occurred.

8821   Fri Feb 14 00:34:57 2014 System    Log       minor
       Host: Solaris running
8820   Fri Feb 14 00:34:57 2014 Chassis   Action    minor
       Inventory has been updated starting at node '/SYS'
8819   Fri Feb 14 00:34:47 2014 System    Log       minor
       Host: Host started
8818   Fri Feb 14 00:34:40 2014 System    Log       minor
       Host: HV started
8817   Fri Feb 14 00:34:40 2014 Chassis   Action    minor
       Inventory has been updated starting at node '/SYS/MB'
8816   Fri Feb 14 00:34:37 2014 System    Log       minor
       Host: Warm start
8815   Fri Feb 14 00:29:41 2014 Audit     Log       minor
       root : Close Session : object = "/SP/session/type" : value = "shell" : su
       ccess
8814   Fri Feb 14 00:06:26 2014 Audit     Log       minor
       root : Open Session : object = "/SP/session/type" : value = "shell" : suc
       cess

ILOM snapshot vbsc logs will contain the following warning messages and a restart.

Feb 14 00:08:08 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:09:11 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:10:14 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:11:17 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:12:22 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:13:11 DEBUG:   snmp_thr: exhausted retries
Feb 14 00:14:16 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:15:21 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:16:26 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:17:29 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:18:40 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
...
Feb 14 00:31:42 WARNING: snmp_data_handler: timed out waiting for request in
                         progress, dropping new request
Feb 14 00:34:26 NOTICE: Build Version: GM 1.1.4.a
Feb 14 00:34:26 NOTICE: Build User: cboland
Feb 14 00:34:26 NOTICE: Build Date: Jan 14 2012, 15:59:35
Feb 14 00:34:26 NOTICE: Build Path:
                         /re/wa-builds-3/sysfw-gates/8.1/8.1.4.e/components/nyx/
                         src/gm/yf/release
Feb 14 00:34:26 NOTICE: Build Machine: sanpen-cs10-0.West.Sun.COM
Feb 14 00:34:26 NOTICE: Checking Flash File System
Feb 14 00:34:27 DEBUG:   Pushing LDom configs info to SP.
Feb 14 00:34:27 NOTICE: Using FPGA Interrupts
Feb 14 00:34:28 DEBUG:   bstore_get_var: Couldn't find service variables
Feb 14 00:34:28 DEBUG:   bstore_get_var: Couldn't find service variables
Feb 14 00:34:28 DEBUG:   init_bootmode_vars
Feb 14 00:34:33 DEBUG:   Loaded static ASR DB data for 142 components. Ver. 3
Feb 14 00:34:36 NOTICE: Warm Start: Processing Mbox Messages on Node 0
...

Cause

Although we do not see explicit "System restarted due to Out-Of-Memory condition" log, this ILOM warm restart is the result of memory leak in the ILOM software.

Solution

Please update system firmware 8.3.0.b (or later).

References

<BUG:15619601> - SUNBT6922507-X64_3.0.9 POD MEMORY LEAK
<BUG:15733002> - SUNBT7074596 ILOM REBOOTS ITSELF AND GETS "SYSTEM RESTARTED DUE TO OUT-OF-MEMORY
<NOTE:1483873.1> - Service Processor is Faulted and Not Accessible (SPARC T3-1/T3-2/T3-4/T3-1B/T4-1/T4-2/T4-4/T4-1B/Netra T3-1/Netra T3-1B/Netra T4-1/Netra T4-2)

Attachments

This solution has no attachment