ILOM events not captured in OS

Asset ID:	1-72-1517832.1
Update Date:	2018-01-16
Keywords:

Solution Type Problem Resolution Sure

Solution 1517832.1 : ILOM events not captured in OS

Applies to:

SPARC T4-4 - Version All Versions to All Versions [Release All Releases]
SPARC T4-1 - Version Not Applicable and later
SPARC T4-2 - Version Not Applicable and later
Information in this document applies to any platform.

Symptoms

The customer reported that the ilom events are not getting generated in the /var/adm/messages of the OS

Changes

No recent changes, they have restarted the OS.

Cause

In Solaris, an FMD plugin called the Event Transport Module (ETM) subscribes to selected fault events and transports them to ILOM.
ILOM then updates its state and view of the components in the system and for faults received from Solaris, ILOM will also generate an SNMP trap.
All the event lists in the ilom are sent to the OS messages file

If the module is not responding or hung, the event will not be reported by the fmd.

Solution

Tested in Lab
==============================================================

bash-3.00# svcs -a | grep fmd
online          1:04:43 svc:/system/fmd:default
bash-3.00# svcadm restart /system/fmd
bash-3.00# fmadm config | grep etm
MODULE                   VERSION STATUS DESCRIPTION
etm                      1.2     active FMA Event Transport Module

->
bash-3.00# fmadm unload etm
fmadm: module 'etm' unloaded from fault manager

bash-3.00# fmadm unload syslog-msgs
fmadm: module 'syslog-msgs' unloaded from fault manager

bash-3.00# fmadm config | grep etm
bash-3.00#

Check the Event logs in the messages files and it is not captured.

-> set /SYS/LOCATE/ value=Fast_Blink
Set 'value' to 'Fast_Blink'

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started. To stop, type #.

bash-3.00# tail -f /var/adm/messages

Jan 2 00:53:50 nt3-1-ftlauder-a SC Alert: [ID 738335 daemon.notice] Audit | minor: root : Set : object = "/SP/logs/event/clear" : value = "true" : success
Jan 2 00:54:17 nt3-1-ftlauder-a SC Alert: [ID 655827 daemon.notice] Audit | minor: root : Set : object = "/SYS/LOCATE/value" : value = "Off" : success
Jan 2 00:56:00 nt3-1-ftlauder-a sshd[4882]: [ID 800047 auth.crit] fatal: Timeout before authentication for 10.79.214.214

Started the Fmd service

bash-3.00# svcs -a | grep fmd
online          1:04:43 svc:/system/fmd:default
bash-3.00# svcadm restart /system/fmd

bash-3.00# fmadm config | grep etm
MODULE                   VERSION STATUS DESCRIPTION
etm                      1.2     active FMA Event Transport Module
bash-3.00#

-> set /SYS/LOCATE/ value=Off
Set 'value' to 'Off'

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started. To stop, type #.

The logs have started getting captured in the system.
Jan 2 00:56:00 nt3-1-ftlauder-a sshd[4882]: [ID 800047 auth.crit] fatal: Timeout before authentication for 10.79.214.214
Jan 2 01:09:22 nt3-1-ftlauder-a SC Alert: [ID 594792 daemon.notice] Audit | minor: root : Set : object = "/SYS/LOCATE/value" : value = "Fast_Blink" : success <<< old log which was not captured before the module was disabled
Jan 2 01:09:52 nt3-1-ftlauder-a SC Alert: [ID 655827 daemon.notice] Audit | minor: root : Set : object = "/SYS/LOCATE/value" : value = "Off" : success <<<< recent log

===========================

Action Plan/Workaround
- Check for the etm service running on the OS or not.
- Restart the fmd service
- Perform a test event in ilom and check the same in the OS messages

Attachments

This solution has no attachment