hwmgmtd could cause high CPU usage on Solaris 11.2 when snmpd runs with non root user

Asset ID:	1-72-2104705.1
Update Date:	2017-03-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2104705.1 : hwmgmtd could cause high CPU usage on Solaris 11.2 when snmpd runs with non root user

Applies to:

Sun Server X3-2L - Version All Versions and later
Information in this document applies to any platform.

Symptoms

The below is a part of prstat(1M) and truss(1) command for hwmgmtd. prstat command indicates hwmgmtd is constantly running on a cpu.

Also, truss command will show accept() with EAGAIN error for file descriptior#9 which is one of socket descriptor.

: prstat (prstat -L -c -p <hwmgmtd_pid>) :
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
1290 root 51M 9732K cpu4 0 0 2004:23:2 9.4% hwmgmtd/2
1290 root 51M 9732K sleep 59 0 0:29:19 0.0% hwmgmtd/4
1290 root 51M 9732K sleep 59 0 0:09:21 0.0% hwmgmtd/3
1290 root 51M 9732K sleep 58 0 0:00:03 0.0% hwmgmtd/1
Total: 1 processes, 4 lwps, load averages: 0.93, 1.03, 1.05
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
1290 root 51M 9732K cpu7 0 0 2004:23:5 12% hwmgmtd/2
1290 root 51M 9732K sleep 59 0 0:29:19 0.0% hwmgmtd/4
1290 root 51M 9732K sleep 59 0 0:09:21 0.0% hwmgmtd/3
1290 root 51M 9732K sleep 58 0 0:00:03 0.0% hwmgmtd/1
Total: 1 processes, 4 lwps, load averages: 1.00, 1.04, 1.05

: truss (truss -faeld -o truss.out -p <hwmgmtd_pid> ):
1290/1: psargs: /usr/lib/ssm/hwmgmtd
1290/2: 0.0030 recv(8, 0xFB8CD9A4, 4, 0) Err#9 EBADF
1290/2: 0.0032 accept(9, 0x00000000, 0x00000000, SOV_DEFAULT) Err#11 EAGAIN
1290/2: 0.0033 pollsys(0xFB8CD860, 2, 0xFB8CD928, 0x00000000) = 1
1290/2: 0.0034 recv(8, 0xFB8CD9A4, 4, 0) Err#9 EBADF
1290/2: 0.0035 accept(9, 0x00000000, 0x00000000, SOV_DEFAULT) Err#11 EAGAIN
1290/2: 0.0036 pollsys(0xFB8CD860, 2, 0xFB8CD928, 0x00000000) = 1
1290/2: 0.0037 recv(8, 0xFB8CD9A4, 4, 0) Err#9 EBADF
1290/2: 0.0038 accept(9, 0x00000000, 0x00000000, SOV_DEFAULT) Err#11 EAGAIN
1290/2: 0.0039 pollsys(0xFB8CD860, 2, 0xFB8CD928, 0x00000000) = 1
1290/2: 0.0040 recv(8, 0xFB8CD9A4, 4, 0) Err#9 EBADF
1290/2: 0.0041 accept(9, 0x00000000, 0x00000000, SOV_DEFAULT) Err#11 EAGAIN
1290/2: 0.0042 pollsys(0xFB8CD860, 2, 0xFB8CD928, 0x00000000) = 1
1290/2: 0.0042 recv(8, 0xFB8CD9A4, 4, 0) Err#9 EBADF
1290/2: 0.0043 accept(9, 0x00000000, 0x00000000, SOV_DEFAULT) Err#11 EAGAIN
1290/2: 0.0044 pollsys(0xFB8CD860, 2, 0xFB8CD928, 0x00000000) = 1

: pfiles (pfiles <hwmgmtd_pid>) :
1290: /usr/lib/ssm/hwmgmtd
Current rlimit: 65535 file descriptors

5: S_IFREG mode:0644 dev:557,1 ino:93587795 uid:0 gid:0 size:5
O_RDWR|O_CREAT|O_EXCL FD_CLOEXEC
/system/volatile/ssm-hwmgmtd.pid
offset:5
7: S_IFDOOR mode:0444 dev:560,0 ino:47 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[721]
/system/volatile/name_service_door
9: S_IFSOCK mode:0666 dev:562,0 ino:62057 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX /var/run/ssm-hwmgmtd.socket
:

Changes

This issue happens when snmpd is being run as a non-root user with the "-u" option like below.

: ps -ef|grep snmpd :
snmpd 1283 1 0 Jul 22 ? 51:27 /usr/sbin/snmpd -u 100

Cause

snmpd with non root user does not have a permission to establish a connection to hwmgmtd. However, hwmgmtd is not proper error handling.

Solution

The fix has been cotained in OHMP 2.4 (Solaris 11.3 SRU18.2 or later).

If unable to upgrade to the fixed OHMP version, there are two workarouns for this issue.

1. Run snmpd as root user without "-u" option.

2. If snmpd runs as non-root, disable the SNMP functionality of hwmgmtd.
This can be done by adding "request_thread_enabled=0" to the /etc/ssm/hwmgmtd.conf file.

References

<BUG:22305376> - HWMGMTD: HIGH CPU USAGE ON SOLARIS 11.2 SRU12.6.
<BUG:21087125> - HWMGMTD: HIGH CPU USAGE AFTER PATCHED SOLARIS 11.2 SRU2

Attachments

This solution has no attachment