Commands such as drvconfig can hang on fully populated servers : Can't open /dev/ksyms

Asset ID:	1-72-1011677.1
Update Date:	2017-03-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1011677.1 : Commands such as drvconfig can hang on fully populated servers : Can't open /dev/ksyms

Applies to:

Sun Enterprise 4500 Server - Version All Versions and later
Sun Enterprise 5500 Server - Version All Versions and later
Sun Enterprise 3500 Server - Version All Versions and later
Sun Enterprise 6500 Server - Version All Versions and later
All Platforms

Symptoms

Various Solaris[TM] commands can hang on fully configured Ultra[TM] Enterprise Servers running Solaris 2.5.1 and 2.6

Some commands which can hang include:
drvconfig
modinfo
prtconf
netstat
dmesg
crash
adb

Cause

The problem is a direct result of the OS's inability to "open" /dev/ksyms.
The root cause is the exhaustion of kobj symbol space as outlined in Internal BUG ID 14951853 (previously, this was bug number 4100378) "kobj symbol space should grow dynamically."

As more and more drivers are added to a server, this problem becomes more evident, especially in the case of large arrays being added to a server (often seen with EMC arrays) and causing a drvconfig hang situation.
An excellent way to confirm the problem is to compare truss output for the failed command against truss output for the same (successful) command on another system that is running the same OS.

For example, the truss output for a failed dmesg command on a 2.5.1 server looks something like:

dilbert#truss -o /tmp/dmesg dmesg
execve("/usr/sbin/dmesg", 0xEFFFFE68, 0xEFFFFE70) argc = 1
*** SGID: rgid/egid/sgid = 1 / 3 / 3 ***
open("/dev/zero", O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF7C0000
open("/usr/lib/libkvm.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFA80) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
munmap(0xEF796000, 57344) = 0
mmap(0xEF7A4000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 16384) = 0xEF7A4000
close(4) = 0
open("/usr/lib/libelf.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFA80) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 131072, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF760000
munmap(0xEF770000, 57344) = 0
mmap(0xEF77E000, 5328, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 57344) = 0xEF77E000
close(4) = 0
open("/usr/lib/libc.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFA80) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 622592, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF680000
munmap(0xEF700000, 57344) = 0
mmap(0xEF70E000, 29304, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 516096) = 0xEF70E000
mmap(0xEF716000, 5320, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xEF716000
close(4) = 0
open("/usr/lib/libdl.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFA80) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
close(4) = 0
open("/usr/platform/SUNW,Ultra-Enterprise/lib/libc_psr.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFF8A8) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF750000
mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF730000
munmap(0xEF734000, 57344) = 0
mmap(0xEF742000, 5464, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 8192) = 0xEF742000
close(4) = 0
close(3) = 0
munmap(0xEF750000, 8192) = 0
open("/usr/platform/SUNW,Ultra-Enterprise/lib/libkvm_psr.so.1", O_RDONLY) = 3
fstat(3, 0xEFFFFAC8) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 3, 0) = 0xEF750000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF660000
munmap(0xEF666000, 57344) = 0
mmap(0xEF674000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 16384) = 0xEF674000
close(3) = 0
munmap(0xEF750000, 8192) = 0
brk(0x00021320) = 0
Running truss against the same command ( that runs successfully ) on
a box with an identical OS ( in this case 2.5.1 ) looks something
like:
dogbert# truss -o /tmp/dmesg dmesg
execve("/usr/sbin/dmesg", 0xEFFFFE90, 0xEFFFFE98) argc = 1
*** SGID: rgid/egid/sgid = 1 / 3 / 3 ***
open("/dev/zero", O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xEF7C0000
open("/usr/lib/libkvm.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFB44) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
munmap(0xEF796000, 57344) = 0
mmap(0xEF7A4000, 4687, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 16384) = 0xEF7A4000
close(4) = 0
open("/usr/lib/libelf.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFB44) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF760000
munmap(0xEF76E000, 57344) = 0
mmap(0xEF77C000, 4460, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 49152) = 0xEF77C000
close(4) = 0
open("/usr/lib/libc.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFB44) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
mmap(0x00000000, 622592, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF680000
munmap(0xEF700000, 57344) = 0
mmap(0xEF70E000, 26688, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 516096) = 0xEF70E000
mmap(0xEF716000, 2696, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xEF716000
close(4) = 0
open("/usr/lib/libdl.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFFB44) = 0
mmap(0xEF7B0000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) = 0xEF7B0000
close(4) = 0
open("/usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1", O_RDONLY) = 4
fstat(4, 0xEFFFF9A4) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF750000
mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF730000
munmap(0xEF734000, 57344) = 0
mmap(0xEF742000, 5440, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 8192) = 0xEF742000
close(4) = 0
close(3) = 0
munmap(0xEF750000, 8192) = 0
open("/usr/platform/SUNW,Ultra-1/lib/libkvm_psr.so.1", O_RDONLY) Err#2 ENOENT
brk(0x00021320) = 0
brk(0x00023320) = 0
open("/dev/ksyms", O_RDONLY) = 3
...
...
...

By comparing the truss output, it becomes apparent that dmesg is hanging when trying to "open" /dev/ksyms.

Solution

To resolve, either upgrade to Solaris 7 or above, or just manually define a kobj symbol space that is large enough to accommodate all required devices and modules on the afflicted server.

To check the current kobj_map_space_len setting, run command:

# echo kobj_map_space_len/X|adb -k
physmem f832
kobj_map_space_len:
kobj_map_space_len: 100000

The example above displays a default setting of 1MB ( 0x100000); to double the size of kobj_map_space_len from the example above, add the following entry into /etc/system then reboot:

set kobj_map_space_len=0x200000

The minimum required size of kobj_map_space_len will is function of the number of drivers and modules that the server requires. Hence, required minimum values will vary.
Please note that any defined value must fall upon an even page boundary.

Typically, given these constraints, the following definition should be more than adequate for most large fully populated servers:

set kobj_map_space_len=0x300000

If there is any reason to suspect that 3MB is not large enough to accommodate the server's needs, then try:

set kobj_map_space_len=0x400000

A reboot is required for the new table size to take effect.

Products
Solaris 7 Operating System
Sun Enterprise 6500 Server
Sun Enterprise 5500 Server
Sun Enterprise 4500 Server
Sun Enterprise 3500 Server

Internal Section

Solaris 7 resolves this problem by altering the method by which the kobj symbol space is allocated (In Solaris 7 kobj symbol space is allocated dynamically, whereas, in 2.5.1 and 2.6 kobj symbol space is statically defined when the system first boots ).

Previously Published As 20267

Attachments

This solution has no attachment