Sun Storage 7000 Unified Storage System: SNMP times out and leaks memory

Asset ID:	1-72-1539983.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1539983.1 : Sun Storage 7000 Unified Storage System: SNMP times out and leaks memory

Applies to:

Sun ZFS Storage 7420 - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7320 - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7110 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun Storage 7210 Unified Storage System - Version Not Applicable to Not Applicable [Release N/A]
Sun ZFS Storage 7120 - Version Not Applicable to Not Applicable [Release N/A]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

On ZFS Storage Applicance 7000 when SNMP is enabled the clients reports snmpwalk and snmpget times out.

Additionally SNMP will start consuming much of the memory.

Disabling SNMP service may fail with Transient Errors.

Cause

We found that if the number of interfaces are above 10 then SNMP times out.

And SNMP is doing a lot of iteration loops to get the kernel statistics of interfaces.

This seems to be consuming memory over a period of time.

Bug: 15811787 - SUNBT7193490 snmpwalk and snmpget times out

1. snmpwalk and snmpget does not work and times out :
    snmpwalk and snmpget will fail even on the local host i.e. ZFS Appliance itself.

# snmpwalk -v2c -c public localhost SUN-FM-MIB::sunFmProblemTable
Timeout: No Response from localhost

#snmpwalk -O n -v 1 -c public 10.160.120.141 .1
Timeout: No Response from 10.160.120.141

# snmpget -m SUN-FM-MIB -v2c -c public localhost
SUN-FM-MIB::sunFmFaultEventClass.\"1f49d9a6-5a45-417b-8405-e836d935b3c7 \".1
Timeout: No Response from localhost.

2. SNMP consumes a lot of memory may go around 40 G
snmp core shows :
> ::memstat Page Summary                Pages                MB  %Tot ------------     ----------------  ----------------  ---- Kernel                    8083360             31575   12% ZFS File Data            46436598            181392   69% Anon                     10899018             42574   16% >>>> Anon taking a lot of space Exec and libs                3114                12    0% Page cache                  26758               104    0% Free (cachelist)             8259                32    0% Free (freelist)           1647363              6435    2% Total                    67104470            262126 Physical                 67104469            262126

3. Disabling SNMP fails in the CLI and svcs shows transitional state
# svcs -a | grep -i snmp disabled       Mar_08   svc:/application/management/snmpdx:default online*        Aug_22   svc:/application/management/net-snmp:default <<<<<< online         Aug_22   svc:/system/fm/snmp-notify:default

4. pstack of snmp core shows :
core 'core.14850' of 14850:/usr/sbin/snmpd fffffd7fff21dbaa ioctl () + a fffffd7ffe20d494 getKstat () + 224                <<<<<< getKstat fffffd7ffe21276e get_if_stats () + 1be fffffd7ffe211dbf getif () + 62f fffffd7ffe20f13f getMibstat () + 45f fffffd7ffe229faf netsnmp_arch_interface_container_load () + a7f fffffd7ffe1fed95 netsnmp_access_interface_container_load () + 185 fffffd7ffe1fe9f1 netsnmp_access_interface_init () + a1 fffffd7ffe23c962 init_mib_modules () + 12 0000000000405011 main () + 1ab1 0000000000402f7c ???????? ()

Solution

Solution is available in Appliance Firmware Release 2011.1.5.0

Workaround:
1) Disable SNMP altogether in not needed.
OR
2) Reduce the number of interfaces (to below 10)

References

<BUG:15811787> - SUNBT7193490 SNMPWALK AND SNMPGET TIMES OUT
<BUG:15680487> - SUNBT6998845 SNMPD DOES NOT PROCESS DUPLICATE INTERFACE CONTAINERS
<NOTE:2021771.1> - Oracle ZFS Storage Appliance: Software Updates

Attachments

This solution has no attachment