Oracle ZFS Storage Appliance: System Hung due to Memory Leak in vfs_shadow_ioc

Asset ID:	1-72-2046957.1
Update Date:	2018-01-08
Keywords:

Solution Type Problem Resolution Sure

Solution 2046957.1 : Oracle ZFS Storage Appliance: System Hung due to Memory Leak in vfs_shadow_ioc_fid2path

Applies to:

Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

System hung due to kernel memory shortage and very high kernel memory utilization due to a memory leak in vfs_shadow_ioc_fid2path().

Cause

System hung due to kernel memory shortage and very high kernel memory utilization due to a memory leak.

ZFS file data utilizes a small amount of memory and there is no indication of any process that is using a large amount of memory.

To assist diagnosis: Enable kmem_flags to confirm the code path and capture another crash dump for Root Cause.

Memory leaks in vfs_shadow_ioc_fid2path in early 2011.1.9 firmware code.

> ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 15221668 59459 91% <<-- high kernel consumption
ZFS File Data 517866 2022 3%
Anon 490397 1915 3%
Exec and libs 0 0 0%
Page cache 479637 1873 3%
Free (cachelist) 0 0 0%
Free (freelist) 65419 255 0% <<-- insufficient free memory

Total 16774987 65527
Physical 16774985 65527

There is a memory leak and kmem_flags need to be enabled to find the code path that is causing memory leak.

kmem_bufctl_cache
suspect                            buf        buf        buf       memory
cache name                        size      avail      total       in use
============================== ======= ========== ========== ============
kmem_bufctl_cache                   24    1855898   58115499   1425395712 PK
kmem_alloc_48                       48    2199663   22471005   1108930560 PKU
kmem_alloc_896                     896     252492     805266    732971008 PKU
streams_dblk_1936                 2048         13    2838196   5812625408 PKUF
zio_buf_512                        512    4212787   20640784 10568081408 PK
zio_buf_16384                    16384         69     751138 12306644992 PK
zio_data_buf_131072             131072          3      15898   2083782656 PZ
dnode_t                            744     296278   16032599 15199772672 PKU

U - enabling KMF_AUDIT will allow "kma users" to work for this cache

From Bug 19699440 - Severe memory shortage found on system running 2011.1.9.

Memory leak in vfs_shadow_ioc_fid2path as duplicate of Bug 18890725 : A possible memory leak in vfs_shadow_ioc_fid2path.

BUG 19133469 - Backport 18890725 to AK-2011.04.24 -> 2011.1.9.3

BUG 19447628 - Backport 18890725 to AK-2013-MICRO -> 2013.1.2.5

@ > ::kmausers kmem_alloc_48
@ mdb: couldn't read kmem_bufctl_t at addr cc868b41e88b4c: no mapping for address
@ 6664848 bytes for 138851 allocations with data size 48:
@          kmem_cache_alloc_debug+0x283
@          kmem_cache_alloc+0x268
@          kmem_alloc+0xdd
@          vfs_shadow_open_and_read+0xc5
@          vfs_shadow_attr_read+0x7d
@          vfs_shadow_ioc_fid2path+0xc2
@          vfs_shadow_check_ioctl+0x12f
@          fop_ioctl+0x66
@          ioctl+0x18e

Sun Storage 7000 Unified Storage System: Starting kmdb & setting kmem_flags for ZFSSA (Doc ID 1415515.1)

Workflow - Set kmem flags -- https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:WIKI-Workflows

Solution

For 7x10 systems : Upgrade to Appliance Firmware Release 2011.1.9.3 or later

For 7x20 systems and ZS3-x/ZS4-x systems: Upgrade to Appliance Firmware Release 2013.1.2.5 or later

See MOS Document 2021771.1 - Oracle ZFS Storage Appliance: Software Updates

References

<BUG:19591405> - AKD RUNNING IN RT CLASS IS BLOCKING CRITICAL SYSTEM MANAGEMENT THREADS
<BUG:19699440> - SEVERE MEMORY SHORTAGE FOUND ON SYSTEM RUNNING 2011.1.9
<NOTE:2021771.1> - Oracle ZFS Storage Appliance: Software Updates
<NOTE:1415515.1> - Sun Storage 7000 Unified Storage System: Starting kmdb & setting kmem_flags for ZFSSA

Attachments

This solution has no attachment