Sun Storage 7000 Unified Storage System: AK scripts may fail due to 'out of memory' condition

Asset ID:	1-72-1558615.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1558615.1 : Sun Storage 7000 Unified Storage System: AK scripts may fail due to 'out of memory' condition

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

This document is specific to ZFS Storage Appliance.

The given error may be seen while executing script :

aksh: failed to execute /usr/lib/ak/js/shell/akService.js

Cause

After analysis we found that this error is seen when aksh scripts failed to execute.

From the logs we can see that the script fails to execute as akd ran out of memory.

1) Akd is 3.2G which is almost Max size it can handle.
From bundle we can see akd core is almost at the Max size.

File : <bundle>/core/

-rw-rw-rw- 1 sctools other 3.2G Jun 3 04:04 core.29367

2) akd core shows lot of fragmentation (comparing in-use column to total). And akd may also show some allocation failures.

> ::vmem
ADDR NAME INUSE TOTAL SUCCEED FAIL
fe99dc28 sbrk_top 1427566592 3159015424 1170108169 2540758
fe99e09c sbrk_heap 1427566592 1427566592 1170108169 2536524
fe99e510 vmem_internal 52244480 52244480 14269139 0
fe99e984 vmem_seg 48992256 48992256 11961 0
fe99edf8 vmem_hash 3236096 3239936 31 0
fe99f26c vmem_vmem 17100 19128 14257167 0
08062000 umem_internal 12919808 12922880 147179 0
08062474 umem_cache 402320 577536 51 0
080628e8 umem_hash 453632 458752 52 0
08063000 umem_log 0 0 0 0
08063474 umem_firewall_va 0 0 0 0
080638e8 umem_firewall 0 0 0 0
08064000 umem_oversize 176377628 180146176 1154538981 2518017
08064474 umem_memalign 4481040 4489216 1244 18507
080648e8 umem_default 1177763840 1177763840 1151626 0
>

3) akd.ak Logs shows "Not enough space" :

Mon Jun 3 04:01:16 2013: failed to read stat for dataset nfs4.ops[share]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:16 2013: failed to read stat for dataset smb.ops[op]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:16 2013: failed to read stat for dataset ndmp.diskkb[op]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:16 2013: failed to read stat for dataset ndmp.tapekb: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset arc.accesses[L2ARC eligibility]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset io.bytes: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset iscsi.ops: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset nfs3.ops[op]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset arc.size[component]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: failed to read stat for dataset nfs4.ops[op]: asynchronous statistic error: failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'zpool': failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'test': failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'nscd': failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'srp': failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'ip': failed to update kstat chain: Not enough space
Mon Jun 3 04:01:17 2013: asynchronous error on statistics module 'tcp': failed to update kstat chain: Not enough space

------------------------------

Refer to Bug : 16187433

As akd is almost out of memory, it is unable to handle requests from aksh scripts and hence generating the errors.

This 'out of memory' situation could be caused if akd is a victim of memory fragmentation.

Solution

It is recommended to upgrade the Appliance Firmware to the latest to avoid such issues.

If the issue is seen please contact your Oracle support center.
An engineer will generate a core of the process and will send it to engineering for it to be investigated further.

If, and only if having AKD back has a very high business impact, let's note that as a workaround, if the CLI is accessible, you can try to restart the system management interface (BUI) by running :

NAS> maintenance system restart

This will restart management interface and will restart akd as well which should recover from this situation.

In case the CLI is not accessible you should engage Oracle Support to remotely login and resolve the issue.

If you use a lot of analytics you should refer to How to configure Analytics for dataset retention policies that will help to keep akd under limits and avoid fragmentation.

References

<BUG:16187433> - DATASETS WITH LOTS OF BREAKDOWNS CAUSES MEMORY FRAGMENTATION IN AKD

Attachments

This solution has no attachment