Sun Storage 7000 Unified Storage System: aksh fatal error: no memory

Asset ID:	1-72-1325025.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1325025.1 : Sun Storage 7000 Unified Storage System: aksh fatal error: no memory

Applies to:

Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

The message 'aksh fatal error: no memory' is seen when customer tried to login to CLI (Admin command Line interface). But all the data services are running fine and shares are accessible.

CLI login fails with error:

#ssh root@192.168.181.61 Password: Last login: Mon May 9 02:00:31 2011 from 172.22.20.2 aksh fatal error: no memory

Also the BUI is not accessible, and fails with errors.

Exception type: coXmlrpcFault
Native message: malformed XML at line 1, column 0: out of memory
Mapped file: https://192.168.181.61:215/lib/crazyolait/coError.js line 37
Mapped stack trace:
:0 @Error()

Native file: https://192.168.181.61:215/lib/crazyolait/index.js line 364
Native stack trace:
Error()@:0
.@https://192.168.181.61:215/lib/crazyolait/index.js:364

Additional native members:
faultCode: 16
faultString: malformed XML at line 1, column 0: out of memory
coStack: coXmlrpcProxy.unmarshallDoc(doc:<object> "[object XMLDocument]", xml:"<methodResponse><fault>
<value><struct><member>
<name>faultCode</name>
<value><int>16</int></value>
</member>
<member>
<name>faultString</name>
<value><string>malformed XML at line 1, column 0: out of memory</string></value>

Cause

This error occurs when the appliance kit service akd runs out of memory.

Solution

Please vefify that the appliance is running the latest Appliance system software, as this issue is not seen in latest releases ( Please see MOS Document ID 2021771.1 - Oracle ZFS Storage Appliance: Software Updates )

If already at the latest firmware then please contact Oracle Support to help resolve the issue.

Verify with customer if they are dropping into Solaris shell/Failsafe mode after logging into Appliance CLI.

If not even able to login to the shell and getting this error then the NMI dump should be collected and the system restarted as mentioned in Document 1173064.1
After reboot the Appliance BUI/CLI should start working fine, later a support bundle should be run so that Oracle Support can analyze the dump.

Incase it is dropping to Failsafe/Solaris shell then verify the issue as follows.
1. Follow steps mentioned in <Document 1401288.1> to collect akd related logs.
2. Verify the physical memory and used memory :

# echo "::memstat" | mdb -k > /var/ak/dropbox/memstat.txt

3. Check the stash size

# df -h | sed -ne 1p -e /stash/p

4. Check the akd process size, if akd > 2.5 then it could be a concern.

# prstat -n 150 1 1 | sed -ne 1p -e /akd/p

5. Check if the analytics datasets are taking too much space

Bugs that may be of interest but not be directly related :

16187433 - datasets with lots of breakdowns causes memory fragmentation in akd

15781962 - (SUNBT7157268-AK-8) REPEATED ANALYTICS GRAPHS AND DRILLDOWNS FRAGMENT MEMORY UNTIL

15676439 - accrued analytics data can completely wedge akd

15613474 - akd should globally enable extended FILE stdio

Back to Document 1401282.1 - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI Hang).

References

<NOTE:1173064.1> - Oracle ZFS Storage Appliance: How to generate a system core dump in case of system hang (BUI and CLI fails to respond) using NMI when directed to do so by an Oracle Support Engineer
<NOTE:1401288.1> - Sun Storage 7000 Unified Storage System: Data collection for akd hang issues

Attachments

This solution has no attachment