Sun Storage 7000 Unified Storage System: BUI/CLI hang due to 'excessive' amount of 'old' log files

Asset ID:	1-72-1408493.1
Update Date:	2018-05-24
Keywords:

Solution Type Problem Resolution Sure

Solution 1408493.1 : Sun Storage 7000 Unified Storage System: BUI/CLI hang due to 'excessive' amount of 'old' log files

Applies to:

Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Symptoms observable by the customer:

Cannot login to BUI/CLI
Logging in via SP/console, appliance drops to 'failsafe' shell:

appliance/kit/akd: default:method "exec /usr/lib/ak/akd" failed with exit status 1.

Failure to configure storage pool - error message is "failed to initialize libzfs".
Replication failures

Symptoms observable by the Oracle Support engineer:

aklog may have the following message logged:

akd: failed to start appliance kit: failed to locate metadata schema fma: Too many open files

Replication failures may log similar messages in rm.ak.txt:

      Tue Aug 17 04:21:12 2010: failed to remote insert ak:/replication/action-7fef609f-5e95-c76a-fbf2-a04fd7615f7d: rpc com.sun.ak.stash.teleportFini:
                1 failed on remote peer: cannot access object com/sun/ak/replication_action/7fef609f-5e95-c76a-fbf2-a04fd7615f7d: No such file or directory
      Tue Aug 17 04:21:12 2010: import of ak:/replication/action-7fef609f-5e95-c76a-fbf2-a04fd7615f7d succeeded in 0.000s

The /var/ak/logs directory contains very many 'old' log files:

# cd /var/ak/logs
      # ls -ltr | wc -l
        496                            => 496 files

      # du -hs .
        41M   .                        => total size of logs directory = 41Mb

      # ls
      < snipped >
      alert.ak.40                httpd.txt.3                   scrk_curl.txt.16.gz
      alert.ak.41                httpd.txt.4                   scrk_curl.txt.2
      alert.ak.42                httpd.txt.5                   scrk_curl.txt.3
      alert.ak.43                httpd.txt.6.gz                scrk_curl.txt.4
      alert.ak.44                httpd.txt.7.gz                scrk_curl.txt.5
      alert.ak.45                httpd.txt.8.gz                scrk_curl.txt.6.gz
      alert.ak.46                httpd.txt.9.gz                scrk_curl.txt.7.gz
      alert.ak.47                jobs.txt.1                    scrk_curl.txt.8.gz
      alert.ak.48                jobs.txt.10.gz                scrk_curl.txt.9.gz
      < snipped >

Cause

This is a known issue. The root cause of this problem is too many open files by akd.

A few libraries (including libzfs) used by akd have a 256 file descriptor limit.

Known issue - <BUG 15613474> - akd should globally enable extended FILE stdio (Fixed in 2010.Q1.0.0)

Solution

Recommended action for the customer:

For the fix for this issue, the Appliance Firmware Release should be upgraded to 2010.Q1.0.0 or later.

Oracle Support always recommend that you upgrade to the latest Appliance Firmware Release.

To remove the (excessive) 'old' log files, you will need to engage Oracle Support, by opening a Service Request, so that Oracle Support Services can provide confirmation of this issue and then carry out the appropriate activities to resolve the issue.

Recommended actions for the Oracle Support engineer:

Delete the 'old' log files from /var/ak/logs - see the following wiki document:

https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Solaris+shell+procedure+to+%27clean+up%27+%28and+rotate%29+log+files
If you cannot access this document engage NAS Storage-TSC for assistance

Back to Document 1401282.1 Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface.

***Checked for relevance on 24-MAY-2018***

References

SUPPORT WIKI - ROTATE LOG FILES: HTTPS://STBEEHIVE.ORACLE.COM/TEAMCOLLAB/WIKI/AMBERROADSUPPORT:SOLARIS+SHELL+PROCEDURE+TO+%27CLEAN+UP%27+%28AND+ROTATE%29+LOG+FILES
<BUG:15613474> - SUNBT6914407 AKD SHOULD GLOBALLY ENABLE EXTENDED FILE STDIO
<NOTE:1401282.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)
<BUG:6966620> - QUICK OUTLINE:SELECT METHODS ICON, FIELDS DISPLAYED

Attachments

This solution has no attachment