Sun Storage 7000 Unified Storage System: How To Monitor Appliance Kit Daemon (AKD) Process Memory Size and Alert if Oversize.

Asset ID:	1-71-1582107.1
Update Date:	2018-01-05
Keywords:

Solution Type Technical Instruction Sure

Solution 1582107.1 : Sun Storage 7000 Unified Storage System: How To Monitor Appliance Kit Daemon (AKD) Process Memory Size and Alert if Oversize.

Applies to:

Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
SPARC SuperCluster T4-4 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Goal

To monitor the size of the Appliance Kit Daemon ( AKD ) using Analytics datasets and Threshold Alerts.

This can be useful if you have been effected by 'Bug 16187433 - datasets with lots of breakdowns causes memory fragmentation in akd', 'Bug 22331668 - update dataspan propagation to not include all data - elide some breakdowns'.
Please see the document linked below for more details on how to confirm if you have been effected by 'Excessive kernel virtual memory (exceeding the 32-bit VM limit)' or 'Excessive amount of AKD process (memory) heap fragmentation'.

Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang) (Doc ID 1401282.1)

Please note: 'Bug 16187433 - datasets with lots of breakdowns causes memory fragmentation in akd' is fixed in 2011.1.9.0 and 2013.1.1.5 code.

We have another known issue - Bug 22331668 - update dataspan propagation to not include all data - elide some breakdowns. This is effecting current code so its still recommended to reduce the amount of analytics drilldowns.

Solution

If your appliance has been confirmed to be affected by akd memory fragmentation and an upgrade to a release containing a fix is not yet possible its recommended the following steps be taken to allow you to monitor akd size on your appliances

To limit the chance of being impacted by this condition you can setup a worksheet to monitor akd size so it can be restarted before it gets over 3GB in size.

Both CLI and BUI methods are demonstrated below:

Step 1. Ensure advanced analytics are made available via "Configuration Preferences".

:> configuration preferences get advanced_analytics
advanced_analytics = false

:> configuration preferences set advanced_analytics=true
advanced_analytics = true

:> configuration preferences get advanced_analytics
advanced_analytics = true

BUI-pt1

Step 2. Under Analytics create a "New" worksheet and title it "akd_size" and open it.

:> analytics worksheets create akd_size

:analytics worksheets> ls
Worksheets:

WORKSHEET       OWNER   NAME
worksheet-000   root    NFS Packet Loss
worksheet-001   root    akd_size

:analytics worksheets> select worksheet-001

:analytics worksheet-001> ls
Properties:
                          uuid = 5b1f3614-da66-e498-b71d-8374e96eb00a
                          name = akd_size
                         owner = root
                         ctime = 2013-9-6 05:20
                         mtime = 2013-9-6 05:20

BUI-pt2

Step 3. Add the following dataset statistic to your worksheet - "Memory:dynamic memory usage broken down by application name".

Step 3. is not required when performing this procedure from the appliance CLI.

BUI-pt3

Step 4. Select "mgmt" from the list and select the Drilldown button and drilldown on "mgmt" as a raw statistic.

:analytics datasets create mem.heap[application=mgmt]

:analytics worksheet select worksheet-001

:analytics worksheet-001> dataset

:analytics worksheet-001 dataset (uncommitted)> set name="mem.heap[application=mgmt]"
name = mem.heap[application=mgmt]

:analytics worksheet-001 dataset (uncommitted)> commit

:analytics worksheet-001> ls
Properties:
                          uuid = 5b1f3614-da66-e498-b71d-8374e96eb00a
                          name = akd_size
                         owner = root
                         ctime = 2013-9-6 05:20
                         mtime = 2013-9-6 05:34:40

Datasets:

DATASET      DATE        SECONDS NAME
dataset-000 -               600 mem.heap[application=mgmt]

BUI-pt4

Step 5. Close the "Memory: dynamic memory usage broken down by application name" statistic so you are only left with the "Memory: dynamic memory usage for application 'mgmt' statistic.

Step 5. is not required when performing this procedure from the appliance CLI.

BUI-pt5a

BUI-pt5b

Step 6. Save and close the worksheet.

Step 6. is not required when performing this procedure from the appliance CLI.

BUI-pt6

Step 7. Set a threshold alert via 'configuration alerts' for Memory:dynamic memory usage for application 'mgmt' of 2.5 GB.

The procedure below is just an example you can customize the alert to your preferences

:analytics worksheet-001> cd /

:> configuration alerts thresholds create

:configuration alerts threshold (uncommitted)> set statname="mem.heap[application=mgmt]"
                      statname = mem.heap[application=mgmt] (uncommitted)
:configuration alerts threshold (uncommitted)> set type=normal
                          type = normal (uncommitted)
:configuration alerts threshold (uncommitted)> set limit=2560
                         limit = 2560 (uncommitted)
:configuration alerts threshold (uncommitted)> set minpost=300
                       minpost = 5 minutes (uncommitted)
:configuration alerts threshold (uncommitted)> set days=all
                          days = all (uncommitted)
:configuration alerts threshold (uncommitted)> set frequency=3600
                     frequency = 1 hours (uncommitted)

:configuration alerts threshold (uncommitted)> commit
Created watch 253f7e8a-9d33-c7b6-c4aa-8bb0dca4f0d6
:configuration alerts thresholds> ls
Thresholds:

THRESHOLD LIMIT TYPE STATNAME
threshold-000 2560 normal mem.heap[application=mgmt]

:configuration alerts thresholds> cd ..

:configuration alerts> actions create

:configuration alerts actions (uncommitted)> set category=thresholds
                      category = thresholds
:configuration alerts actions (uncommitted)> set thresholdid=253f7e8a-9d33-c7b6-c4aa-8bb0dca4f0d6
                   thresholdid = 000 (uncommitted)
:configuration alerts actions (uncommitted)> commit
:configuration alerts actions> ls
Actions:

ACTIONS      CATEGORY             ACTION       HANDLER
actions-000 thresholds           -            -

:configuration alerts actions> select actions-000
:configuration alerts actions-000> action
:configuration alerts actions-000 action (uncommitted)> set address=xyx@xyz.com
                       address = daniel.j.green@oracle.com (uncommitted)
:configuration alerts actions-000 action (uncommitted)> set subject="ZFSSA AKD oversize alert hostname=xyz"
                       subject = ZFSSA AKD oversize alert hostname=xyz (uncommitted)
:configuration alerts actions-000 action (uncommitted)> commit

:configuration alerts actions-000> ls
Properties:
                      category = thresholds
                   thresholdid = 253f7e8a-9d33-c7b6-c4aa-8bb0dca4f0d6

Actions:

ACTION       HANDLER           ARGS
action-000   email             address = xyz@xyz.com
                               subject = ZFSSA AKD oversize alert hostname=xyz

:configuration alerts actions-000> done

BUI-pt7a

BUI-pt7b

Step 8. ( Optional ) You can modify your alert or add further alerts to perform different actions as needed.

BUI-pt8

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance Community

Checked for Currency 09-OCT-2017

References

<NOTE:1401282.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Unresponsive Administrative Interface (BUI/CLI hang)
<BUG:22331668> - UPDATE DATASPAN PROPAGATION TO NOT INCLUDE ALL DATA - ELIDE SOME BREAKDOWNS
<BUG:16187433> - DATASETS WITH LOTS OF BREAKDOWNS CAUSES MEMORY FRAGMENTATION IN AKD

Attachments

This solution has no attachment