Oracle ZFS Storage Appliance: "Unanticipated System Error Occurred" when trying to access Maintenance Workflows

Asset ID:	1-72-2251317.1
Update Date:	2018-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2251317.1 : Oracle ZFS Storage Appliance: "Unanticipated System Error Occurred" when trying to access Maintenance Workflows

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage ZS5-2 - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

When uploading a workflow under 'Maintenance Workflows' in the BUI the following error is seen -

Screenshot of workflow error

Once this is seen further entries into "maintenance workflows' via the BUI or CLI will report the same error preventing workflow access.

Example from CLI below -

:> maintenance workflows
error: An unanticipated system error occurred: array.chunk() attempted to return 1048576 bytes, exceeding maximum of 917504 bytes

Cause

This is due to a known issue - Bug 25053034 : large workflows can overflow akx_response_size estimate.

It is triggered by large workflow uploads and the order in which they are uploaded.

Engineering are still working on a fix for this bug, please review the bug notes for more details.

Solution

If the issue is seen please open a new SR so that Oracle Support can determine which workflow is triggering this issue and provide a workaround.

This can be triggered by any large workflow but its commonly triggered by the ZS_Basic_Config_Chk workflow due to its large size.

Workaround from Bug using ZS_Basic_Config_Ckh workflow as example:

We can remove the long workflow by uuid using a raw XML-RPC call.

Look at the size of the objects in the stash:

s7310-tvp540-d-h1# ls -Slh /var/ak/stash/com/sun/ak/xmlrpc/object/*/obj
-rw-------   1 root     root        126K Mar 24 13:45        /var/ak/stash/com/sun/ak/xmlrpc/object/23a827fe-a5bc-c679-a5ef-d1d008f76c42/obj
-rw-------   1 root     root         27K Feb 24 11:46        /var/ak/stash/com/sun/ak/xmlrpc/object/37b853c4-8c37-6803-f184-a5946a9f7c1d/obj
-rw-------   1 root     root         21K Feb 24 11:46        /var/ak/stash/com/sun/ak/xmlrpc/object/1ccb1c92-2347-623f-9097-92be91f43d0e/obj
-rw-------   1 root     root         20K Feb 24 11:46        /var/ak/stash/com/sun/ak/xmlrpc/object/cc3c5ad1-77db-ec06-b1ee-8ca891bb4411/obj
[...]

The ZS_Basic_Config_Chk workflow is around 126k.

Confirm that the large object is indeed the right workflow:

s7310-tvp540-d-h1# aknv -n name
/var/ak/stash/com/sun/ak/xmlrpc/object/23a827fe-a5bc-c679-a5ef-d1d008f76c42/obj
Oracle Inc.: ZS_Basic_Config_Chk, version 2.4

Then from within the CLI, use the stash object UUID to remove the object:

s7310-tvp540-d-h1:> object.removeNamed('workflow', { uuid:'23a827fe-a5bc-c679-a5ef-d1d008f76c42' })
result = 0

This will remove the workflow on both heads of the cluster.

References

<BUG:25053034> - LARGE WORKFLOWS CAN OVERFLOW AKX_RESPONSE_SIZE ESTIMATE

Attachments

This solution has no attachment