Oracle ZFS Storage Appliance: Snap Management Utility (SMU) Backup Fails

Asset ID:	1-72-2290604.1
Update Date:	2017-08-23
Keywords:

Solution Type Problem Resolution Sure

Solution 2290604.1 : Oracle ZFS Storage Appliance: Snap Management Utility (SMU) Backup Fails

Applies to:

Sun ZFS Storage 7120 - Version All Versions and later
Oracle ZFS Storage ZS3-2 - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage ZS5-2 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Approximately 5 minutes after submitting an SMU backup request the customer saw this error:

Backup data file shares
Look up shares mounted on database host: OK
Remote share list
10.x.x.x:/export/v_tst1_smu_oracle_data
Look up storage shares: OK

Undo backup
Disconnect from the application: OK

Result
State: FAILED
Completed: 2017-05-04 01:39:50
Error: HTTP 500 - Internal Server Error: "internal error (AKSH_ERROR: CLI command timed out)"

Cause

This is an instance of the following bug:

Bug 26222968 : REST ERROR: HTTP 500 - Internal Server Error: "internal error

Solution

The customer was requested to collect the following information:

1/ Please can you confirm when you are seeing this message?
2/ Please collect the output from the tasks where these issue were observed.

To collect this, find the task number from GUI, then find the file for that task under /var/opt/ORClsmu/tasks/.

3/ Also please can you upload the smu.log.x log file.

4/ For the ZFSSA storage appliance in use, please gather support bundle data.

The following document contains the data gathering procedure: Sun Storage 7000 Unified Storage System: How to Collect a Support Bundle using the BUI or CLI (Doc ID 1019887.1)

Questions:

What platform is in use for the Management Host (any detail will help - hardware and OS version). Provide details on the database host.

What version of Oracle DB is in use?

SMU logs and support bundle logs

See the knowledge document for further details: Document: 1522925.1 - Snap Management Utility for the Oracle Database - Information and Troubleshooting

From the details in the Bug 26222968, the SMU backline engineer suggested to increase the property value of rest/aksh_wait_run.

Action plan:

To do this the TSC engineer should open a shared shell to the appliance.

The customer should open a ssh to the appliance command line interface (CLI)

CLI> confirm shell

First check the current setting of the property rest/aksh_wait_run:

# svccfg -s akrestd listprop rest/aksh_wait_run

Set the new value of aksh_wait_run to 400:

# svccfg -s akrestd setprop rest/aksh_wait_run = integer: 400
# svcadm refresh akrestd
# svcadm restart akrestd

Then confirm it has been set:

# svccfg -s akrestd listprop rest/aksh_wait_run

Then ask the customer to retry his SMU backup to confirm this has resolved the command timeout.

References

<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices
<BUG:26222968> - REST ERROR: HTTP 500 - INTERNAL SERVER ERROR: "INTERNAL ERROR
<BUG:26035664> - SMU - BACKUP FAILS - HTTP 500 TIMEOUT WHEN LISTING INTERFACES
<NOTE:1019887.1> - Sun Storage 7000 Unified Storage System: How to Collect a Support Bundle using the BUI or CLI
<NOTE:1522925.1> - Snap Management Utility for the Oracle Database - Information and Troubleshooting

Attachments

This solution has no attachment