Oracle ZFS Storage Appliance: Replication Action Fails With "Dataset Does Not Exist" Error.

Asset ID:	1-72-1927379.1
Update Date:	2016-02-09
Keywords:

Solution Type Problem Resolution Sure

Solution 1927379.1 : Oracle ZFS Storage Appliance: Replication Action Fails With "Dataset Does Not Exist" Error.

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Oracle ZFS Storage ZS3-4 - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

A scheduled or manual replication action kicks off at its designated start time. The BUI will display the typical message indicating replication has begun.

Began replicating 'Project-DB' to appliance '7000-Target'.

The status bar will initiate with a percent complete. However, shortly after the replication starts, it will fail with the following BUI error.

Replication of 'Project-DB' to '7000-Target' failed.

The ./logs/replication.ak.txt file from the support bundle of the Source NAS Appliance will contain one of the following error messages. (parsed data for clarity)

Thu Sep 11 09:40:13 2014
       target_label = 7000-target
       target_uuid = 8da90648-fc07-6854-f54f-bf54b86a84c1
       uuid = ab4cf8fe-6bc1-4aa5-c703-b38574edf032
       state = sending
       dataset = pool-0/local/Project-DB
       event = update done
       result = failure
       errmsg = stage 'stream_send' failed: zfs_send: cannot open 'pool-0/local/Project-DB/Database1': dataset does not exist
       remote_status = ok

Tue Sep 16 19:44:15 2014
       target_label = 7000-Target
       target_uuid = 763a4ade-9bb7-ef55-bb60-fc6da9763bc6
       uuid = 591b0d09-3acd-c0da-e9c7-85099aa5a83f
       state = sending
       dataset = pool-0/local/Project-DB
       event = update done
       result = failure
       errmsg = stage 'stream_send' failed: zfs_send: cannot open 'pool-0/local/Project-DB/Database1': filesystem does not exist
       remote_status = ok

Changes

There was no identifiable change to the system. The appliance contains no evidence that the dataset (aka share) even exists.

# zfs list -t all | grep Project-DB
NAME                                        USED AVAIL REFER MOUNTPOINT
pool-0/local/Project-DB                       3.52T 2.48T 44.9K /export/Project-DB
pool-0/local/Project-DB/Database2             3.52T 2.48T 44.9K /export/Project-DB/Database2
pool-0/local/Project-DB/Database3             71.4G 61.5G   288G /export/Project-DB/Database3
pool-0/local/Project-DB/Database4             1.99G 9.69G 20.3G /export/Project-DB/Database4
pool-0/local/Project-DB/Database5             1.26G 9.62G 5.38G /export/Project-DB/Database5
pool-0/local/Project-DB/Database6             2.35G 7.31G 7.69G /export/Project-DB/Database6
pool-0/local/Project-DB/Database7             44.9K 32.0G 44.9K /export/Project-DB/Database7
pool-0/local/Project-DB/Database8             92.3G 2.48T 92.3G /export/Project-DB/Database8

7000:> shares select Proj-DB show
............
Filesystems:
NAME             SIZE    MOUNTPOINT
Database2        3.52T   /export/Project-DB/Database2
Database3        71.4G   /export/Project-DB/Database3
Database4        1.99G   /export/Project-DB/Database4
Database5        1.26G   /export/Project-DB/Database5
Database6        2.35G   /export/Project-DB/Database6
Database7        44.9K   /export/Project-DB/Database7
Database8        92.3G   /export/Project-DB/Database8

However, it is important to note that automated schedules for replication and manipulation of shares were running.

Cause

In this particular instance, an automated replication action was configured to run weekly.

The administrator destroyed an associated share by mistake after the replication began. Once the share was destroyed, the replication failed.

Solution

The error described in this article can be avoided by not destroying shares during a scheduled replication.

There are some safeguards in place to prevent this, but they are not foolproof.

For example, an attempt to delete a share which is currently being replicated will fail, with a message similar to the following:

7000:> shares select Project-DB replication show
       Actions:
                  TARGET          STATUS     NEXT
       action-000  7000-target    sending    Sync now

7000:> shares Project-DB destroy Database2
This will destroy all data in "Database2"! Are you sure? (Y/N)
error: The action could not be completed because the target
       'Project-DB/Database2' is in use. It cannot be modified while it, or
       its children, are actively changing. Make sure no other users are
       editing the share configuration and try again. If this problem persists,
       contact your service provider.

However, if you delete a share in the Project which is queued for replication during the scheduled action, it will work. This is what needs to be avoided.

7000:> shares select Project-DB replication show
       Actions:
                  TARGET          STATUS     NEXT
       action-000  7000-target    sending    Sync now

7000:> shares Project-DB destroy Database3
This will destroy all data in "Database3"! Are you sure? (Y/N)
7000:>

Attachments

This solution has no attachment