Oracle ZFS Storage Appliance: Replication error - "failed to invoke receive() XDR: bad status: 'exists'"

Asset ID:	1-72-2085160.1
Update Date:	2017-10-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2085160.1 : Oracle ZFS Storage Appliance: Replication error - "failed to invoke receive() XDR: bad status: 'exists'"

Applies to:

Oracle ZFS Storage ZS3-4 - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

Replication is failing with an alert such as:

SUNW-MSG-ID: NAS-8001-64, TYPE: alert, VER: 1, SEVERITY: Minor
EVENT-TIME: Wed Dec 2 04:00:07 2015
PLATFORM: i86pc, CSN: 0916AMF003, HOSTNAME: ZFSSA-1-appl
SOURCE: appliance/kit/akd:default, REV: 1.0
EVENT-ID: 0ecd940f-6712-6cad-b7c3-9ac16e98c97f
DESC: Replication of 'MyProject/MyShare' to '7420-ZFS-appl' failed.
AUTO-RESPONSE: None.
IMPACT: Some or all of the changes in the last replication update may not have been sent successfully.
REC-ACTION: Check for alerts posted on the target appliance. Contact your service provider if this problem persists.

Additionally, the replication.ak log from the support bundle of the source appliance reports:

Wed Dec 2 04:00:07 2015
nvlist version: 0
time = 0x565e6cc7
hrtime = 0xa78e895f421755
action = (embedded nvlist)
nvlist version: 0
target_label = 7420-ZFS-appl-target
target_uuid = 05df94eb-3f75-ef76-8243-f56acf87a41b
uuid = b883c7f5-943f-6a19-c2ba-a1585145bbe3
state = sending
dataset = pool-0/local/MyProject/MyShare
(end action)

event = update done
result = failure
errmsg = stage 'stream_setup' failed: failed to invoke receive() XDR: bad status: 'exists'
remote_status = exists

Cause

The target appliance responds to the source appliance with error status "exists" when it already has replicated data associated with the referenced replication ID but is unable to use it to carry out the requested update.

For example:

1 - The target system receives a "full update" while expecting an "incremental update".

The source initiates a "full update" if there has not yet been a successful "initial send" of the complete dataset under this replication ID.

If the target already has any replication data associated with this replication package (even if that data is invalid), it accepts only incremental updates.

This will happen if the initial send ("full update") was cancelled or otherwise failed to complete.

2 - The target system receives an "incremental update" but cannot reconcile this with the existing snapshot data.

Such a scenario can happen with older software (below 2013.1.1.1) if a share with a replication action is cloned with "Retain Other Local Settings" and the clone is kept in the same project, resulting in replication actions with duplicate IDs.

See Document ID 1554806.1.

Solution

The only solution is to destroy this replication action on the source appliance and create a new one.

It is best practice to also clean up the corresponding replication package on the target appliance.

As stated in point 5. of Doc ID 1595549.1 - How to set up Replication - the initial/first replication MUST complete successfully or you must clean up any remnants ('old' actions/snapshots etc.) and start again. This document has helpful replication configuration tips.

References

<NOTE:1595549.1> - Oracle ZFS Storage Appliance: How to set up Replication

Attachments

This solution has no attachment