Oracle ZFS Storage Appliance: Replication Action Fails with "rpc com.sun.ak.nas_repl.compat:1 failed on remote peer: package not found"

Asset ID:	1-72-2119903.1
Update Date:	2016-03-29
Keywords:

Solution Type Problem Resolution Sure

Solution 2119903.1 : Oracle ZFS Storage Appliance: Replication Action Fails with "rpc com.sun.ak.nas_repl.compat:1 failed on remote peer: package not found"

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

An established replication between source and target has been successfully running at regular intervals.

From the CLI, you will see the action successfully run :

s7320-1:> shares select Dbase replication select action-000
s7320-1:shares Dbase action-000> sendupdate
s7320-1:shares Dbase action-000> show
Properties:
id = c3e41239-7fa9-e908-89d3-e870fa7d8c79
target = 192.168.10.10
enabled = true
continuous = false
include_snaps = true
max_bandwidth = unlimited
bytes_sent = 3.42G
estimated_size = 4.6G
estimated_time_left = 00:00:04
average_throughput = 292MB/s
.....

A subsequent replication of the project (Dbase) fails. The CLI reports nothing, other than the failed attempt :

s7320-1:shares Dbase action-000> sendupdate
s7320-1:shares Dbase action-000> show
Properties:
id = c3e41239-7fa9-e908-89d3-e870fa7d8c79
target = 192.168.10.10
enabled = true
................
next_update = Sync now
last_sync = Thu Mar 24 2016 12:56:49 GMT+0000 (UTC)
last_try = Thu Mar 24 2016 13:02:01 GMT+0000 (UTC)
last_result = The most recent replication update failed. No additional information is available. Check replication status on the target system. See the replication documentation for more details.

However, the replication logs on the source provide a clue as to what caused the failure :

Wed Mar 24 13:02:01 2016
nvlist version: 0
time = 0x56f2df17
hrtime = 0xac09685b3337
action = (embedded nvlist)
nvlist version: 0
target_label = 192.168.10.10
target_uuid = 09fe27a0-1a34-62b7-a278-d78c61109898
uuid = a80f2312-6f2d-c229-a7f0-f93708288d83
state = sending
dataset = pool-a/local/Dbase
(end action)
event = update done
result = failure
errmsg = stage 'compat' failed: compat failed: rpc com.sun.ak.nas_repl.compat:1 failed on remote peer: package not found
remote_status =

Cause

In this particular case, the error message in the replication logs is telling us exactly what happened. The package of replicated data for the project named "Dbase" does not show up on the target system.

When you list out the packages on the target, there is a package for a project called "VM", but none for "Dbase". "Dbase" was, most likely deleted.

s7320-2:> shares replication sources select source-000
s7320-2:shares replication source-000> show
Properties:
name = s7320-1
ip_address = 192.168.10.11:216
asn = 0e4ab067-942c-6b9b-da8a-d7a7985e7d93

Packages:

PROJECT STATE LAST UPDATE
package-001 VM idle Thu Mar 24 2016 13:33:23 GMT+0000 (UTC)

Great care needs to be given to administration of replication packages.

The relationship between source ZFS replication actions, and target ZFS replication packages is not maintained for you. If you delete one, the other one remains on the system, thus rendering it useless.

Solution

Your only recourse at this point is to destroy the action on the source and re-create a new action.

This will, in turn, perform a new 'initial' replication from the source to the target :

s7320-1:> shares select Dbase replication destroy action-000
This will destroy "action-000". Are you sure? (Y/N)

s7320-1:> shares select Dbase replication action
s7320-1:shares Dbase action (uncommitted)> set........(steps not shown)
Properties:
target = (unset)
pool = (unset)
enabled = true

Attachments

This solution has no attachment