Oracle ZFS Storage Appliance: Replication Fails with "failed on remote peer: structure member 'usedbyrefreservation' must be an int" on Target NAS

Asset ID:	1-72-1933532.1
Update Date:	2016-02-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1933532.1 : Oracle ZFS Storage Appliance: Replication Fails with "failed on remote peer: structure member 'usedbyrefreservation' must be an int" on Target NAS

Applies to:

Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7310 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Replications failing with the following error messages;

errmsg='stage 'stream_setup' failed: failed to invoke receive() XDR: rpc com.sun.ak.nas_repl.receive:1 failed on remote peer: structure member 'usedbyrefreservation' must be an int'
errmsg='stage 'stream_setup' failed: failed to invoke receive() XDR: rpc com.sun.ak.nas_repl.receive:1 failed on remote peer: receive request rejected by sender'
target_label='metdr-repl-channel'

Changes

None noted

Cause

The replication interface used - 'igb0' - is a private interface, and private interfaces should not be configured for replication.

From Document ID 1595549.1 - On cluster systems, ensure that the network interfaces used for replication and the associated zpool are 'tied' together (ie. owned by the same cluster head at all times). So that, on cluster takeover or failback, the network interfaces ownership always moves with the zpool ownership.

Source SourceHead2 - 4 Actions Configured For Replication

2 akpt_addr = 0x9181b18 "10.251.41.1:216"
2 akpt_addr = 0x9181c38 "10.249.68.1:216" ----> 2 Actions configured on the Source To Replicate the IPMP2 Interface on Target metf-h2-TargetHead2

ipmp2: flags=108001000843 mtu 1500 index 8 inet 10.249.68.1 netmask ffffff00 broadcast 10.249.68.255 groupname ipmp2

Target TargetHead2 - 1 Actions Configured For Replication

1 akpt_addr = 0x9392848 "10.251.41.1:216"

Target side Package: LISTINGS

ADDR      PKG                                   PROJECT (SOURCE)
11fb9108 3f405cb3-0cff-e7de-8bba-a0ff7af0a699 ASAP_Pro (metp-h1-storadm) STATE: IDLE FLAGS: ENABLED
11fba308 fd8a7f9d-4b85-6010-ede9-f77bbd495b6a M6_Pro   (metp-h1-storadm) STATE: IDLE FLAGS: ENABLED
127b8948 5ded6e8a-6b82-c090-d500-c9d7150219ea           (metp-h1-storadm) STATE: IDLE FLAGS: ENABLED

Target TargetHead2

3 nrs_ipaddr = 0x9bf2968 "10.251.65.15:216" #### 3 packages are configured to replicate FROM the 'igb0' interface on metp-h2-SourceHead2

Now which Interface has IP 10.251.65.15 ?

Source SourceHead2

igb0: flags=100001000842 mtu 1500 index 7 <<<<<<<<<<<<<<< DOWN>>>>>>
inet 10.251.65.15 netmask ffffff00 broadcast 10.251.65.255
ether 0:10:e0:3e:3a:b2
igb1: flags=100001000843 mtu 1500 index 5
inet 10.251.65.16 netmask ffffff00 broadcast 10.251.65.255
ether 0:10:e0:3e:3a:b3

Source SourceHead1

igb0: flags=100001000843 mtu 1500 index 7 <<<<<<<<<<<<>>>>>>>>
inet 10.251.65.15 netmask ffffff00 broadcast 10.251.65.255
ether 0:10:e0:3d:27:88
igb1: flags=100001000842 mtu 1500 index 4
inet 10.251.65.16 netmask ffffff00 broadcast 10.251.65.255
ether 0:10:e0:3d:27:89

Summary: SourceHead2 owns the POOL (Active Node) and the igb0 interface is down, whereas the igb0 interface is UP on SourceHead1 (Standby Node)

NOTE: The above error message may also be reported for replication failure, when the the AKD deamon process size approaches was 3.5Gb.

Solution

A takeover by SourceHead1 (Standby) will offer immediate relief - because interface 'igb0' is active on Standby node (workaround).

For a complete resolution:

Fix the Routing issue
Ensure Only a SINGLETON Interface is used for Replication. See Document ID 1595549.1

References

<BUG:15706493> - SUNBT7032945 REMOTE REPLICATION NOT FAILING OVER, ''USEDBYREFRESERVATION' MUST B
<NOTE:166650.1> - Working Effectively With Oracle Support - Best Practices
<NOTE:1595549.1> - Oracle ZFS Storage Appliance: How to set up Replication

Attachments

This solution has no attachment