Sun ZFS Storage Appliance: Project Replication is failing while doing Active Head failover

Asset ID:	1-72-1508081.1
Update Date:	2017-05-08
Keywords:

Solution Type Problem Resolution Sure

Solution 1508081.1 : Sun ZFS Storage Appliance: Project Replication is failing while doing Active Head failover

Applies to:

Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle Exalogic Elastic Cloud Software - Version 1.0.0.0.0 to 2.0.0.0.3
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7410 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

Replication used in ZFS cluster configuration ie, 2 nodes on source and target. Each time a takeover/failover is done on the source or on the target, the replication fails. For example, on target node, we can have the following error :

Tue Nov 13 23:19:06 2012
nvlist version: 0
time = 0x50a2d56a
hrtime = 0x3da24334406188
pkg = (embedded nvlist)
nvlist version: 0
source_asn = f8d753fe-0885-c923-dbdb-9b3a62bfc406
source_name = exln01sn02
uuid = 1fa3ac0b-0ccb-6b24-cd3b-acc987709a4c
state = cancelling
(end pkg)

event = recv_done
result = failed
error = zfs_receive failed: cannot receive: failed to read from stream

Cause

When configuring replication between cluster peers, configure replication with both controllers in the CLUSTERED state.

Do not use private network addresses and use separate replication targets for each ZFS pool.

The IP address of the source node is known on the target. If changed, there may be a problem.

For cluster, use an IP address tied to a pool. In case of failover, this will not change so that the replication peer will still be able to use that IP address.

Solution

In cluster environment we need to make sure that the IP interface used for replication is able to follow the pool for which a replication is configured.

From what is described in the embedded documentation : https://zfsApplianceIPaddress:215/wiki/index.php/Shares:Projects:Replication#Replication_and_Clustering :

Replication can be configured from any ZFS Storage 7000 appliance to any other ZFS Storage 7000 appliance regardless of whether each is part of a cluster and whether the appliance's cluster peer has replication configured in either direction, except for the following constraints:

Configuring replication from both peers of a cluster to the same replication target is unsupported, but a similar configuration can be achieved using two different IP addresses for the same target appliance. Administrators can use the multiple IP addresses of the target appliance to create one replication target on each cluster head for use by that head.
When configuring replication between cluster peers, configure replication with both controllers in the CLUSTERED state. Do not use private network addresses and use separate replication targets for each controller's pools.

The following rules govern the behavior of replication in clustered configurations:

Replication updates for projects and shares are sent from whichever cluster peer has imported the containing storage pool.
Replication updates are received by whichever peer has imported the IP address configured in the replication action on the source. Administrators must ensure that the head using this IP address will always have the storage pool containing the replica imported. This is performed by assigning the pool and IP address resources to the same head during cluster configuration.
Replication updates (both to and from an appliance) that are in progress when an appliance exports the corresponding storage pool or IP address (as part of a takeover or failback) will fail. Replication updates using storage pools and IP addresses unaffected by a takeover or failback operation will be unaffected by the operation.

Set up appropriate routes for the interfaces used for replication. Eventually, if multiple interfaces are configured on the same network, use multihoming = adaptive feature.

See https://zfsApplianceIPaddress:215/wiki/index.php/Configuration:Network#Routing_Properties for further discussions.

Attachments

This solution has no attachment