Sun ZFS Storage 7420
 
Oracle ZFS Storage ZS5-2
 
Oracle ZFS Storage ZS3-2
 
Oracle ZFS Storage ZS4-4
 
Sun Storage 7410 Unified Storage System
 
Oracle ZFS Storage ZS5-4
 
Sun Storage 7310 Unified Storage System
 
Oracle ZFS Storage ZS3-4
 
Oracle ZFS Storage Appliance Racked System ZS4-4
 
Sun ZFS Storage 7320
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 
_Old GCS Categories>Sun Microsystems>Storage - Disk>Unified Storage
 

Applies to:

Purpose

The appliance does fail-back and takeover and there is an important distinction between them since the former requires resources be exported before they are imported by the other head, whereas takeover just imports them.

In the case of a slow fail-back, its worth figuring out if the relinquishing head is slow to export or whether the claiming head is slow to import the resources.

Troubleshooting Steps

The takeover and fail-back times depends on the number of objects that need to be iterated during the resource import phase.

On the 7x20 and 7x10 series system those objects include: shares, LUNs, data-links, V-LANs, network interfaces, IPMP/LACP setup, iscsi/fc targets, initiators, and groups, etc. Simple configurations are faster than complex configurations.

Other considerations:

• If there is IO to the pool at the time of takeover/failback... especially writes since that would yield more dirty data in the Logzillas to be "replayed" during takeover. Time should increase. Reads should not make a difference

• When iSCSI/FC LUNs are in use and in the case of a takeover, the contents of the logzilla will need to be replayed before the zpool can be imported

• If there are many CIFS clients authorized by an Active Directory Server, more time will be needed to perform re-authorization upon the peer cluster head after takeover or failback.

• If a destroy is in progress this needs to be completed before the zpool can be imported. This has been seen with snapshots especially where the head being taken over was in the process of destroying a snapshot which the other head then had to complete before the pool could be imported. This situation is remedied in appliance software 2011.1.1.0.

• Finally, if they are available the readzilla L2ARC caches will need warming up after their associated pool is imported. Note that this does not apply to the logzillas, because they are imported along with the rest of the pool whereas the readzillas are specific to each cluster head.

Identifying the problem
For reference, the expected takeover time (for 7x10 SAS-1 systems) is:
Time in seconds = (20 * D) + (.03 * S)
D is # of disksets (half JBODs)
S is # of shares (filesystems)

NOTE: For 7x20/ZS3-x SAS-2 systems, see below.

How to make sure failover times are acceptable

Given the considerations above partially explaining expected failover times, failover times should now be less than 5 minutes in 2013 release.

Appliance cluster failover times have substantially decreased with the new 2013 software release.

If a time greater than 5 minutes (3 minutes in 2013.1.4.x code) is observed either when failing back, or taking over, then here are the decisions customers might make :

1. Make sure the cluster is upgraded to the latest AK 2013.1.x release

        If this is not already done, then a cluster activity must be scheduled.

2. Limit LUNs/Shares to 500 or less.  This is the most impactful item to failover times

3. Limit Analytics and using only when needed. The biggest 'offenders' are generally ARC-related DTrace Probes

4. For analysis, open a Service Request to TSC, and send one supportbundle for each node to the engineer in charge of your Service Request.

        Also provide accurate times when such a long failback and/or takeover has been observed.

Support bundles contain logs recording the latest failbacks and takeovers having been issued on the cluster.

Therefore TSC engineer will be able to tell if this failback time has been greater than the 5 minutes (3 minutes in 2013.1.4.x code) expectation.

If this is the case, further actions will be done to point this issue to engineering and some additional traces might be requested in a second time.

Logs can be found in /var/ak/logs on the live system and in /logs directory in the bundle.

Failover times can be found in rm.ak* logs, using aklog macro.  MOS documentation below may help to use aklog properly.

Here is an example showing how long lasted a takeover : 

 
$aklog ./rm.ak | egrep -e 'import|export|takeover|failback'

... output cut

Mon Oct 21 20:09:29 2013: import of ak:/fct/DE24P_20-10k-2wa-SP succeeded in 0.027s 

Mon Oct 21 20:09:29 2013: import of ak:/fct/DE24C_20-7.2k-1wa-SP succeeded in 0.030s

Mon Oct 21 20:09:30 2013: takeover completed in 92.469s 

It lasted 92 seconds, that is to say 1 minute and a half.

Failback times can be found in the rm.ak* log as well, but on both cluster peers : one peer exports resources, the other peer imports those resources

On node 1

On node 2

Mon Oct 21 20:28:18 2013: failback completed in 16.809s 

For failback, time on node 1 and time on node2 must be added to get the total time of failback. Here it lasted about 37 seconds in total.
 
Those times are reliable in ak.2013 firmware and they tell how long customer has effectively waited to have data services transferred from one node to the other.

NOTE: For later Appliance Firmware Release versions, the 'akrmzfs.txt' file is included in the supportbundle - it prints EVERYTHING about a takeover/failback.

• It reports how long every routine took to run (in micro seconds), how long the datasets took, as well as times to Unmount, Export, Import and Mount.
• It even makes a little bar graph to review

New Data gathering script (01-2013)

Here a  link to a new data gathering script called failback_monitor.akwf,
supplied by RPE, to be used to diagnose long fail-over issues
more then 5 minutes. For access to the wiki/AmberRoadSupport do you need privileges.

Ask Sailesh Thanki for this

Link to the workflow:  https://stbeehive.oracle.com/teamcollab/wiki/AmberRoadSupport:Long+failback+-+Data+collection

Checking takeover/failback times from a support bundle:

Refer to the supportbundle log-file rm.ak
for example :
bash-3.2$  cd  /cores/sr-id/supportbundle
bash-3.2$  find . -type f -exec grep "in 0." {} /dev/null \;


It will give you an overview about how long the export and import of some items takes.

Example from 15772458

adc26stor08:configuration cluster> date
2012-2-11 08:59:46
adc26stor08:configuration cluster> failback
Continuing will immediately fail back the resources assigned to the cluster
peer. This may result in clients experiencing a slight delay in service.

Are you sure? (Y/N)
date
adc26stor08:configuration cluster> date
2012-2-11 09:06:24

on the exporting node (08) we see that the pools take the longest:
adc26stor08# aklog rm | grep -i export | grep "Sat Feb 11 09:0" | grep -v "in 0." | tail -20
Sat Feb 11 09:01:26 2012: export of ak:/nas/pool07a succeeded in 95.727s
Sat Feb 11 09:04:10 2012: export of ak:/zfs/pool07a succeeded in 164.224s
adc26stor08#

On the importing node (07), they are biggest hitters also:
adc26stor07# aklog rm | grep -i import | grep "Sat Feb 11 09:0" | grep -v "in 0." | tail -20
Sat Feb 11 09:05:14 2012: [zfs import] zpool_import_props() succeeded in 61.090s
Sat Feb 11 09:05:14 2012: import of ak:/zfs/pool07a succeeded in 61.129s
Sat Feb 11 09:06:03 2012: [nas import] discovery completed in 48.400s
Sat Feb 11 09:06:16 2012: [nas import] mounted 673 datasets in 6.989s
Sat Feb 11 09:06:17 2012: import of ak:/nas/pool07a succeeded in 62.531s
Sat Feb 11 09:06:20 2012: import of ak:/net/ixgbe93003 succeeded in 1.649s
adc26stor07#


Here is very useful dtrace script that allows checking which operation takes the most of time.

Dtrace script import.d helps to troubleshoot long cluster takeover and fail-back times. The script measures the time to import each resource.

Output:
The first table is the aggregate time spent importing each resource, the second is the number of times it was imported. The special "resource" SAS LOCK is just the time taken to grab all the zone locks in the expanders. These two activities are basically all there is to takeover so they should capture everything that consumes time.

Checked for relevancy - 10-May-2018

References

This solution has no attachment