Sun ZFS Storage 7320
 
Oracle ZFS Storage ZS5-4
 
Oracle ZFS Storage ZS3-2
 
Oracle ZFS Storage ZS3-4
 
Oracle ZFS Storage ZS5-2
 
Sun ZFS Storage 7420
 
Oracle ZFS Storage ZS4-4
 
Sun ZFS Storage 7120
 

PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
 

In this Document

Symptoms

Changes

Cause

Solution

References

Applies to:

Symptoms

After resilver completion of a replacement disk drive, data corruption is seen in a zpool.

Changes

Upgrading to the AK8.7.x code baseline from an older version brings in effect a change in the resilver methodology utilizing a sequential resilver algorithm.

Cause

There are 4 bugs currently identified involved with the sequential resilver algorithm that can cause data corruption on a resilver in the ak8.7.x code versions.

1. As described in this Alert knowledge document:

Resilvering a Device in a ZFS Pool may Result in checksum Errors and Possible Data Integrity Issues (Doc ID 2340720.1)

the listed bug 27200134 can cause data corruption on a resilver in AK code versions 8.7.0 through 8.7.12.

2. Also as described in this Alert knowledge document:

Solaris 11 Systems may Experience Checksum Errors and Possible Data Integrity Issues After ZFS Resilvers a Pool (Doc ID 2300662.1)

The listed in this document bug 25674602 can also cause data corruption on a resilver in AK code versions 8.7.0 through 8.7.14.

NOTE: Please read the 2300662.1 document carefully as there are circumstances where this bug can happen in AK code versions

8.2.0 through 8.6.15.

3. Also as described in this Alert knowledge document:

ZFS Storage Appliance (ZFSSA) Pool may Report Persistent Checksum Errors Post-Resilver when Deduplication is Enabled (Doc ID 2377391.1)

The listed in this document bug 27503371 can also cause data corruption on a resilver on zpools with deduplication enabled in AK code versions

8.7.0 through 8.7.14.

4. Also as described in this Alert knowledge document:

ZFS Storage Appliance (ZFSSA) Pool may Report Persistent Checksum Errors Post-Resilver (Doc ID 2389403.1)

The listed in this document bug 27730831 is resolved in released code version ak8.7.18.

For this bug the problem can happen in ak code versions 8.7.0 through 8.7.17.

Solution

 It is the recommendation of the ZFSSA Array Support group that all arrays running ak code versions ak8.7.0 through ak8.7.17 be upgraded to the ak8.7.18 version released May 1st.

NOTE: As of the release of ak code ak8.7.18 there are no known resilver issues and there is no need for any resilver workflows to be applied to the ak8.7.18 code.

If the upgrade to that recommended version can not be done and the ak code version is ak8.7.0 through ak8.7.17 then the resilver workflow resilver-1pass.akwf

should be downloaded and applied and in place.

1. NOTE: As indicated by the referenced in the cause section knowledge document 2340720.1, bug 27200134 is resolved in ak code 8.7.13 and above.

2. The Bug 25674602 referenced in the cause section of knowledge document 2300662.1 is resolved in code version ak8.7.17 and above.

3. The Bug 27503371 referenced in the cause section of knowledge document 2377391.1 is resolved in code version ak8.7.17 and above and if the code version is below 8.7.17 it's related document also lists this as another workaround besides the workflow below: "Wait for any spare resilvering to complete before replacing a failed drive. Note that when a drive is pulled, the auto spare kicks in, and then if the pulled drive is replaced, it will start to resilver that drive, which will hit this issue."

4. The Bug 27730831 referenced in the cause section of knowledge document 2389403.1 is resolved in released May 1st ak code version 8.7.18.  For this bug if the ak code version is ak8.7.0 through 8.7.17 then the recommended action is to upgrade to ak8.7.18 or implement the resilver workflow.

The workflow can be downloaded here:

 resilver-1pass.akwf  This link will download the version 1.7 workflow

NOTE: The implementation of this workflow changes the resilver algorithm back to the previous methodology.

NOTE: When applying this workflow to an array cluster the workflow must be individually run on both nodes in order for it to be active on both.

NOTE: Any code upgrade removes this workflow and if the upgrade is to any version still in the affected range the workflow must be reapplied to still be in effect.

NOTE: There are 2 versions of this workflow that have been distributed.

The original version 1.6 of workflow was only able to be applied to arrays running ak code versions ak8.5.4 thru ak8.7.12.

If you have workflow version 1.6 it will not allow it to be applied on ak code versions 8.7.13 or above you must have the 1.7 version.

The version 1.7 downloadable from the above link can be applied to ak code versions ak8.5.4 and above including all versions of 8.7.

Use this version 1.7 to apply this workflow on arrays running from ak8.7.0 and above (no top limit).

NOTE: If recommended and necessary based on special circumstance described in document 2300662.1 above,

either version of the workflow above can be applied to ak code version 8.5.4 through 8.6.15.

NOTE: The only difference between the 2 versions of this workflow is the upper end code version range it can be applied to.

TSC engineers can run this one line from the array shell to implement this workaround:

echo zfs_no_seq_resilver/W 1 | mdb -kw ; grep -v zfs_no_seq_resilver /etc/system > /tmp/system && echo 'set zfs:zfs_no_seq_resilver=1' >> /tmp/system && mount -o remount,rw / && cp /etc/system{,-} && mv /tmp/system /etc/system && mount -o remount,ro / && bootadm update-archive

The /etc/system file can be checked for the "set zfs:zfs_no_seq_resilver=1" line to see if the workflow is in place or check the audit logs for the run of the workflow.

NOTE: If a resilver is in progress the workflow can be applied however it will not affect the current resilver the recommendation if a resilver is in progress is to apply the workflow and scrub the pool when the current resilver completes.

 If there is ever any doubt about the integrity of a zpool the process is to run a scrub on that zpool.

References

This solution has no attachment