Oracle ZFS Storage Appliance: Disabling Write Cache Enable (WCE) for ZFSSA exported LUNs on Solaris Clients

Asset ID:	1-71-2063355.1
Update Date:	2017-08-01
Keywords:

Solution Type Technical Instruction Sure

Solution 2063355.1 : Oracle ZFS Storage Appliance: Disabling Write Cache Enable (WCE) for ZFSSA exported LUNs on Solaris Clients

Applies to:

Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
Sun Storage 7310 Unified Storage System - Version All Versions and later
Sun Storage 7410 Unified Storage System - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
7000 Appliance OS (Fishworks)
Due to Solaris Bug 15662604 - "ZFS should retry or abort an entire transaction when a WCE LUN gets a power-on-reset condition" and ZFS-SA Bug 15753332 - "appliance should ignore over the wire dkiocsetwce" Solaris clients may experience data integrity issues when the write cache has been enabled on LUNs exported from an Oracle ZFS Storage Appliance.

NOTE: This also applies to Solaris LDOM's using Vdisks created on top of ZFS-SA LUNs, so if the client LDOM or zone is putting a zpool on the vdisk the fix below will also need to be applied.

Goal

To permanently disable Write Cache on LUNs exported from an Oracle ZFS Storage Appliance when used in a zpool on a Solaris client.

Background:

ZFS enables the write cache on pool devices upon zpool import, and safely handles cache flushing in the event of a system power loss. However, a power-on-reset condition can potentially occur while data has not yet been committed to stable storage.

In an environment with no single point of failure, this situation is automatically detected and corrected by ZFS the next time the data is read. Routine pool scrubs of the pool can increase the detection and repair of any lost writes.

In an environment with a single point of failure, this problem could lead to data loss.

This problem might also occur more frequently when accessing LUNs that are exported from a clustered configuration. During cluster failover, data cached by the failing head may be lost due to a power-on-reset event that is explicitly sent by the SCSI target on the surviving head. In this situation, even pools with no single point of failure might be affected.

A symptom of this issue is clusters of persistent checksum errors. You can use the output from fmdump –eV to determine whether the checksum errors have been diagnosed as persistent. The zio_txg entry in the fmdump –eV output represents the time that a block of data is written. Note that a pattern of persistent checksum errors could also be a symptom of failing devices, software, or hardware.

Solution

First of all, the procedures described below will not disable the write cache if it is already enabled, it will only prevent the enable to occur during zpool import.

You will first need to manually disable the write cache using Solaris 'format -e' or by using the Oracle ZFS Storage Appliance CLI or BUI to disable the write cache for each LUN.

Procedure for releases before s11.1

In previous Solaris releases, you will have to use a slice of the LUNs to prevent the the write-cache from being enabled on the ZFS appliance, this method will also work on newer Solaris releases.

This means you will need to put an partition/slice table on the LUN using Solaris 'format' and define a slice which will consume all the space on the LUN. (Do not use s2)

Then when creating the zpool make sure to use the configured slice. ie 'zpool create tank mirror c1t0d0s0 c1t1d0s0' instead of c0t0d0 and c0t1d0. Using the latter devices will put a new label on the LUNs and enable the write cache.

Procedure for s11.1 and later

With this approach, the write cache will never be enabled, even if you recreate the pool or reboot.

For a Solaris client using LUNs exported via ISCSI from a ZFS Storage Appliance, sd.conf will have to contain the following entries :

On a Sparc system :

      sd-config-list="*ZFS Storage*","disable-cache-suppress-flush";
      disable-cache-suppress-flush=1,0x40010,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1;

Non-bitmap style :

sd-config-list = "*ZFS Storage*", "cache-nonvolatile:true, disable-caching:true, disksort:false";

On an x64 system:

sd-config-list="*ZFS Storage*","disable-cache-suppress-flush";
disable-cache-suppress-flush=1,0x40008,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1;

Non-bitmap style :

sd-config-list = "*ZFS Storage*", "cache-nonvolatile:true, disable-caching:true, disksort:false";

For a Solaris client using LUNs exported via FC from a ZFS Storage Appliance, ssd.conf will have to contain the following entries:

On a Sparc system :

ssd-config-list="*ZFS Storage*","disable-cache-suppress-flush";
disable-cache-suppress-flush=1,0x40010,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1;

Non-bitmap style :

ssd-config-list = "*ZFS Storage*", "cache-nonvolatile:true, disable-caching:true, disksort:false";

On an x64 system:

ssd-config-list="*ZFS Storage*","disable-cache-suppress-flush";
disable-cache-suppress-flush=1,0x40008,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1;

Non-bitmap style :

ssd-config-list = "*ZFS Storage*", "cache-nonvolatile:true, disable-caching:true, disksort:false";

If you are using both FC and ISCSI or you may plan to do it in the future, you can do the changes in both files.

To verify the changes were made properly, issue the following :

% for d in 'echo "*ssd_state::walk softstate" | sudo mdb -k' ; do echo "$d::print struct sd_lun un_sd | ::print struct scsi_device sd_dev | ::print struct dev_info devi_addr" | sudo mdb -k ; echo "$d::print struct sd_lun un_sd | ::print struct scsi_device sd_inq | ::print struct scsi_inquiry inq_vid inq_pid inq_revision inq_serial" | sudo mdb -k ; echo "$d::print struct sd_lun un_f_opt_disable_cache un_f_disksort_disabled un_f_suppress_cache_flush" | sudo mdb -k ; done
devi_addr = 0x3040047031500 "g600144f0b4fe45f4000057b3b7050005"
inq_vid = [ "SUN " ]
inq_pid = [ "ZFS Storage 7330" ]
inq_revision = [ "1.0 " ]
inq_serial = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0',
'\0' ]
un_f_opt_disable_cache = 1
un_f_disksort_disabled = 1
un_f_suppress_cache_flush = 1

NOTE: Use of the new simpler format [s]sd-config-list cannot be intermixed with the old bitmask style [s]sd-config-list, as the old bitmask style settings will just be ignored.

Support of the old bitmap style will be removed in a future Solaris release.
Bug 18894959 - Remove support for old-style sd.conf tunables

Then reboot the client.

References

https://docs.oracle.com/cd/E36784_01/html/E36797/goggo.html
<BUG:15662604> - SUNBT6977015 ZFS SHOULD RETRY OR ABORT AN ENTIRE TRANSACTION WHEN A WCE LUN GETS
<BUG:15675550> - SUNBT6992811 SUPPORT FOR ZVOL DKIOCSETWCE SHOULD BE CONFIGURABLE
<BUG:15753332> - SUNBT7109194 APPLIANCE SHOULD IGNORE OVER THE WIRE DKIOCSETWCE

Attachments

This solution has no attachment