| Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback |
| Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition | ||
|
|
||
 |
||||||||||||||||||||
Solution Type Problem Resolution Sure Solution 1551633.1 : iostat H/W Errors Increment for Multipathed Disks due to "SCSI Sense Key: Unit_Attention ASC: 0x2a (parameters changed), ASCQ: 0x0" when ZFS Modifies the Write Cache Mode
In this Document
Created from <SR 3-6889469681> Applies to:SPARC T4-2 - Version All Versions to All Versions [Release All Releases]Solaris Operating System - Version 10 1/13 U11 to 11.2 [Release 10.0 to 11.0] Information in this document applies to any platform. SymptomsThe iostat H/W Errors Increment for Multipathed Disks due to "SCSI Sense Key: Unit_Attention ASC: 0x2a (parameters changed), ASCQ: 0x0" when ZFS Modifies the Write Cache Mode. % iostat -en
---- errors --- s/w h/w trn tot device 0 2 0 2 c0t60060E800561CF00000061CF00000059d0 0 3 0 3 c0t60060E800561CF00000061CF00000058d0 0 3 0 3 c0t60060E800561CF00000061CF00000057d0 0 3 0 3 c0t60060E800561CF00000061CF00000056d0 0 3 0 3 c0t60060E800561CF00000061CF00000055d0 0 3 0 3 c0t60060E800561CF00000061CF00000054d0 0 3 0 3 c0t60060E800561CF00000061CF00000053d0 0 3 0 3 c0t60060E800561CF00000061CF00000052d0 0 2 0 2 c0t60060E800561CF00000061CF00000051d0 0 1 0 1 c0t60060E800561CF00000061CF00000050d0 Example messages: Apr 26 21:40:20 host scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g60060e800561cf00000061cf00000051 (ssd28):
Apr 26 21:40:20 host Error for Command: read(10) Error Level: Retryable Apr 26 21:40:20 host scsi: [ID 107833 kern.notice] Requested Block: 8448 Error Block: 8448 Apr 26 21:40:20 host scsi: [ID 107833 kern.notice] Vendor: HITACHI Serial Number: 50 061CF0051 Apr 26 21:40:20 host scsi: [ID 107833 kern.notice] Sense Key: Unit_Attention Apr 26 21:40:20 host scsi: [ID 107833 kern.notice] ASC: 0x2a (parameters changed), ASCQ: 0x0, FRU: 0x0 Example FMA events: % fmdump -eV -t "04/26/13 21:40" -T "04/26/13 21:41" -n "devid = id1,ssd@n60060e800561cf00000061cf00000051"
TIME CLASS Apr 26 2013 21:40:20.035801936 ereport.io.scsi.cmd.disk.dev.rqs.derr nvlist version: 0 class = ereport.io.scsi.cmd.disk.dev.rqs.derr ena = 0x29def6b388c0e001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev cna_dev = 0x517ad01b00000053 device-path = /pci@400/pci@2/pci@0/pci@a/SUNW,qlc@0,1/fp@0,0/ssd@w50060e800561cf10,1 devid = id1,ssd@n60060e800561cf00000061cf00000051 (end detector) devid = id1,ssd@n60060e800561cf00000061cf00000051 driver-assessment = retry op-code = 0x28 cdb = 0x28 0x0 0x0 0x0 0x1 0x20 0x0 0x0 0xe0 0x0 pkt-reason = 0x0 pkt-state = 0x3f pkt-stats = 0x0 stat-code = 0x2 key = 0x6 asc = 0x2a ascq = 0x0 sense-data = 0x70 0x0 0x6 0x0 0x0 0x0 0x0 0x58 0x0 0x0 0x0 0x0 0x2a 0x0 0x0 0x0 0x0 0x0 0x28 0x80 __ttl = 0x1 __tod = 0x517ae634 0x2224b50 Apr 26 2013 21:40:20.130640012 ereport.io.scsi.cmd.disk.dev.rqs.derr nvlist version: 0 class = ereport.io.scsi.cmd.disk.dev.rqs.derr ena = 0x29def6b388c0e005 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev cna_dev = 0x517ad01b00000053 device-path = /pci@400/pci@2/pci@0/pci@a/SUNW,qlc@0/fp@0,0/ssd@w50060e800561cf00,1 devid = id1,ssd@n60060e800561cf00000061cf00000051 (end detector) devid = id1,ssd@n60060e800561cf00000061cf00000051 driver-assessment = retry op-code = 0x28 cdb = 0x28 0x0 0x0 0x0 0x1 0x20 0x0 0x0 0xe0 0x0 pkt-reason = 0x0 pkt-state = 0x3f pkt-stats = 0x0 stat-code = 0x2 key = 0x6 asc = 0x2a ascq = 0x0 sense-data = 0x70 0x0 0x6 0x0 0x0 0x0 0x0 0x58 0x0 0x0 0x0 0x0 0x2a 0x0 0x0 0x0 0x0 0x0 0x28 0x80 __ttl = 0x1 __tod = 0x517ae634 0x7c9688c Apr 26 2013 21:40:20.230473769 ereport.io.scsi.cmd.disk.recovered nvlist version: 0 class = ereport.io.scsi.cmd.disk.recovered ena = 0x29def6b388c0e009 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev cna_dev = 0x517ad01b00000053 device-path = /pci@400/pci@2/pci@0/pci@a/SUNW,qlc@0,1/fp@0,0/ssd@w50060e800561cf10,1 devid = id1,ssd@n60060e800561cf00000061cf00000051 (end detector) devid = id1,ssd@n60060e800561cf00000061cf00000051 driver-assessment = recovered op-code = 0x28 cdb = 0x28 0x0 0x0 0x0 0x1 0x20 0x0 0x0 0xe0 0x0 pkt-reason = 0x0 pkt-state = 0x1f pkt-stats = 0x0 __ttl = 0x1 __tod = 0x517ae634 0xdbcc029 Another related symptom reported is Oracle Explorer run on Solaris host without flush write cache disabled may trigger the logging of many 1513 and 1517 errors on EMC VMAX storage. EMC reported the errors seen on the Symmetrix storage do not cause any known impact beyond filling the error buffer. 7C.1513.10 Indicates a MODE SELECT (SCSI CMD 15) issued by the host tried to change a non-changeable parameter.
7C.1517.00 Indicates reason MODE SELECT (SCSI CMD 15) failed with Sense Key = 05 (ILLEGAL REQUEST).
ChangesThis change in behavior was introduced by the fix for <Patch 147147-26> Synopsis: SunOS 5.10: kernel patch CauseNote there were no known data errors associated with the zpool devices due to this change in behavior. % zpool status -v mypool
pool: mypool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM mypool ONLINE 0 0 0 c0t60060E800561CF00000061CF00000050d0 ONLINE 0 0 0 c0t60060E800561CF00000061CF00000052d0 ONLINE 0 0 0 c0t60060E800561CF00000061CF00000054d0 ONLINE 0 0 0 c0t60060E800561CF00000061CF00000056d0 ONLINE 0 0 0 c0t60060E800561CF00000061CF00000058d0 ONLINE 0 0 0 errors: No known data errors The format utility can be used to verify the write cache mode of the disk. The example below was from a disk not even associated with a zpool, but the same behavior was observed for disk that were. % sudo format -e c0t60060E800561CF00000061CF00000051d0
selecting c0t60060E800561CF00000061CF00000051d0 [disk formatted] FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels inquiry - show disk ID scsi - independent SCSI mode selects cache - enable, disable or query SCSI disk cache volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> cache CACHE MENU: write_cache - display or modify write cache settings read_cache - display or modify read cache settings !<cmd> - execute <cmd>, then return quit cache> write_cache WRITE_CACHE MENU: display - display current setting of write cache enable - enable write cache disable - disable write cache !<cmd> - execute <cmd>, then return quit write_cache> display Write Cache is disabled write_cache> q CACHE MENU: write_cache - display or modify write cache settings read_cache - display or modify read cache settings !<cmd> - execute <cmd>, then return quit cache> q FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels inquiry - show disk ID scsi - independent SCSI mode selects cache - enable, disable or query SCSI disk cache volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> scsi Warning: these functions are intended for expert use only, for debugging disk devices and for unusual configuration settings. It is recommended that you do not use this menu for normal disk configuration and formatting, unless you have explicit instructions, or know exactly what you are doing. SCSI MENU: p<n> - display a mode sense page p<n> b<n> <op> [~]<n> - change a byte and issue mode select b<n> <op> [~]<n> - add an operation to the mode select list for the current page where: p<n> specifies the page with page code <n> b<n> specifies byte <n> of the page <op> can be one of the following operators: = (set specified value) |= (bitwise OR with current value) &= (bitwise AND with current value) <n> can be a decimal value in the range 0-255, or two hexadecimal digits, in the form 0x<xx>. [~] complements the specified value apply - apply mode select list cancel - cancel mode select list display - display mode select list all - display all supported mode sense pages default p<n> - mode select page <n> to default values default all - mode select all pages to default values format - format without standard mode selects inquiry - display device's inquiry response list - list common SCSI-2 mode pages !<cmd> - execute <cmd> , then return quit scsi> list Common SCSI-2 pages applicable to direct-access devices: Page 0x1 - Read-Write Error Recovery Page Page 0x2 - Disconnect-Reconnect Page Page 0x3 - Format Device Page Page 0x4 - Rigid Disk Geometry Page Page 0x7 - Verify Error Recovery Page Page 0x8 - Caching Page Page 0xA - Control Mode Page scsi> p8 Page 0x8: default: 88 0a 04 00 00 00 00 00 00 00 00 00 current: 88 0a 00 00 00 00 00 00 00 00 00 00 saved: 88 0a 00 00 00 00 00 00 00 00 00 00 changeable: 88 0a 04 00 00 00 00 00 00 00 00 00 The CACHE MENU provides a method to display or modify the contents of the Caching Page without having to understand the bits and bytes. Reference SCSI Block Commands (SBC-3) for the bit/byte definitions of the caching mode page (08h), where: How did the write cache get disabled? #!/usr/sbin/dtrace -Cqs
#define DKIOC (0x04 << 8) #define DKIOCSETWCE (DKIOC|37) BEGIN { printf("Control+C to interrupt\n"); } /* * Display set write cache enable/disable call from zfs */ fbt:sd:sdioctl:entry, fbt:ssd:ssdioctl:entry /args[1] == DKIOCSETWCE/ { printf("%Y %u %s : ",walltimestamp,timestamp,probefunc); printf("dev=0x%x cmd=%d --> DKIOCSETWCE \n",args[0],args[1]); printf("%s+%x triggered by: ", probefunc, arg0); printf("UID=%d PID=%d PPID=%d CMD=%s\n", curpsinfo->pr_euid,pid,curpsinfo->pr_ppid,curpsinfo->pr_psargs); stack(); } Used the dtrace script above to identify the functions where the write cache enable/disable was occurring: % sudo DKIOCSETWCE.d
Control+C to interrupt 2013 Apr 26 21:40:19 3134285886083274 ssdioctl : dev=0x8f00000000 cmd=1061 --> DKIOCSETWCE ssdioctl+8f00000000 triggered by: UID=0 PID=0 PPID=0 CMD=sched genunix`ldi_ioctl+0x5c zfs`vdev_disk_open+0x518 zfs`vdev_open+0xb0 zfs`vdev_open_child+0x8 genunix`taskq_thread+0x3a8 unix`thread_start+0x4 2013 Apr 26 21:40:20 3134287073902151 ssdioctl : dev=0x8f00000000 cmd=1061 --> DKIOCSETWCE ssdioctl+8f00000000 triggered by: UID=0 PID=0 PPID=0 CMD=sched genunix`ldi_ioctl+0x5c zfs`vdev_disk_close_handle+0x34 unix`thread_start+0x4 ZFS attempts to enable write caching on a disk whenever it performs an open on the whole disk, but won't if just a partition on the disk. The fix for Bug 15588234 changed the behavior of ZFS such that it will now attempt to disable write caching on a disk whenever it performs a close on the whole disk. The unintended side effect is that a zpool import scan can now result in write cache enable/disable on a disk that is not even associated with a zpool. These functions are defined in usr/src/uts/common/fs/zfs/vdev_disk.c: static int
vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift) { ... /* * If we own the whole disk, try to enable disk write caching. * We ignore errors because it's OK if we can't do it. */ if (vd->vdev_wholedisk == 1) { int wce = 1; (void) ldi_ioctl(dvd->vd_lh, DKIOCSETWCE, (intptr_t)&wce, FKIOCTL, kcred, NULL); dvd->vd_wholedisk = B_TRUE; } ... } static void
vdev_disk_close_handle(vdev_disk_t *dvd) { int wce = 0; if (dvd->vd_wholedisk) { (void) ldi_ioctl(dvd->vd_lh, DKIOCSETWCE, (intptr_t)&wce, FKIOCTL, kcred, NULL); } (void) ldi_close(dvd->vd_lh, dvd->vd_mode, kcred); kmem_free(dvd, sizeof (vdev_disk_t)); atomic_add_32(&vdev_active_threads, -1); } As the target device cannot assume that all initiators connected to a LUN belong to the same host, a unit attention is the correct behavior of a target device to report to other initiators that the caching mode page (08h) parameters have been changed by a "mode select" command and any assumptions regarding the caching state on the LUN now require reevaluation. SolutionSpecifying "cache-nonvolatile:true" in a [s]sd-config-list entry for the associated device type sets the un_f_suppress_cache_flush property on the LUN, whereby the DKIOCSETWCE IOCTL function of the [s]sd driver assumes the disk controller firmware will take care of managing the write cache, so there is no need to explicitly disable it. The following are examples of the ssd-config-list entries for setting max throttle, disabling flush write cache, and disksort in /kernel/drv/ssd.conf for Solaris 10 SPARC and /etc/driver/drv/ssd.conf for Solaris 11 SPARC: ssd-config-list = "EMC SYMMETRIX", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or: ssd-config-list = "HITACHI OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or: ssd-config-list = "HP OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or for all: ssd-config-list = "EMC SYMMETRIX", "cache-nonvolatile:true, disksort:false, throttle-max:32", "HITACHI OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32", "HP OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
The following are examples of the sd-config-list entries for setting max throttle, disabling flush write cache, and disksort in /kernel/drv/sd.conf for Solaris 10 x86/x64 and /etc/driver/drv/sd.conf for Solaris 11 x86/x64: sd-config-list = "EMC SYMMETRIX", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or: sd-config-list = "HITACHI OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or: sd-config-list = "HP OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
or for all: sd-config-list = "EMC SYMMETRIX", "cache-nonvolatile:true, disksort:false, throttle-max:32", "HITACHI OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32", "HP OPEN", "cache-nonvolatile:true, disksort:false, throttle-max:32";
NOTE:
NOTE:
Please reference the section on Customizing Driver Configuration in Oracle Solaris Administration: Devices and File Systems. Oracle Solaris 11: In this Solaris release, you can provide a supplemental driver configuration file, driver.conf, in the /etc/driver/drv directory. In previous Solaris releases, you had to modify the vendor's driver.conf files directly. In this release, the system automatically merges the driver vendor's /kernel or /platform driver.conf files with the customized /etc/driver/drv driver.conf files so that the driver sees both sets of property values. You can use the prtconf -u command to display the original vendor and customized property values. Separating vendor driver.conf files and locally customized driver.conf files allows the system to be upgraded with new vendor driver.conf files without overwriting your administrative customizations. If you have existing modifications in a driver configuration file in the /kernel/drv directory from a previous Solaris release or system, you should copy them over to the /etc/driver/drv directory. For information about creating an /etc/driver/drv/driver.conf file, see How to Customize a Driver Configuration, driver.conf(4), and driver(4). For example, ssd-config-list entries should go in a new file /etc/driver/drv/ssd.conf not /kernel/drv/ssd.conf. The new file /etc/driver/drv/ssd.conf should contain only properties which require customization.
It will also be necessary to manually reenable write cache on any existing LUNs using format. This will cause SCSI Sense Key: Unit_Attention ASC: 0x2a (parameters changed), ASCQ: 0x0 messages to occur one more time upon I/O down each path after the write cache is manually enabled using format. A command line similar to the following can be used to re-enable write cache on any existing LUNs: % for d (/dev/rdsk/c0t6*s0) { echo "cache\r write_cache\r enable\r quit\r quit\r quit\r" | sudo format -e $d }
Fix for <Bug 16777772> - fix for 15588234 causes zpool import to disable write cache on non zpool disks, was integrated into Solaris 11.3 GA and the following Solaris 10 patches: References<BUG:16777772> - FIX FOR 15588234 CAUSES ZPOOL IMPORT TO DISABLE WRITE CACHE ON NON ZPOOL DISKS<NOTE:1003338.1> - Sun StorEdge[TM] 9900 Series: Setting Queue Depth <NOTE:1122223.1> - How ZFS Manages Write Cache on Disk. <BUG:15318768> - SUNBT6397679 SCARY UNIT ATTENTION WARNING/ERROR WHEN ZFS ENABLES CACHE ON MULTIP <BUG:15588234> - SUNBT6880334-SOLARIS_11 ZPOOL DESTROY DOES NOT DISABLE THE WRITE CACHE ON A DISK <NOTE:1010505.1> - Solaris sd / ssd Driver - How to Disable the disksort Flag for Externally Cached Storage, Such as EMC and HDS Attachments This solution has no attachment |
||||||||||||||||||||
|
||||||||||||||||||||