Scsi Error Sense Key: Not Ready ASC: 0x4 (<vendor unique code 0x4>), ASCQ: 0x3

Asset ID:	1-72-1924145.1
Update Date:	2018-03-07
Keywords:

Solution Type Problem Resolution Sure

Solution 1924145.1 : Scsi Error Sense Key: Not Ready ASC: 0x4 (), ASCQ: 0x3 - Reported by EMC Disk Storage Array

Applies to:

Qlogic FC HBA - Version Not Applicable and later
Emulex FC HBA - Version All Versions and later
Sun Fire V440 Server - Version All Versions and later
Solaris Operating System - Version 8.0 and later
Information in this document applies to any platform.

Symptoms

This Solaris 9 SF V440 server1 has two native Emulex LP10000 FC HBAs (non-Oracle FC HBA, using third party driver lpfc)
two ports connected to the SAN to access an EMC storage array, emc powerpath used as multipathing software, LUNs under vxvm control.

There are no SAN packages or patches installed, expected as lpfc is a third party driver.

We can see the EMC disks under c3 and c4 controller:

bash-3.2$ more format.out
Searching for disks...done

  4. c3t30d1
  /pci@1e,600000/lpfc@2/sd@1e,1
...
  11. c3t30d8
  /pci@1e,600000/lpfc@2/sd@1e,8

  12. c4t31d1
  /pci@1e,600000/lpfc@3/sd@1f,1
...
  19. c4t31d8
  /pci@1e,600000/lpfc@3/sd@1f,8

  20. emcpower0a
  /pseudo/emcp@0
...
  29. emcpower297a
  /pseudo/emcp@297

The two third party FC HBAs LP10000

bash-3.2$ more prtdiag-v.out

pci 33 PCI0 lpfc-pci10df,fa00 (scsi) LP10000
  okay /pci@1e,600000/lpfc@2
pci 33 PCI0 lpfc-pci10df,fa00 (scsi) LP10000
  okay /pci@1e,600000/lpfc@2,1

pci 33 PCI1 lpfc-pci10df,fa00 (scsi) LP10000
  okay /pci@1e,600000/lpfc@3
pci 33 PCI1 lpfc-pci10df,fa00 (scsi) LP10000
  okay /pci@1e,600000/lpfc@3,1

On the messages we can see errors from the EMC storage on both FC HBA ports (/pci@1e,600000/lpfc@2 and /pci@1e,600000/lpfc@3 ) that cause the vxfs file systems to fail:

Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 107247376 Error Block: 107247376
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 117733136 Error Block: 117733136
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,8 (sd2459):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,6 (sd2457):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:23 server1 Error for Command: write Error Level: Fatal
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,6 (sd2457):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 117733136 Error Block: 117733136
Aug 22 05:00:23 server1 Error for Command: write Error Level: Fatal
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 65808 Error Block: 65808
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7512D008Q
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,6 (sd2457):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 Error for Command: write Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,8 (sd2459):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,6 (sd2457):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 65808 Error Block: 65808
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 Error for Command: write Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 109344528 Error Block: 109344528
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7512D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:24 server1 vxio: [ID 771159 kern.warning] WARNING: VxVM vxio V-5-0-2 Subdisk EMC375L0127E-01 block 16: Uncorrectable write error
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@3/sd@1f,8 (sd2459):
Aug 22 05:00:24 server1 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 13 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/PSSAPAUZ44501/data_app file system super-block write error
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:24 server1 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 14 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/PSSAPAUZ44501/data_app file system disabled
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 109344528 Error Block: 109344528
Aug 22 05:00:24 server1 vxio: [ID 771159 kern.warning] WARNING: VxVM vxio V-5-0-2 Subdisk EMC375L0128E-04 block 16: Uncorrectable write error
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Requested Block: 107247376 Error Block: 107247376
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:24 server1 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 15 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/PSSAPAUZ44501/var_app file system super-block write error
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:24 server1 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 16 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/PSSAPAUZ44501/var_app file system disabled
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0

Cause

Undetermined, not a Solaris problem, but an EMC storage problem.

The EMC storage is reporting that the LUNs are Not Ready, ie for LUN 8

Aug 22 05:00:23 server1 scsi: [ID 107833 kern.warning] WARNING: /pci@1e,600000/lpfc@2/sd@1e,8 (sd2204):
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Vendor: EMC Serial Number: 7514D008Q
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 Error for Command: write(10) Error Level: Fatal
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
Aug 22 05:00:23 server1 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 0x3, FRU: 0x0

The meaning of "ASC: 0x4 (), ASCQ: 0x3" can be found here:

From http://www.t10.org/lists/asc-alph.htm
04/03 DTLPWROMAEBKVF LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

We can find another example on Symantec Document TECH83145 , the "ASC: 0x4 (), ASCQ: 0x3" can happen
"due to transient failure such as a Service Processor (SP) going temporarily offline (for example, due to a reboot of the SP)"

Solution

Check with EMC what is happening on the storage side when the errors occurs.

Check the FC switches where the FC HBAs and EMC storage arrays are connected to discard any FC errors ,
BUT if each FC HBA port is connected to a different switch, it is very improbable that the same problem occurrs on two different fabrics, so the problem seems to be on the EMC storage side.

In case you think there is a problem on the FC HBAs or with lpfc driver, notice these are non-Oracle FC HBA Oracle does not support , neither lpfc driver.

Notice. These errors can bee also reported by EMC special logical unit called "LUNZ", see this document, but this is not our case:
Solaris Storage Driver Best Practice for EMC Array Special Logical Device (LUNZ, VMCDB, Gatekeeper, etc ...) (Doc ID 1006505.1)

References

<NOTE:1006505.1> - Solaris Storage Driver Best Practice for EMC Array Special Logical Device (LUNZ, VMCDB, Gatekeeper, etc ...)

Attachments

This solution has no attachment