Solaris Operating System
 
SPARC T4-1
 

PLA-Support>Sun Systems>DISK>HBA>SN-DK: FC HBA
 

In this Document

Symptoms

Changes

Cause

Solution

References

Created from <SR 3-7774615141>

Applies to:

Symptoms

This is a Solaris 11.1 SRU 4.5 T4-1 SPARC server t4server1 with Oracle FC HBA Dual port,
both ports connected to a SAN to access a Netapp storage array:


c9 = emlxs1 (fp3) -> /devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0:devctl
c8 = emlxs0 (fp2) -> /devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0/fp@0,0:devctl


There are several LUNs mapped from the storage to this server under mpxio control , ie:

Each LUN has 8 paths under mpxio, all ONLINE,
four primary to storage controller A ports and four secondary to storage controller B ports:

--> There was failure on the Netapp storage controller A and all ports disappear from fabric as expected:

Access to the LUNs of the Netapp storage was provided through the other paths to controller B

The next day the storage controller A was fixed and its ports reappear:

...but just before that, with no other explanation there is a failure on zpool pool1 (disk 3. c0t60A9800037537478742443536131576Fd0 )

Then mpxio place the multipath status as optimal for each LUN/path from the Netapp storage, ie for LUN 0x21 :

The only errors we found are on fma, due to that the zfs pool was marked as faulty:

Here is the detailed fmdump events that lead to the zfs failure,
same error (asc = 0x4, ascq = 0xa) on the four available paths under mpxio (to controller B)

This LUN is ssd10 :
"/scsi_vhci/ssd@g60a9800037537478742443536131576f" 10 "ssd"

At this point customer initiated manually a Solaris crash dump to collect information.

Changes

Netapp storage controller failure and controller fixed reappear into the fabric.

Cause

All the data points to this mpxio bug as the RCA:
Bug 15822598 SUNBT7204589 scsi_vhci not handling the takeover on netapp storage correctly
and

Bug 17228789  scsi_vhci not handling the takeover on netapp storage correctly in non cluster

Based on the fma data and the dump collected and the Solaris 11.1 version , the server did not know how to handle that 04/0A sense data (probably due to bug 15822598 )
and was passed to the upper layers so fmd catch the error and caused ZFS to fail the pool.
 

Solution

Upgrade to Solaris 11.2 SRU 11.2.12.5.0 (or greater) that fixes Bug 17228789

On Solaris 10, fix for Bug 17228789 has been provided on:

<SunPatch:150400-28>         Sep/10/2015   SunOS 5.10: Kernel Patch
<SunPatch:150401-28>         Sep/10/2015   SunOS 5.10_x86: Kernel Patch

Notice that the fix provided on Bug 15822598 does not cover all scenarios:

On Solaris 11, fix for Bug 15822598 has been provided on:

Solaris 11.1 SRU 7.5 

On Solaris 10, fix for Bug 15822598 has been provided on:

<SunPatch:150400-23>         Apr/08/2015   SunOS 5.10: Kernel Patch
<SunPatch:150401-23>         Apr/08/2015   SunOS 5.10_x86: Kernel Patch

Internal notes from the core dump analysis:

 -----------------------
The analysis of the core dump is not conclusive in relation to the bug "Bug 15822598 SUNBT7204589 scsi_vhci not handling the takeover on netapp storage correctly"
as the dump was taken after the storage issue was seen and the multipathing devices has sorted themselves out.ssd9 and ssd11 have no outstanding commands in the driver or transport and the
Last pkt reason:
CMD_CMPLT - no transport errors- normal completion

ssd10 and ssd12 have no outstanding commands in the driver or transport but the
Last pkt reason:
  CMD_TRAN_ERR - unspecified transport error


we can see from the kstat errors that ssd10 had many transport and hard errors logged against it

kid: 1772 @ 0x3000099e5b0, data[14] @ 0x3000099e6c0
mod: ssderr name: ssd10,err class: device_error type: KSTAT_TYPE_NAMED
inst: 10 flags: 8 size: 672
  KSTAT_FLAG_PERSISTENT - kstat is to be persistent over time
creation time: 0x26d79efb14 (144 days 3 hours 51 minutes 26.806454690 seconds earlier)
last snapshot: 0x2bed0b55065713 (1 days 1 hours 26 minutes 36.707272099 seconds earlier)
update: genunix:nulldev
snapshot: unix:default_kstat_snapshot
Soft Errors: 0
Hard Errors: 2698
Transport Errors: 3294
Vendor: "NETAPP "
Product: "LUN "
Revision: "811a"
Serial No: "7Stxt$CSa1Wo"
Size: 4294967296000
Media Error: 0
Device Not Ready: 5
No Device: 5
Recoverable: 0
Illegal Request: 3481
Predictive Failure Analysis: 0

Looking at the ssd10 we can see historically that the sense data from the array
is asc/ascq ( 04/ 0A) ( LOGICAL UNIT NOT ACCESSIBLE, ASYMMETRIC ACCSSS STATE TRANSITION)


> 0x1001915c0700::print -t struct sd_lun un_xbuf_attr
void *un_xbuf_attr = 0x1001913f0a00
> 0x1001913f0a00::print -t struct __ddi_xbuf_attr xa_reserve_headp
void *xa_reserve_headp = 0x100190de6550
> 0x100190de6550::print -t ddi_xbuf_t
ddi_xbuf_t 0x100190dba660
> 0x100190dba660::print -t sd_xbuf xb_sense_data
uchar_t [20] xb_sense_data = [ 0x70, 0, 0x2, 0, 0, 0, 0, 0xe, 0, 0, 0, 0, 0x4, 0xa, 0, 0, 0, 0, 0, 0 ]
>

So as per the bug you mentioned if the array is returning this sense data the version of Solaris the customer is currently using
will not understand what to do with it.


The sense data returned by the array

sense-data = 0x70 0x0 0x2 0x0 0x0 0x0 0x0 0xe 0x0 0x0 0x0 0x0 0x4 0xa 0x0 0x0 0x0 0x0 0x0 0x0

contains a sense key = 0x2 ( Not Ready)

asc/ascq = 0x4/0xA

If the customer does not have the version of Solaris with the fix as defined in the bug then Solaris does not know what to do with the the 0x4/0xa additional sense data.

It knows that the lun is "not ready" and the only code before the fix is

} else if ((skey == KEY_NOT_READY) &&
(asc == STD_LOGICAL_UNIT_NOT_ACCESSIBLE) &&
((ascq == STD_TGT_PORT_UNAVAILABLE) ||
(ascq == STD_TGT_PORT_STANDBY))) {
rval = SCSI_SENSE_INACTIVE;
VHCI_DEBUG(4, (CE_NOTE, NULL, "!std_analyze_sense:"
" sense_key:%x, add_code: %x, qual_code:%x"
" sense:%x\n", skey, asc, ascq, rval));

but will ignore this else if statement and rval will be set to SCSI_SENSE_UNKNOWN ( as per the bug comments ) and dealt with accordingly as it would with unknown sense data.


With the fix in place ( additional else if )

} else if ((skey == KEY_NOT_READY) &&
(asc == STD_SCSI_ASC_STATE_TRANS) &&
(ascq == STD_SCSI_ASCQ_STATE_TRANS)) {
rval = SCSI_SENSE_NOT_READY;
VHCI_DEBUG(4, (CE_NOTE, NULL, "!std_analyze_sense:"
" sense_key:%x, add_code: %x, qual_code:%x"
" sense:%x\n", skey, asc, ascq, rval));

where

#define STD_SCSI_ASC_STATE_TRANS 0x04
#define STD_SCSI_ASCQ_STATE_TRANS 0x0A

so now the return value is set to SCSI_SENSE_NOT_READY and dealt with differently.

dissembling std_analyze_sense() in the dump we only see ref to the first (if else )

std_analyze_sense+0xf0: cmp %i2, 0x2 ( KEY_NOT_READY )
std_analyze_sense+0xf4: bne,pn %icc, +0x3c
std_analyze_sense+0xf8: cmp %i3, 0x4 (STD_LOGICAL_UNIT_NOT_ACCESSIBLE)
std_analyze_sense+0xfc: be,pn %icc, +0x14
std_analyze_sense+0x100: mov 0x5, %l3
std_analyze_sense+0x104: sra %l3, 0x0, %i0
std_analyze_sense+0x108: ret
std_analyze_sense+0x10c: restore
std_analyze_sense+0x110: sub %l1, 0xb, %l5 ( STD_TGT_PORT_STANDBY )

So this customer does not have that bug fix on their system, and they should update their packages to get it installed.

I would say that the customer should update their version of Solaris
so that this bug will not cause them any further issue when the array returns 04/0A sense data.
-------------------------------

References

This solution has no attachment