Solaris Operating System
 
Sun Blade T6320 Server Module
 
Solaris Operating System
 
Qlogic FC HBA
 

PLA-Support>Sun Systems>DISK>Storage Drivers>SN-DK: Storage Drivers
 

Applies to:

Goal

This article addresses the SCSI transport errors and how they can be identified and investigated.

Solution

1. Introduction

In the Solaris Operating System, the architecture of the SCSI subsystem can be summarized by a stack of three layer:

• upper/target layer: target device driver like
  
  • sd: driver for disk device
  • st: driver for tape device
  • sg: generic driver
• mid layer: MPxIO multipathing driver or other 3rd party multipathing driver such as Symantec's VxDMP or EMC's PowerPath
• lower/transport layer: transport layer implementing various transport protocol (non exhaustive list of host bus adapters drivers included in Solaris Operating System)
  
  • scsi: isp, esp, fas, glm, mpt drivers
  • sas: mpt_sas, pmc drivers
  • fc: fc/fcp and qlc or emlxs driver
  • fcoe: fc/fcp and qlge
  • iscsi: iscsi driver

Upon completion of an SCSI command the host adapter calls the completion routine provided by the target driver. The scsi_pkt structure member pkt_reason contains a completion code that indicates why the SCSI command completion function (pkt_comp) was called. The most often seen reason in case of transport error are:

• CMD_INCOMPLETE:  Transport stopped with abnormal state.
• CMD_TRAN_ERR:  Unspecified transport error.
• CMD_RESET:  SCSI bus reset destroyed command.
• CMD_ABORTED:  Command transport aborted on request.
• CMD_TIMEOUT:  Command timed out.
• CMD_DEV_GONE:  The device has been removed.

During the completion of the SCSI command, transport, multipathing and target driver could produce various error message reporting the error. The decision to retry the SCSI command or fail it will be taken by the target device driver with the exception of some retries done automatically by multipathing driver.
For example the sd target driver will retry each command after waiting 100ms (default sd restart timeout value) and up to the default count of three retries. After three retries if the command is still in error following message will be printed. Some command may also not be retried at all if the application specified to failfast them:

When investigating SCSI transport error it is needed to look at different source of information:

• console and /var/adm/dmesg or /var/adm/messages* files
• "iostat -iE" command
• kstat command

2. Transport error with reason 'reset'

The 'reset' reason indicates that after the SCSI packet was sent a reset occured on the bus and therefore prevented the completion of the packet.
This kind of error can happen on any type of transport protocol (scsi, sas, fc, etc...) and may be a indication of faulty component on the channel (target device, controller, cable, switch, etc...)

a) parallel SCSI bus example

 In this case the tran_err error is due to a reset of the target device initiated by the controller in order to clear it's fault. At the end disk sd0 was replaced.

3. Transport error with reason 'tran_err'

This reason is by definition more vague and not necessarily implies and hardware issue. The error tells that the i/o chipset on the hba was not able to send the SCSI command to the target device or that the target or any other equipement on the channel returned it with this pkt_reason value.

a) parallel SCSI bus transport errors

This example shows transport error on parallel SCSI bus caused by parity error:

In this the controller got transfer issue with the target 0 and decided to reduce the transfer rate and recover the fault using a target reset which generated the tran_err error.

Usually SCSI bus DATA phase parity errors are caused by bad cable and/or termination. It is also recommended to check for patches of hba driver and disk firmware.

b) Fiber channel transport errors

In this example the mid layer multipathing driver VxDMP also reported the error and took action (disabling a path), One can see the error was not transient and forced the target driver to fail the command (giving up):

The disk ssd30 was later replaced.

In recent version of MPxIO, the driver will print the path on which the transport error occured:

Transport error can be caused by a target device port going offline. Notice how MPxIO triggered a path failover. As no more error were reported by the kernel the retry was successful:

At this point one need to investigate why the target device port gone offline.

Typical complex to troubleshoot issue are when you have a mix of transport error which occur intermittently and does not trigger command to fail:

c) Conclusion

Transport error with reason 'tran_err' are not always easy to investigate. Troubleshooting these errors request to carefully check the various error counter on hba, switch port, target device and compare with different systems which can access the same target device. As these error can also be caused by bug in multipathing or hba driver, it is strongly recommended to check patch level.

4. Transport timeout

Transport timeout error occurs when the SCSI command has been processed by the hba driver and the SCSI packet completion did not happened after the default target device timeout (for the sd/ssd driver the default packet timeout value is 60 sec). This can be caused by:

• non responding target device or FC switches.
• hba firmware bug.
  
• hba / transport drive bug.

These are theorically possible but unlikely to be seen:

• systemic issue (HW interrupt not processed). Imply that you have a specific issue with HW interrupt _and_ the cyclic/callout monitoring hba state and queue is able to run
  
• drift in system clock (hba driver look frequently at outstanding request and check if the (current time - stored packet start time) is not superior to the packet timeout, when the clock drift or is adjusted it changes the value of the current time)

a) Examples

The following example t shows intermittent issue on a private loop affecting all target device. This can be cause by faulty channel or spurious error generated by one of the target. Looking at counter statistic of the hba using "luxadm -e rdls /dev/cfg/cN" may indicates where the issue is:

The above errors we generated by faulty controller which was replaced.

This example is typical of a failing internal SCSI hard disk drive which will need to be replaced. This can be confirmed by the number of error reported for that device in "iostat -iE" output and the fact the the error is isolated to that specific device :

The hard disk driver on target 1 was replaced.

In this example you can see how a faulty hard disk drive ssd11 can impact other due to bus reset initiated to recover from a failing target device on a private FC loop:

The disk ssd11 was replaced.

b) Conclusion

Timeout transport on a parallel SCSI bus, SAS bus or FC private loop (directly connected no FC Switch) are more likely generated by misbehaving component.

On FC switched fabric topology, "Transport timeout" may indicate an issue on fc switches (loss FC frame) or Storage array (lack of performance to process all request i/o, reboot of controller, etc...). Especially if when "Command Timeout on path ..." type errors are not accompanied with scsi transport errors.

In any case, on the server side, one should have a close look at all the error counter and check if patch or firmware level are not at a too low revision. When FC Switches and external storage arrays are involved, they should also be checked for issues as well.

5. Summary of action to take

1. review carefully console and messages files. Look for any occurence.
2. check error counters to determine if the error affects particular device:
   • use "iostat -iE" to check disk errors.
   • use "luxadm -e rdls /dev/cfg/cN" or Solaris 10 or above "fcinfo hba-port -l" command to check Fiber Channel port statistics.
3. if no persistent hardware error can be found, check for driver and firmware level of all component.

6. References:

man page:

• buf(9S)
• strategy(9E)
• scsi(4)
• scsi_pkt(9S)
• scsi_transport(9F)

List of scsi key code: http://www.t10.org/lists/asc-num.htm

References

This solution has no attachment