Sun Storage 7000 Unified Storage System: Fibre Channel Traffic Drops to Zero

Asset ID:	1-72-1603211.1
Update Date:	2018-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 1603211.1 : Sun Storage 7000 Unified Storage System: Fibre Channel Traffic Drops to Zero

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Sun ZFS Storage 7320 - Version All Versions and later
Sun ZFS Storage 7120 - Version All Versions and later
Sun Storage 7110 Unified Storage System - Version All Versions and later
Sun Storage 7210 Unified Storage System - Version All Versions and later
7000 Appliance OS (Fishworks)

Symptoms

A brief outage for Fibre Channel storage connections to ZFS Storage Appliance (Series 7000 NAS).

No ASR was created and no Fibre Channel LINK DOWN alert reported on the ZFS Storage Appliance.

Connected hosts went down or experienced various Fibre Channel SCSI errors.

Cause

A Fibre Channel port got reset when the system is under heavy load.

System alert show a Fibre Channel Target LINK UP message and analytics showed no Fibre Channel traffic.

debug.sys showed LINK UP messages and stmf buffer (in this SR, stmf buffer has rotated)

Sep 9 10:16:01 zfs-sa-head fct: [ID 132490 kern.notice] NOTICE: qlt0,0 LINK UP, portid 300ab, topology Fabric Pt-to-Pt,speed 8G
Sep 9 10:16:01 zfs-sa-head fct: [ID 132490 kern.notice] NOTICE: qlt1,0 LINK UP, portid 101ad, topology Fabric Pt-to-Pt,speed 8G

Solaris client reporting:

Sep 9 22:14:59 solaris-client scsi: [ID 107833 kern.warning] WARNING: /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000024ff392dc2,1 (ssd207):
Sep 9 22:14:59 solaris-client SCSI transport failed: reason 'timeout': retrying command
Sep 9 22:15:00 solaris-client fctl: [ID 517869 kern.warning] WARNING: fp(5) ::N_x Port with D_ID=101ad, PWWN=21000024ff392dc3 disappeared from fabric
Sep 9 22:15:18 solaris-client fctl: [ID 517869 kern.warning] WARNING: fp(2) ::GPN_ID for D_ID=300ab failed
Sep 9 22:15:18 solaris-client fctl: [ID 517869 kern.warning] WARNING: fp(2) ::N_x Port with D_ID=300ab, PWWN=21000024ff392dc2 disappeared from fabric

Sep 9 22:14:58 solaris-client scsi: [ID 107833 kern.warning] WARNING: /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000024ff392dc2,5 (ssd100):
Sep 9 22:14:58 solaris-client SCSI transport failed: reason 'timeout': retrying command
Sep 9 22:15:00 solaris-client fctl: [ID 517869 kern.warning] WARNING: fp(2) ::N_x Port with D_ID=101ad, PWWN=21000024ff392dc3 disappeared from fabric
Sep 9 22:15:18 solaris-client fctl: [ID 517869 kern.warning] WARNING: fp(4) ::N_x Port with D_ID=300ab, PWWN=21000024ff392dc2 disappeared from fabric

STMF buffer (in this SR, stmf buffer has rotated) expected to see lport abort timed out

as an example:

:20190282: abort_task_offline called for LPORT: lport abort timed out
:20190282: Calling stmf_ctl to offline LPORT : lport abort timed out
qlt1,0:20190282: port state change from 4 to 11
qlt1,0:20190382: port state change from 11 to 0

This issue related to Bug 15604251 - SUNBT6902152-SOLARIS_11U1 STMF worker thread scaling could be improved.

The fix is available in 2013 and 2011.1.8 firmware code or above.

Solution

The workaround to increase stmf min/max workers is no longer required.

Upgrade firmware code to 2013 or 2011.1.8 depending on your platform.

References

<BUG:15823251> - SUNBT7205097 7420C: FC TRAFFIC DROPPED TO ZERO.
<NOTE:1434184.1> - Sun Storage 7000 Unified Storage System: How to Troubleshoot Fibre-Channel Problems
<NOTE:1416406.1> - Sun ZFS Storage Appliances Troubleshooting Resource Center

Attachments

This solution has no attachment