Oracle ZFS Storage Appliance: Fibre Channel HBA Ports Offline

Asset ID:	1-72-2127910.1
Update Date:	2018-01-05
Keywords:

Solution Type Problem Resolution Sure

Solution 2127910.1 : Oracle ZFS Storage Appliance: Fibre Channel HBA Ports Offline - "set FORCED_OFFLINE now" is reported

Applies to:

Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS4-4 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

ZFS appliance Fibre Channel ports show up as offline.

Windows client is unable to connect to the ZFS appliance over Fibre Channel (FC).

Rebooting the ZFS appliance or Window client does not resolve the problem.

On reboot of the appliance, the console will show the links coming up for all instances:

fct: [ID 469330 kern.notice] NOTICE: qlt2,0 LINK UP, portid 10400, topology Fabric Pt-to-Pt, speed 16G

and then followed by the link going down for the same instances:

fct: [ID 580862 kern.notice] NOTICE: qlt2,0 LINK DOWN, portid 10400, topology Fabric Pt-to-Pt,speed 16G

From the 'debug.sys' log file:

Apr 8 10:31:04 Hostname qlt: [ID 325460 kern.warning] WARNING: qlt(2): <<-- QLT instance #2
cmd received with extended cdb (cdb size = 208 bytes), however the firmware did not DMAed the entire FCP_CMD IU, entry count is 1 while it should be 4
Apr 8 10:31:11 Hostname qlt: [ID 467599 kern.notice] NOTICE: qlt(2): Skipping firmware dump as there is one already outstanding.
Apr 8 10:31:11 Hostname fct: [ID 580862 kern.notice] NOTICE: qlt2,0 LINK DOWN, portid 10400, topology Fabric Pt-to-Pt,speed 16G
Apr 8 10:31:13 Hostname fct: [ID 469330 kern.notice] NOTICE: qlt2,0 LINK UP, portid 10400, topology Fabric Pt-to-Pt, speed 16G

This is repeated many times then the port is brought offline permanently:

Apr 8 11:21:48 Hostname qlt: [ID 325460 kern.warning] WARNING: qlt(2): cmd received with extended cdb (cdb size = 208 bytes), however the firmware did not DMAed the entire FCP_CMD IU, entry count is 1 while it should be 4
Apr 8 11:21:52 Hostname last message repeated 3 times
Apr 8 11:21:59 Hostname qlt: [ID 467599 kern.notice] NOTICE: qlt(2): Skipping firmware dump as there is one already outstanding.
Apr 8 11:21:59 Hostname fct: [ID 580862 kern.notice] NOTICE: qlt2,0 LINK DOWN, portid 10400, topology Fabric Pt-to-Pt,speed 16G
Apr 8 11:22:01 Hostname stmf: [ID 123654 kern.warning] WARNING: stmf_ctl: too frequent request to online the port, set FORCED_OFFLINE now

NOTE: Apr 8 11:22:01 Hostname stmf: [ID 123654 kern.warning] WARNING: stmf_ctl: too frequent request to online the port, set FORCED_OFFLINE now

In this case, we have a Windows 2012 R2 client host with QLE8362 with unsupported firmware 06.06.03, driver version 9.1.17.21 with BIOS 3.19.

# fcinfo hba-port

HBA Port WWN: 2100000e1e289cd0
Port Mode: Target
Port ID: 10000
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: QLE8362
Firmware Version: 6.6.3
FCode/BIOS Version: BIOS: 3.19:PCI3.0; FCode: 4.02; EFI: 5.36;
Serial Number: 463916R+1528265503
Driver Name: COMSTAR QLT
Driver Version: 20141120-2.04
Type: F-port
State: offline <<-- Port state is offline
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 2000000e1e289cd0

Changes

New Microsoft Windows client have just been installed.

Cause

Incorrect and unsupported QLogic driver installed on Microsoft Windows client. Windows Qlogic FC driver sends an incorrect SCSI command to the appliance.

Capture additional QLogic firmware dump and qlt driver trace for instance #2 using mdb dcmd.

>*qlt_state::softstate 2 <==this dcmd will return qlt instance #2 address
xxxxxxxxxxxxxxxx

>::log qlt2-fw-dump.log
>xxxxxxxxxxxxxxxx::qltgetdump

>::log qlt2_drv_trace.log
>xxxxxxxxxxxxxxxx::qltgettrace

>::quit

Solution

Install Oracle supported QLogic Windows driver for particular HBA model.

For HBA model QLE8362, the currently supported firmware Driver version 9.1.11.28 with MBI version 3.60.17

To confirm the currently supported Qlogic driver required, please check the following Qlogic website

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProductOracle.aspx?oemid=124&productid=1146&OSTYPE=Windows&category=11&Product=7101674

The CDB ( command descriptor block) sent by the client is incorrect and when the qlt driver on the appliance receives it, it reports an error as it is unable to DMA the entire fcp command.

The IOCB ( i/o command block) from firmware does not make sense to qlt driver.

And when it experiences this repeatedly, the qlt driver on the appliance brings down the link permanently as designed.

There is no provision to bring up the port online again manually through some commands. The only way it can be brought back online is to shutdown the appliance nodes and also shutdown the windows client.

When the appliance is now booted, it will come up with its FC ports online.

As a workaround:
a) Disconnect the Windows clients physical FC port connection or
b) Remove the initiators from the appliance SAN configuration. This also requires the LUN access settings to be modified.

References

<BUG:23076573> - FC HBA PORTS OFFLINE: CMD RECEIVED WITH EXTENDED CDB

Attachments

This solution has no attachment