QConvergeConsole CLI qaucli Slow Or Fails To Start

Asset ID:	1-72-2276802.1
Update Date:	2017-06-15
Keywords:

Solution Type Problem Resolution Sure

Solution 2276802.1 : QConvergeConsole CLI qaucli Slow Or Fails To Start

Applies to:

Solaris Operating System - Version 10 3/05 and later
Solaris x64/x86 Operating System - Version 10 3/05 and later
SPARC T7-2 - Version All Versions and later
SPARC T5-2 - Version All Versions and later
SPARC T4-2 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

QConvergeConsole CLI qaucli is slow or fails to start. This utility is required to update firmware on Oracle branded QLogic HBAs and is typically installed into one of the following locations:

/usr/lib/ssm/fwupdate/qlogic/qaucli
/opt/QLogic_Corporation/QConvergeConsoleCLI/qaucli

Changes

Large configurations are more likely to run into this issue. For example, on a system where there are 2500 LUNs presented on 4 paths, 2 behind each HBA, equates to 10,000 discreet LUN paths. It takes at least three IOCTLs to each target and LUN to discover all the information, resulting in over 30,000 IOCTLs. The update_sol.sh script included with multiboot updates starts qaucli 15 times. It could take an entire day to complete the update.

Cause

Truss output shows qaucli is using SCSI PASSTHRU (0x0069) IOCTLs to gather information about each target and LUN behind each HBA when it is started, regardless of whether a specific HBA WWN is specified on the command line.

Base time stamp: 1495649712.366281 [ Wed May 24 15:15:12 BRT 2017 ]
16149/1: 0.000000 0.000000 0.000000 execve("/usr/lib/ssm/fwupdate/qlogic/qaucli", 0xFFBFF944, 0xFFBFF958) argc = 4
16149/1: argv: /usr/lib/ssm/fwupdate/qlogic/qaucli -pr fc -z
...
16149/1: 0.697924 0.000092 0.000052 open("/devices/pci@400/pci@1/pci@0/pci@c/SUNW,qlc@0:devctl", O_RDWR) = 4
16149/1: 0.698021 0.000097 0.000034 lwp_sema_post(0xFF2C0000) = 0
16149/1: 0.698177 0.000156 0.000050 ioctl(4, 0x0064, 0xFFBFEE10) = 0
16149/1: 0.698267 0.000090 0.000036 ioctl(4, 0x0064, 0xFFBFE900) = 0
16149/1: 0.698348 0.000081 0.000035 ioctl(4, 0x006E, 0xFFBFE900) = 0
16149/1: 0.698444 0.000096 0.000036 ioctl(4, 0x007F, 0xFFBFEE10) = 0
16149/1: 0.698525 0.000081 0.000038 ioctl(4, 0x007F, 0xFFBFEE10) = 0
16149/1: 0.698633 0.000108 0.000042 ioctl(4, 0x0064, 0xFFBFE528) = 0
16149/1: 0.698714 0.000081 0.000035 ioctl(4, 0x0064, 0xFFBFE018) = 0
16149/1: 0.698816 0.000102 0.000034 ioctl(4, 0x006E, 0xFFBFE018) = 0
16149/1: 0.698894 0.000078 0.000034 ioctl(4, 0x007F, 0xFFBFE528) = 0
16149/1: 0.698971 0.000077 0.000035 ioctl(4, 0x007F, 0xFFBFE528) = 0
...
16149/1: 0.702769 0.000109 0.000034 ioctl(4, 0x0139, 0x001B9428) Err#25 ENOTTY
16149/1: 0.702862 0.000093 0.000036 ioctl(4, 0x0064, 0xFFBFF450) = 0
16149/1: 0.708653 0.005791 0.005737 ioctl(4, 0x0080, 0xFFBFF340) = 0
16149/1: 0.708807 0.000154 0.000039 ioctl(4, 0x0064, 0xFFBFF4A0) = 0
16149/1: 0.708943 0.000136 0.000051 ioctl(4, 0x0064, 0xFFBFF1A8) = 0
16149/1: 0.709038 0.000095 0.000037 ioctl(4, 0x0064, 0xFFBFEC98) = 0
16149/1: 0.709118 0.000080 0.000034 ioctl(4, 0x006E, 0xFFBFEC98) = 0
16149/1: 0.709206 0.000088 0.000036 ioctl(4, 0x007F, 0xFFBFF1A8) = 0
16149/1: 0.709284 0.000078 0.000035 ioctl(4, 0x007F, 0xFFBFF1A8) = 0
16149/1: 0.709373 0.000089 0.000039 ioctl(4, 0x0064, 0xFFBFF450) = 0
16149/1: 0.709479 0.000106 0.000038 ioctl(4, 0x0070, 0xFFBFF498) = 0
16149/1: 0.709663 0.000184 0.000139 ioctl(4, 0x0070, 0xFFBFF498) = 0
16149/1: 0.709746 0.000083 0.000035 ioctl(4, 0x006E, 0xFFBFF4A0) = 0
16149/1: 0.709824 0.000078 0.000034 ioctl(4, 0x006E, 0xFFBFF4A0) = 0
16149/1: 0.709914 0.000090 0.000043 ioctl(4, 0x0064, 0xFFBFEE20) = 0
16149/1: 0.709997 0.000083 0.000036 ioctl(4, 0x0064, 0xFFBFE910) = 0
16149/1: 0.710077 0.000080 0.000035 ioctl(4, 0x006E, 0xFFBFE910) = 0
16149/1: 0.710156 0.000079 0.000034 ioctl(4, 0x007F, 0xFFBFEE20) = 0
16149/1: 0.710234 0.000078 0.000035 ioctl(4, 0x007F, 0xFFBFEE20) = 0
16149/1: 0.711356 0.001122 0.001061 ioctl(4, 0x0064, 0xFFBFF258) = 0
16149/1: 0.711723 0.000367 0.000301 ioctl(4, 0x0069, 0xFFBFF1D8) = 0
16149/1: 0.996619 0.284896 0.284834 ioctl(4, 0x0069, 0xFFBFF220) = 0
16149/1: 0.999318 0.002699 0.002468 ioctl(4, 0x0069, 0xFFBFEE00) = 0
16149/1: 1.296678 0.297360 0.297178 ioctl(4, 0x0069, 0xFFBFF0B0) = 0
16149/1: 1.297037 0.000359 0.000124 ioctl(4, 0x0069, 0xFFBFF068) = 0
16149/1: 1.596371 0.299334 0.299264 ioctl(4, 0x0069, 0xFFBFF0B0) = 0
16149/1: 1.596729 0.000358 0.000176 ioctl(4, 0x0069, 0xFFBFF0B0) = 0
16149/1: 1.597002 0.000273 0.000164 ioctl(4, 0x0069, 0xFFBFF028) = 0
16149/1: 1.597188 0.000186 0.000139 ioctl(4, 0x0069, 0xFFBFF028) = 0
16149/1: 1.597362 0.000174 0.000113 ioctl(4, 0x0069, 0xFFBFF068) = 0
16149/1: 1.896379 0.299017 0.298969 ioctl(4, 0x0069, 0xFFBFF0B0) = 0
16149/1: 1.896752 0.000373 0.000173 ioctl(4, 0x0069, 0xFFBFF0B0) = 0
16149/1: 1.896956 0.000204 0.000135 ioctl(4, 0x0069, 0xFFBFF028) = 0
16149/1: 1.897113 0.000157 0.000099 ioctl(4, 0x0069, 0xFFBFF068) = 0
...

hex 0x0064 = 100
#define EXT_CC_QUERY_OS 100
#define EXT_CC_QUERY EXT_CC_QUERY_OS

hex 0x0069 = 105
#define EXT_CC_SCSI_PASSTHRU_OS 105
#define EXT_CC_SEND_SCSI_PASSTHRU EXT_CC_SCSI_PASSTHRU_OS

hex 0x006E = 110
#define EXT_CC_READ_NVRAM_OS 110
#define EXT_CC_READ_NVRAM EXT_CC_READ_NVRAM_OS

hex 0x0070 = 112
#define EXT_CC_GET_DATA_OS 112
#define EXT_CC_GET_DATA EXT_CC_GET_DATA_OS

hex 0x007F = 127
#define EXT_CC_GET_FCACHE_EX_OS 127
#define EXT_CC_GET_FCACHE_EX EXT_CC_GET_FCACHE_EX_OS

hex 0x0080 = 128
#define EXT_CC_GET_PCI_DATA_OS 128
#define EXT_CC_GET_PCI_DATA EXT_CC_GET_PCI_DATA_OS

Solution

The following enhancement has been filed with QLogic to modify QConvergeConsole CLI qaucli to provide an option to bypass target and LUN discovery:

Bug 26145879 - QConvergeConsole should not require target or LUN discovery for firmware updates

Workaround:

Bring the server down to such a point as it can be disconnected from the SAN, for example with a port disable on the SAN switch for all the associated ports.
Update the firmware.
Power off the server.
Port enable on the SAN switch ports.
Power on the server.

References

<NOTE:1955822.1> - Solaris 11.2 (and later) FC HBA - Update Firmware, FCode/BIOS (ie. Boot Code)
<BUG:26145879> - QCONVERGECONSOLE SHOULD NOT REQUIRE TARGET OR LUN DISCOVERY FOR FIRMWARE UPDATES
<NOTE:1950167.1> - Unable To Create SR-IOV Virtual Instance For Qlogic 16 Gb Fibre Channel PCIe Universal Host Bus Adapter

Attachments

This solution has no attachment