Adding or replacing an Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7 or M8

Asset ID:	1-79-2382998.1
Update Date:	2018-05-23
Keywords:

Solution Type Predictive Self-Healing Sure

Solution 2382998.1 : Adding or replacing an Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7 or M8

Applies to:

Oracle SuperCluster M8 Hardware
Oracle SuperCluster M7 Hardware
Oracle Solaris on SPARC (64-bit)

Purpose

This document describes the procedure that should be used when adding or replacing an Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7 or M8.

Scope

This procedure only applies to an Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7 or M8. Refer to product specific documentation for non-SuperCluster systems.

Note that SuperCluster M7 does not ship with Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter as standard and an engineering hardware exception must be obtained before retro-fitting Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7.

Details

There are two main requirements of the Oracle Quad 10 Gb/Dual 40 Gb Ethernet Adapter on SuperCluster M7/M8:

Fcode 3.9 (or higher)
Mode 2x2x10B

Replacement adapters are set by default with mode 4x10 and may have a lower than required fcode version. Therefore these two attributes must be changed as part of adapter replacement on SuperCluster M8.

This procedure requires the PDom that owns the adapter to be replaced is powered down during replacement. Therefore all domains within that PDom will be unavailable for the duration. The domain that owns the adapter to be replaced requires multiple reboots. You should plan for approximately 1.5 hours of total downtime.

1. Preparation

1.1 Identify the domain that owns the adapter to be replaced (it will be a dedicated or root domain) e.g.

# dladm show-phys -L | grep i40e
net0 i40e0 /SYS/CMIOU0/IOH/IOS0/RP0
net1 i40e1 /SYS/CMIOU0/IOH/IOS0/RP0
net2 i40e2 /SYS/CMIOU0/IOH/IOS0/RP0
net3 i40e3 /SYS/CMIOU0/IOH/IOS0/RP0

# dladm show-link | egrep "^net0 |^net1 |^net2 |^net3 "
net3 phys 1500 down --
net2 phys 1500 up --
net1 phys 1500 down --
net0 phys 1500 up --

1.2 Ensure the domain that owns the adapter to be replaced has the /usr/sbin/fwupdate command e.g.

# file /usr/sbin/fwupdate
/usr/sbin/fwupdate: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, UltraSPARC1 Extensions Required, dynamically linked, not stripped

Install the /usr/sbin/fwupdate command if it is not present e.g.

# pkg install system/management/fwupdate
Packages to install: 1
Create boot environment: No
Create backup boot environment: No

DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 1/1 10/10 2.3/2.3 0B/s

PHASE ITEMS
Installing new actions 45/45
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Updating package cache 2/2

1.3 Download the Intel QSFP+ Configuration Utility (QCU Tool) into the domain that owns the adapter to be replaced, from this location:

http://www.oracle.com/technetwork/server-storage/ethernet-adapters-2970561.html

Extract the downloaded file e.g.

# tar xvf qcu.tar
x QCU, 0 bytes, 0 tape blocks
x QCU/EFI2_x64, 0 bytes, 0 tape blocks
x QCU/EFI2_x64/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/EFI2_x64/qcu64e.efi, 2996352 bytes, 5853 tape blocks
x QCU/EFIx64, 0 bytes, 0 tape blocks
x QCU/EFIx64/qcu64e.efi, 3149312 bytes, 6151 tape blocks
x QCU/EFIx64/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/Linux32, 0 bytes, 0 tape blocks
x QCU/Linux32/qcu32, 6120203 bytes, 11954 tape blocks
x QCU/Linux32/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/Solaris_64s, 0 bytes, 0 tape blocks
x QCU/Solaris_64s/Intel_Proprietary_License.pdf, 170715 bytes, 334 tape blocks
x QCU/Solaris_64s/qcu64s, 2983984 bytes, 5829 tape blocks
x QCU/Linux_x64, 0 bytes, 0 tape blocks
x QCU/Linux_x64/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/Linux_x64/qcu64e, 6919455 bytes, 13515 tape blocks
x QCU/Winx64, 0 bytes, 0 tape blocks
x QCU/Winx64/iqvw64e.sys, 38376 bytes, 75 tape blocks
x QCU/Winx64/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/Winx64/install.bat, 2915 bytes, 6 tape blocks
x QCU/Winx64/iqvw64e.inf, 2127 bytes, 5 tape blocks
x QCU/Winx64/QCUW64e.exe, 3023360 bytes, 5905 tape blocks
x QCU/qcu_user_guide.txt, 5953 bytes, 12 tape blocks
x QCU/Win32, 0 bytes, 0 tape blocks
x QCU/Win32/iqvw32.inf, 2262 bytes, 5 tape blocks
x QCU/Win32/install.bat, 3144 bytes, 7 tape blocks
x QCU/Win32/iqvw32.sys, 35304 bytes, 69 tape blocks
x QCU/Win32/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/Win32/QCUW32.exe, 2437120 bytes, 4760 tape blocks
x QCU/ESXi32, 0 bytes, 0 tape blocks
x QCU/ESXi32/qcu32, 6111694 bytes, 11937 tape blocks
x QCU/ESXi32/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/ESXi_x64, 0 bytes, 0 tape blocks
x QCU/ESXi_x64/Intel_Diagnostic_Tools_License.pdf, 23556 bytes, 47 tape blocks
x QCU/ESXi_x64/qcu64e, 6912099 bytes, 13501 tape blocks
x QCU/README, 160 bytes, 1 tape blocks
x QCU/Solaris_x64, 0 bytes, 0 tape blocks
x QCU/Solaris_x64/Intel_Proprietary_License.pdf, 170715 bytes, 334 tape blocks
x QCU/Solaris_x64/qcu64e, 3015104 bytes, 5889 tape blocks

1.3 Download adapter Fcode version 3.9 (80003643) available in the following MOS patch:

Patch 27425401: ORACLE QUAD 10GB OR DUAL 40GB ETHERNET ADAPTER FIRMWARE 80003643

2. Adapter Replacement

When ready to replace the adapter, shutdown all domains in the affected PDom in the order of - all IO domains, root domains and finally control (primary) domain.

Power off the affected PDom from the Service Processor e.g.

-> stop /Servers/PDomains/PDomain_x/HOST

Follow the standard procedures to install a new or replacement adapter in an M7 or M8.

NOTE THAT NO CABLES SHOULD BE CONNECTED TO THE NEW ADAPTER AT THIS STAGE.

Restart the affected PDom from the Service Processor e.g.

-> start /Servers/PDomains/PDomain_x/HOST

3. Adapter Configuration

3.1 When booted, login to the domain that owns the adapter to be replaced and check the fcode version of the new adapter e.g.

# /usr/sbin/fwupdate list controller

==================================================
CONTROLLER
==================================================
ID    Type Manufacturer   Model Product Name              FW Version BIOS Version EFI Version    FCODE Version Package Version NVDATA Version    XML Support
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c0    IB Mellanox       0x1003    Sun InfiniBand Dual Port 2.35.5532      -          -              -              -                -                 N/A
c1    NET    Intel          0x1583    Oracle 10 Gb/40 Gb Ethern -              -              1.9.29 3.9.0          80003643         -                 N/A
c2    NET    Intel          0x1521    Sun Quad Port GbE PCIe 2. -              -              4.7.12                                   -                 N/A

If the current Fcode version does not match the minimum requirement of 3.9.0, it must be upgraded to at least version 3.9 (80003643) available in the patch:

Patch 27425401: ORACLE QUAD 10GB OR DUAL 40GB ETHERNET ADAPTER FIRMWARE 80003643

Instructions on how to upgrade the Fcode are included in the README file with the patch.

During the upgrade process the fwupdate command will ask if you wish to automatically initiate system power cycle now. Answer NO to this question.

During the upgrade process the Fault Manager will report an error related to the updated adapter e.g.

# UNW-MSG-ID: NIC-8000-1C, TYPE: Fault, VER: 1, SEVERITY: Critical
EVENT-TIME: Tue Jan 16 17:17:22 CET 2018
PLATFORM: SPARC M8-8, CSN: AK00411819, HOSTNAME: scbt-rootadm0101
SOURCE: eft, REV: 1.16
EVENT-ID: d7d5b9a1-5109-431e-86d9-af0d32a1102c
DESC: The network device failed to configure a feature. A(n) invalid_state error has been detected during driver's runtime context causing a(n) config service impact.

Firmware: FW 5.1.42098 API 1.5 eetrack 0x800029a3 NVM 0x5005
AUTO-RESPONSE: One or more device instances may be disabled.
IMPACT: Loss of services provided by the device instances associated with this fault.
REC-ACTION: Use 'fmadm faulty' to provide a more detailed view of this event. Please refer to the associated reference document at http://support.oracle.com/msg/NIC-8000-1C for the latest service procedures and policies regarding this diagnosis.

# fmadm faulty -s
--------------- ------------------------------------ -------------- ---------
TIME EVENT-ID MSG-ID SEVERITY
--------------- ------------------------------------ -------------- ---------
Mar 01 14:58:24 730638e6-ed44-4836-b2f1-d5cd87a010b9 NIC-8000-0Q Critical

This error can safely be ignored and should be acquitted e.g.

# fmadm acquit 730638e6-ed44-4836-b2f1-d5cd87a010b9
fmadm: recorded acquittal of 730638e6-ed44-4836-b2f1-d5cd87a010b9

3.2 Using the QCU Tool, list the current mode of the Oracle Quad 10Gb Ethernet Adapter. If the mode is not set to 2x2x10B it must be changed. The example below shows changing the adapter from the default of 4x10 to 2x2x10B mode.

# cd QCU/Solaris_64s/
# chmod 755 ./qcu64s
# ./qcu64s /DEVICES
Intel(R) QSFP+ Configuration Utility

QCU version: v2.27.10.01
Copyright(C) 2016 by Intel Corporation.
Software released under Intel Proprietary License.

NIC Seg:Bus Ven-Dev Mode Adapter Name
=== ======= ========= ======= ==================================================
1) 000:001 8086-1583 4x10 Oracle 10 Gb/40 Gb Ethernet Adapter

# ./qcu64s /NIC=1 /SET=2x2x10B
Intel(R) QSFP+ Configuration Utility

QCU version: v2.27.10.01
Copyright(C) 2016 by Intel Corporation.
Software released under Intel Proprietary License.

Changing the configuration...
Done.

4. Connect cables and reboot

Now connect the cables to the new adapter.

The domain must be rebooted for the adapter changes to take effect. If the adapter is installed in a root domain, all dependent IO domains must be closed before the domain is rebooted. The dependent domains can be identified as below

# ldm list-dependencies -r ssccn1-dom1
DOMAIN            DEPENDENT TYPE
ssccn1-dom1       ssccn1-io-sc09iod8 IOV
                  ssccn1-io-sc09iod9 IOV

5. Verification

Once the domain has booted, the network port status should be checked to confirm the correct network ports are online. By default i40e0 and i40e2 should be in use e.g.

# dladm show-phys -L | grep i40e
net0 i40e0 /SYS/CMIOU0/IOH/IOS0/RP0
net1 i40e1 /SYS/CMIOU0/IOH/IOS0/RP0
net2 i40e2 /SYS/CMIOU0/IOH/IOS0/RP0
net3 i40e3 /SYS/CMIOU0/IOH/IOS0/RP0

# dladm show-link | egrep "^net0 |^net1 |^net2 |^net3 "
net3 phys 1500 down --
net2 phys 1500 up --
net1 phys 1500 down --
net0 phys 1500 up --

If the adapter is a replacement due to a faulty part, the original fault must be cleared using the 'fmadm replaced <uuid>' command.

Attachments

This solution has no attachment