Using the onboard SFP28 ports on an ODA X7-2 server node

Asset ID:	1-71-2373070.1
Update Date:	2018-05-18
Keywords:

Solution Type Technical Instruction Sure

Solution 2373070.1 : Using the onboard SFP28 ports on an ODA X7-2 server node

Applies to:

Oracle Database Appliance X7-2 HA Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X7-2 S/M Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Goal

This doc describes the procedure to manually update the onboard SFP28 ports, in order to use them with the appropriate transceivers.

Solution

Currently, you can only use one pair of ports, but not all 4.

If you want to use the ports circled in orange, there are some configuration changes that need to be done. This doc will guide you through the steps on these configuration changes.

Make sure the firmware version is at 20.06.04.07 or above. For example

# ethtool -i em2

driver: bnxt_en

version: 1.7.0

firmware-version: 20.6.156/1.8.1 pkg 20.06.04.07 <<<<<<<<< check for this firmware version

bus-info: 0000:18:00.0

supports-statistics: yes

supports-test: no

supports-eeprom-access: yes

supports-register-dump: no

supports-priv-flags: no

# ethtool -i em3

driver: bnxt_en

version: 1.7.0

firmware-version: 20.6.156/1.8.1 pkg 20.06.04.07

bus-info: 0000:18:00.1

supports-statistics: yes

supports-test: no

supports-eeprom-access: yes

supports-register-dump: no

supports-priv-flags: no

If the ports are running at a lower version than 20.06.04.07, you will need to manually update them.

1. Download this file to each node, unpack the package and install the bnxtnvm tool

[root@oak ~]# unzip FW20.08SFP.zip
Archive: FW20.08SFP.zip
inflating: 20.08.01.10-57417-LOM.pkg
inflating: bnxtnvm-1.40.10.el7.rpm
inflating: metadata.xml

[root@oak ~]# rpm -ivh bnxtnvm-1.40.10.el7.rpm
Preparing... ########################################### [100%]
1:bnxtnvm ########################################### [100%]

2. Update the firmware. Update only the first interface - em2 for BM and eth2 for the Virtualized platform.

# bnxtnvm install 20.08.01.10-57417-LOM.pkg -dev=em2
Device: em2
Package 20.08.01.10-57417-LOM.pkg (11 items) verified, installing... bnxtnvmSecureUpdate returned 0

20.08.01.10-57417-LOM.pkg: Successfully installed package.
A system reboot is needed for firmware updates to take effect.

# grep BROADCOM /var/log/messages <<<<<<< verify the update
May 18 09:07:35 oak2 BROADCOM[79508]: Device - em2 : Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller :Successfully installed firmware package 20.08.01.10 : Date and Time of installation - 2018-05-18 15:07:07Z #012

# init 6 <<<<<< reboot the node
Verify the em2 and em3 have updated firmware. The firmware-version should be "20.08.01.10". Check the link speed and status.
With some switch it might say "Link detected: no" until ip-addresses are configured.

# ethtool -i em2
driver: bnxt_en
version: 1.7.0
firmware-version: 20.6.156/1.8.1 pkg 20.08.01.10
bus-info: 0000:18:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no

# ethtool -i em3
driver: bnxt_en
version: 1.7.0
firmware-version: 20.6.156/1.8.1 pkg 20.08.01.10
bus-info: 0000:18:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no

3. If you have not run firstnet, now is the time to do so. Depending on which platform you are running, you will need to execute one of the following to configure the network interfaces for the SFP28 ports

# /opt/oracle/dcs/bin/odacli configure-firstnet <<<<<<<< for dcs (odacli) stack

# oakcli configure firstnet <<<<<<<<<<<< for oakcli stack

Execute the following ethtool commands to manually force the speed to 10000 and turn off autoneg.

# ethtool -s em2 speed 10000 autoneg off
# ethtool -s em3 speed 10000 autoneg off

You will only have to execute the above two commands if your switch speed is set to 10Gb. The ports come from the factory set to 25Gb and the optical transceivers do not auto-negotiate back to 10Gb when a 10Gb optical transceiver is used.

To make the network settings persistent across reboots, you can hard code this in the system bios. In order to enter system bios, reset the system, and press F2 during startup.

Select Advanced -> Oracle Dual Port 10Gbase-T Ethernet Controller -> Device Configuration Menu -> Operational Link Speed -> 10Gb (from AutoNeg). This change needs to be made for each of the two ports.

Alternatively, you can add the following options to each interface /etc/sysconfig/network-scripts/ifcfg-em2 & ifcfg-em3 files:

ETHTOOL_OPTS="speed 10000 duplex full autoneg off"

Attachments

This solution has no attachment