ESXi 5.0 VMotion Configuration Best Practices for Oracle Virtual Networking (OVN -- Formerly Xsigo) Fabric Interconnnect Chassis

Asset ID:	1-71-1519043.1
Update Date:	2017-08-01
Keywords:

Solution Type Technical Instruction Sure

Solution 1519043.1 : ESXi 5.0 VMotion Configuration Best Practices for Oracle Virtual Networking (OVN -- Formerly Xsigo) Fabric Interconnnect Chassis

Applies to:

Oracle Fabric Interconnect F1-15 - Version All Versions and later
Oracle Fabric Interconnect F1-4 - Version All Versions and later
Information in this document applies to any platform.

Goal

What is the Best Practice for configuring VMotion on ESXi 5.0 using Oracle Fabric Interconnect Chassis.

Solution

With Jumbo Frames enabled on VMotion vmkernel vmknic and at the vSwitch, as well as the Oracle Fabric Interconnect Ethernet-port and vnic level, OVN Support uncovered an issue with high VMotion vmkping latency and failure of 8 concurrent VMotions. Single VMotions continued to work, but running multiple parallel (concurrent) VMotions failed in conjunction with observing high VMotion vmkping latency. OVN Support discovered this is due to IB stack TX (transmit side) completion having a longer latency. Oracle Fabric Interconnect uses polling mode for TX side completions and this combined with vNIC ring size is causing this high latency on TX side.

To enable Jumbo Frames (MTU 9000)to work without issue, OVN Support for ESXi 5.0 VMotions requires the workaround below to be performed from the ESXi 5.0 CLI to get 8 concurrent VMotions to work and to reduce VMotion vmkping latency:

# esxcfg-module -s "xsvnic_txring_size=128" xsvnic
# reboot

Next, login to the IOD (IO Director) as user 'admin' and configure the ethernet-port the vnics will be terminated to MTU 9000

# set ethernet-port <slot#/port#> -mtu=9000

[NOTE] You cannot change / set MTU setting on an ethernet-port with vnics terminated to it, so you must have no vnics on the ethernet-port when you set the MTU value.

Add vnics to the server either from Oracle Fabric Interconnect Chassis CLI or Oracle Fabric Manager. The vnics will inherit the MTU setting from the ethernet-port. Default MTU setting is 1500.

On ESXi 5.0 you set the MTU for the vSwitch to MTU 9000 and then when creating the VMotion vmkernel portgroup on the vSwitch you set MTU 9000 for the VMotion vmkernel portgroup.

You can set the MTU setting in vSphere 4.1 and 5.0 by going to:

Networking - vSwitch where VMotion PG resides - Properties - Edit for both the vSwitch and VMotion Portgroup - Advanced Properties - MTU 9000.

You can also set MTU settings for vSwitch, dvSwitch and vmknic from ESXi 5.0 CLI. using esxcfg-vswitch, esxcfg-vmknic or esxcli commands. Please refer to complete VMware ESXi 5.0 documentation on VMware's Support website for complete command syntax.

Here is what it should look like from ESXi 5.0 CLI:

# esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 4 128 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 1 vmnic1
Management Network 0 1 vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 4 128 9000 vmotion

PortGroup Name VLAN ID Used Ports Uplinks
VMotion 0 1 vmotion

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 16 128 9000 vnic1

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 2 309 14 vnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 128 3 128 9000 vnic3

PortGroup Name VLAN ID Used Ports Uplinks
NFS 309 1 vnic3

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 172.16.10.209 255.255.255.0 172.16.10.255 00:15:17:62:e1:64 1500 65535 true DHCP
vmk2 NFS IPv4 172.16.10.127 255.255.255.0 172.16.10.255 00:50:56:7c:60:0c 9000 65535 true STATIC
vmk1 VMotion IPv4 10.0.0.75 255.255.255.0 10.0.0.255 00:50:56:73:1d:4f 9000 65535 true STATIC

You can also use these commands as well to set and view the MTU settings:

~ # esxcli network vswitch standard list

vSwitch0

Name: vSwitch0
Class: etherswitch
Num Ports: 128
Used Ports: 3
Configured Ports: 128
MTU: 1500
CDP Status: listen
Beacon Enabled: false
Beacon Interval: 1
Beacon Threshold: 3
Beacon Required By:
Uplinks: vmnic1
Portgroups: VM Network, Management Network

vSwitch1
Name: vSwitch1
Class: etherswitch
Num Ports: 128
Used Ports: 3
Configured Ports: 128
MTU: 9000
CDP Status: listen
Beacon Enabled: false
Beacon Interval: 1
Beacon Threshold: 3
Beacon Required By:
Uplinks: vmotion
Portgroups: VMotion

~ # esxcli network ip interface list
vmk0
Name: vmk0
MAC Address: 00:15:17:62:e1:64
Enabled: true
Portset: vSwitch0
Portgroup: Management Network
VDS Name: N/A
VDS Port: N/A
VDS Connection: -1
MTU: 1500
TSO MSS: 65535
Port ID: 16777219

vmk2
Name: vmk2
MAC Address: 00:50:56:7c:60:0c
Enabled: true
Portset: vSwitch3
Portgroup: NFS
VDS Name: N/A
VDS Port: N/A
VDS Connection: -1
MTU: 9000
TSO MSS: 65535
Port ID: 67108867

vmk1
Name: vmk1
MAC Address: 00:50:56:73:1d:4f
Enabled: true
Portset: vSwitch1
Portgroup: VMotion
VDS Name: N/A
VDS Port: N/A
VDS Connection: -1
MTU: 9000
TSO MSS: 65535
Port ID: 33554

Attachments

This solution has no attachment