Sun Storage 7000 Unified Storage System: Changing MTU of existing network interfaces causes them to go into 'maintenance' state

Asset ID:	1-72-1558636.1
Update Date:	2018-05-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1558636.1 : Sun Storage 7000 Unified Storage System: Changing MTU of existing network interfaces causes them to go into 'maintenance' state

Applies to:

Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Sun Storage 7210 Unified Storage System - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Community - Disk Storage ZFS Storage Appliance

The customer attempted to set the MTU parameter to 9000.

When we set the value to 9000 following happens :

1. Instead of green light we see orange LED on data links in the BUI

2. System complains about network settings

3. We still can use the datalink and interface BUT:

      - IPMP does not seem to work
      - No decrease in TCP packets
      - BUI/CLI hangs , transfers slow down when a load is applied to the system through those links

Alerts

Hostname: NASsrv
Product Type: SUN ZFS STORAGE 7420
Summary: %fault-list[0].svc-string is unavailable.

Service failed.

Service svc: /appliance/kit/network/datalink:ixgbe2 failed - a start, stop or ref
Full Description: Service svc:/appliance/kit/network/datalink:ixgbe2 failed - a start, stop or refresh method failed.
Event-ID: 41286bad-b107-60df-97cd-ddb41d796834
Auto-Response: None
Impact: None
Rec-Action: None
Software Component:
Name: svc:///appliance/kit/network/datalink:ixgbe2

BUI 'problems'

NASsrv:> maintenance problems show
Problems:

COMPONENT DIAGNOSED TYPE DESCRIPTION
problem-000 2013-4-12 06:16:19 Major Defect Service
svc:/appliance/kit/network/datalink:ixgbe1
failed - a start, stop or
refresh method failed.

problem-001 2013-4-12 06:16:19 Major Defect Service
svc:/appliance/kit/network/datalink:ixgbe2
failed - a start, stop or
refresh method failed.

problem-002 2013-4-12 06:16:20 Major Defect Service
svc:/appliance/kit/network/datalink:ixgbe3
failed - a start, stop or
refresh method failed.

problem-003 2013-4-12 06:44:33 Major Defect Service
svc:/appliance/kit/network/datalink:ixgbe0
failed - a start, stop or
refresh method failed.

NOTE: Marking the issue resolved on the appliance does NOT resolve the issue.

Network interface 'services' logs

$ tail appliance-kit-network-datalink:ixgbe0.log
akdatalink: ixgbe0: could not set MTU 9000
[ Apr 12 06:43:56 Method "start" exited with status 96. ]
[ Apr 12 06:44:33 Leaving maintenance because clear requested. ]
[ Apr 12 06:44:33 Enabled. ]
[ Apr 12 06:44:33 Executing start method ("exec /usr/lib/ak/svc/method/akdatalink start"). ]
Configuring device datalink ixgbe0
Setting ixgbe0 MTU to 9000
dladm: warning: cannot set link property 'mtu' on 'ixgbe0': link busy
akdatalink: ixgbe0: could not set MTU 9000
[ Apr 12 06:44:33 Method "start" exited with status 96. ]

$ tail appliance-kit-network-datalink:ixgbe1.log
akdatalink: ixgbe1: could not set MTU 9000
[ Apr 12 06:13:52 Method "start" exited with status 96. ]
[ Apr 12 06:16:19 Leaving maintenance because clear requested. ]
[ Apr 12 06:16:19 Enabled. ]
[ Apr 12 06:16:19 Executing start method ("exec /usr/lib/ak/svc/method/akdatalink start"). ]
Configuring device datalink ixgbe1
Setting ixgbe1 MTU to 9000
dladm: warning: cannot set link property 'mtu' on 'ixgbe1': link busy
akdatalink: ixgbe1: could not set MTU 9000
[ Apr 12 06:16:19 Method "start" exited with status 96. ]

$ tail appliance-kit-network-datalink:ixgbe2.log
akdatalink: ixgbe2: could not set MTU 9000
[ Apr 12 06:12:38 Method "start" exited with status 96. ]
[ Apr 12 06:16:18 Leaving maintenance because clear requested. ]
[ Apr 12 06:16:18 Enabled. ]
[ Apr 12 06:16:18 Executing start method ("exec /usr/lib/ak/svc/method/akdatalink start"). ]
Configuring device datalink ixgbe2
Setting ixgbe2 MTU to 9000
dladm: warning: cannot set link property 'mtu' on 'ixgbe2': link busy
akdatalink: ixgbe2: could not set MTU 9000
[ Apr 12 06:16:19 Method "start" exited with status 96. ]

$ tail appliance-kit-network-datalink:ixgbe3.log
akdatalink: ixgbe3: could not set MTU 9000
[ Apr 12 06:15:04 Method "start" exited with status 96. ]
[ Apr 12 06:16:19 Leaving maintenance because clear requested. ]
[ Apr 12 06:16:19 Enabled. ]
[ Apr 12 06:16:19 Executing start method ("exec /usr/lib/ak/svc/method/akdatalink start"). ]
Configuring device datalink ixgbe3
Setting ixgbe3 MTU to 9000
dladm: warning: cannot set link property 'mtu' on 'ixgbe3': link busy
akdatalink: ixgbe3: could not set MTU 9000
[ Apr 12 06:16:20 Method "start" exited with status 96. ]

Cause

This is a known issue, resolved in Appliance Firmware Release 2013.1.0.1 or later. (There is no fix, as yet, for the 2011.1.x Appliance Firmware Release)

Bug 16096421 (Changing both vnic mtu and underlying aggr mtu puts aggr to maintenance state) is fixed in the AK-8 release.

Solution

The 'workaround' for this issue is to destroy the actual interfaces and create new ones along with datalinks that have the correct jumbo frames size option from the start.

1) Destroy the affected interfaces and the datalinks (in a 'top down' manner, eg. ipmp, vlan, interface then datalink)

2) Recreate each datalink one at a time. Ensure the MTU is set to the required value (eg. MTU = 9000)

=> Make sure each datalink is FULLY created before going on the next.

3) Recreate each interface, one at a time. Ensure each interface is fully created before moving on to the next.

4) Check that the MTU is set correctly for each interface.

On 'Configuration > Network' screen, click the 'pencil' icon (Edit network datalink properties) of each underlying datalink.

Check 'ifconfig -a':

aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.10.130 netmask ffffff00 broadcast 192.168.10.255
        ether 0:21:28:3e:12:fa
        pffff_ibp0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 65520 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp1
        ipib 80:0:0:49:fe:80:0:0:0:0:0:0:0:21:28:0:1:3f:24:17
        pffff_ibp1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 65520 index 8
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp1
        ipib 80:0:0:4b:fe:80:0:0:0:0:0:0:0:21:28:0:1:3f:24:18
        igb0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        inet 10.145.229.130 netmask fffffc00 broadcast 10.145.231.255
        ether 0:21:28:3e:12:f8
        ixgbe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 index 6         <<<<<<<<
        inet 192.168.100.130 netmask ffffff00 broadcast 192.168.100.255
        ether 0:1b:21:81:4f:e4
        ixgbe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 index 10        <<<<<<<<
        inet 198.168.20.130 netmask ffffff00 broadcast 198.168.20.255
        ether 0:1b:21:81:4f:e5
        ipmp1: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 65520 index 9
        inet 192.168.2.130 netmask ffffff00 broadcast 192.168.2.255
        groupname ipmp1

Possible alternative solution for VNICs in a cluster:

Initiate a normal reboot of the Secondary node
Change the MTU value on the Secondary node's VNICs
Failback
Reboot Primary node
Change the MTU value on the Primary node's VNICs
Failback

Destroying the datalinks and re-configuring all the networking was not necessary in this case.

See also, CR 15797440 (After VM2 integration throughput with uperf on x86 using jumbo frames)

This bug effects code earlier then 2013.1.4 (fixed in 2013.1.4.0).

This also leads to hangs or extremely excessive delays when adjusting MTU.

It is recommended that customers running 2013 code upgrade to the latest code BEFORE making MTU adjustments to configured datalinks.

***Checked for relevance on 25-MAY-2018***

References

<NOTE:1395461.1> - Sun Storage 7000 Unified Storage System: Best Practice Recommendations for Network Configuration
<NOTE:1396100.1> - Sun Storage 7000 Unified Storage System: Causes and Solutions for Well Known General Networking Problems

Attachments

This solution has no attachment