Sun Storage 7000 Unified Storage System: ip.bytes[hostname] Analytics Dataset enabled can inhibit Network Performance

Asset ID:	1-72-1589345.1
Update Date:	2018-05-25
Keywords:

Solution Type Problem Resolution Sure

Solution 1589345.1 : Sun Storage 7000 Unified Storage System: ip.bytes[hostname] Analytics Dataset enabled can inhibit Network Performance

Applies to:

Sun ZFS Backup Appliance - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7120 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7320 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

There were 2 nodes of a 7420 connected to an exadata over 10 Gbe network interfaces for RMAN backups.

When running 'dd' commands from the exadata to each head, it was discovered that one head performs 2x as well as the other.

We will call the heads 'T' and 'U'. the 'T' head performed at 1/2 the performance of the 'U' head

[linux_box ~]# dd if=/dev/zero of=/export/share7/X2_DD.out bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 53.3967 seconds, 196 MB/s # 'T' head

[linux_box ~]# dd if=/dev/zero of=/export/share8/X2_DD.out bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 27.5093 seconds, 381 MB/s # 'U' head

Changes

A certain analytic was enabled as described in the cause section.

Cause

In this situation the analytics dataset ip.bytes[hostname] was enabled on the 'T' head.

The following is a summary of the cause from Engineering:

> Based on the success of the test, it looks like this issue is related to the way Dtrace handles structure alignment as well as bitfield handling.
> Basically the Dtrace was not able to handle the misaligned IP headers and was expecting a 4 byte alignment.
>
> Why is Dtrace involved here?
>
> When we run Analytics, internally it uses DTrace.
>
> How did the this issue happen?
>
> While running 'dd' command (which we don't recommend), one of the advanced statistic was enabled, called "IP Bytes", on Node T, which triggered the DTrace issues.
>
> Also, if the "IP Bytes" was enabled on Node U, the same issue would have occurred.

The following recent fixes to S12 would resolve this issue:

15348062 SUNBT6465277 bitfield handling could use some work
15823151 SUNBT7205009 dtrace test suite tst.SizeofBitField.d does not adhere to sparc/x86
15652868 SUNBT6965013 DTrace compiler doesn't align structs properly
15499992 SUNBT6734858 dtrace misaligns some structs containing shorts
15639237 SUNBT6947457 DTrace mistakes at calculation of "sizeof()" for structure

As well as the following bug fix in 2013.1:

16407365 Graph on status page randomly becomes solid yellow

Solution

On the 7420 array, destroy the analytics dataset ip.bytes[hostname].

These were the results afterwards:

[linux_box ~]# dd if=/dev/zero of=/export/restore1/X3_DD.out bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 25.2983 seconds, 414 MB/s

[linux_box ~]# dd if=/dev/zero of=/export/restore2/X3_DD.out bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 31.4028 seconds, 334 MB/s

***Checked for relevance on 25-MAY-2018***

Attachments

This solution has no attachment