Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 
Oracle SuperCluster Specific Software
 
Oracle SuperCluster M7 Hardware
 
Oracle SuperCluster M8 Hardware
 
SPARC SuperCluster T4-4
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 

Instructions for capping the NFS traffic between the SuperCluster domains or zones on the one hand and the ZFS Storage Array on the other hand.

In this Document

Goal

Solution

Evaluate ISCSI Latency

Identify The NFS Clients

Set Up The NFS Flows

Check Results

Remove Statistics

References

Applies to:

Goal

On SuperCluster Solaris Zones and IO Domains are installed on iSCSI LUNs exported by the internal ZFS Storage Appliance (ZFSSA). On SuperCluster M7, Dedicated and Root Domains are also installed on such LUNs.

A SuperCluster configured with many IO Domains and Solaris Zones - especially an M7 SuperCluster - can become sensitive to iSCSI performance: an additional load on the ZFSSA - such as some NFS traffic generated by backup activities - can have a negatively impact on the iSCSI latency and on the overall performance.

In such a case Solaris Crossbow flows can be put in place to control the NFS load on the ZFSSA.

Solution

Evaluate ISCSI Latency

Connect to the ZFSSA kiosk (a.k.a. browser user interface), select Analytics located on the right hand side. Click on Add statistic and from the pull-down menu, in the PROTOCOL section, select ISCSI operations/Broken down by latency.

The image below (ISCSI Latency) illustrates a situation where the latency is above one second on a regular basis: a backup is going on creating a heavy NFS load on the ZFSSA.
This is a typical situation where Crossbow flows can be used to limit the network bandwidth available to NFS and thus reduce the pressure on the ZFSSA.
The net result is an improvement of the iSCSI latency.

Identify The NFS Clients

Crossbow flows must be setup on the NFS-client side. It is then necessary to identify the clients that generate the NFS load.

Connect to the ZFSSA kiosk, select Analytics located on the right hand side.
Click on Add statistic and from the pull-down menu, in the PROTOCOL section, select NFSv3 operation/Broken down by client.

The image below (NFS Clients) shows a situation where many NFS clients - identified by their IP addresses 192.168.30.14 to 192.168.30.11 - are generating some NFS load. The IP addresses are located on the Infiniband
storage partition of the SuperCluster, the one that provides access to the ZFSSA.

Set Up The NFS Flows

This operation must be repeated on each NFS client for which it is deemed appropriate to limit the NFS traffic.

This operation can be completed without any service interruption.

Connect as root to the NFS client generating the load and identify the network interface on which the IP address is defined:

This is an IP Multipathing (IPMP) group.
Identify the network links used by this group:

The IPMP group uses the stor_ipmp0_0 and stor_ipmp0_1 links. The latest is configured as a standby link.

Optionally, ensure that the links are Infiniband partitions:

The two links are indeed Infiniband partitions. They are using the 8503 key, which identifies the storage partition that gives access to the ZFSSA.
Good.

Set up a Crossbow flow on each link:

* remote_port is set up to the standard port for an NFS server (2049)

* transport is set up to cover both TCP and UDP. Note that this option is only valid and required on Solaris 11.2. With Solaris 11.3 and above this option is obsolete

* maxbw defines the maximum bandwidth allocated to NFS. In this example, as one of the link is configured as standby, the bandwidth consumed by NFS won't go over 200 Mbits/sec

Once a flow is created, its maxbw parameter can be adjusted at will with the command flowadm set-flowprop -p maxbw=newbandwidth. This does not require any service interruption.

Flows can be removed with the command flowadm remove-flow.

Check Results

Once the flows are in place, the analytics feature of the ZFSSA described above can be used to check the result. The picture below (Flow Results) illustrates the situation with flows set up on all the NFS clients identified previously.

All the flows have been tuned with a 200Mb/sec bandwidth:

* The iSCSI latency does not go over one second anymore

* The number of NFS operations per second is evenly balanced across the different clients

Remove Statistics

Fron the ZFSSA kiosk, select Analytics located on the right hand side. Remove the statistics created to evaluate iSCSI latency and NFS clients. These are high overhead statistics and should not be left running.

References

This solution has no attachment