Sun Blade 6000 System
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Interfaces
 
_Old GCS Categories>Sun Microsystems>Operating Systems>Solaris Network
 

In this Document

Purpose

Details

Product Support Team

Alerts

Description

Abbreviations/Acronyms

Versions

Hardware Versions

Software Versions

Compatibility/Patches

Hardware Compatibility Matrix

Software Compatibility Matrix

Configuration

Pathes/Packages

Configuration Files Location

Important configuration commands

FAQ

Information Gathering

Installation

De-installation

Troubleshooting

Performance

Lab

Contacts

Proactive

Further Reading

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Solaris and Network Domain (SaND) Product Page, internal content

Applies to:

Purpose

This document contains the Virtualized 40 GbE Network Express Module (sxge) Product Page

Details

Product Support Team

PLA: SN-SND: Sun Network Interfaces (L2)

Alerts

Description

• The Sun Blade 6000 Virtualized 40 GbE Network Express Module (NEM) is a multi-purpose connectivity module for the Sun Blade 6000 modular system.
• The NEM supports connection to external devices through four 10 Gigabit Ethernet small form-factor pluggable (SFP)+ ports and connects server modules (blades) in a Sun Blade 6000 modular system chassis.
• The SFP+ ports can be used in 1x 10GBE, 2x10GBE or 1x 40GBE configuration
• The NEM's interface is visible on the blade system via the sxge driver.
• The NEM is also known as "SOL NEM" (project name).
• sxge is a multithreaded, loadable, clonable, GLDv3-based STREAMS driver.
• GLD is a multithreaded, clonable Generic LAN Driver support module which resides in a loadable kernel module, /kernel/misc/gld. It implements most of the STREAMS functions and DLPI(7P) functionality required of a LAN driver. Several Solaris network drivers are implemented using GLD.
• The NEM also provides the magnetics and RJ-45 connectors for the ten 10/100/1000 BASE-T Ethernet interfaces that come from server modules through the midplane. There is no active circuitry on the NEM for these GbE ports.
• The total number of Ethernet ports is 14.

Abbreviations/Acronyms

• 10GBE - 10 Gigabit Ethernet (according to IEEE 802.3ae)
• 40GBE - 40 Gigabit Ethernet (according to IEEE 802.3ba)
• FDX - Full Duplex: Capability to send and receive frames at the same time.
• HDX - Half Duplex: Either send or receive frames at given time.
• IEEE - Institute of Electrical and Electronics Engineers, Inc., standardizes Ethernet
• Jumbo Frame - An Ethernet frame which is larger than 1518 bytes (typical 9000 bytes).
• NIC - Network Interface Card
• VLAN - Virtual LAN, the ability to have several independent net segments within one hardware.
• LAN - Local Area Network, network which is geographically limited (typically to a 1 km radius, e.g. within a building)
• IEEE 802.3ae - 10 Gigabit Ethernet Standard

Versions

Hardware Versions

Software Versions

• unbundled sxge driver for older Solaris, Linux, Windows - obtain via MOS Patches and Updates
• Solaris bundled sxge driver startin with Solaris 10 U11

Note - do not overwrite the Solaris bundled version with downloaded unbundled version

Compatibility/Patches

Hardware Compatibility Matrix

• See System Handbook Blade 6000 Full Components Lists, Ethernet Section
• Check the NEM User's Guide and Release Notes for hardware requirements and restrictions
• SFP+ ports support optical module cables (short or long range) or twin-axial cables (User's Guide E21078-03 page 11)
• Supported transceivers and cables are found on System Handbook full components list page.
• X2130A-1M-N X2130A-3M-N 10Gbps SFP+ TwinX Cable, 1 and 3 m
• X5561A-Z-N 10Gbps Short Wave Pluggable Transceiver (SFP+)
• X5562A-Z 10Gbps Long Wave Pluggable Transceiver (SFP+)
• In 10GbE and 40GbE configurations, the 5-meter copper splitter (P/N X2125A-5M) is not supported on the SFP+ ports (User's Guide E21078-03 page 49)
• Use latest firmware, download via MOS Patches and Updates <patch 17497193>

Software Compatibility Matrix

Configuration

• Pathes/Packages

  • SUNWcarx - Core Architecture Root (64-bit), contains gld (SPARC)
  • SUNWcsr - Core Solaris Root, contains gld (x86)
  • SUNWdsxge - Contains the sxge driver
  • 5.11 pkg : driver/network/ethernet/sxge
• Path:
• Manual search path: /usr/man

• Configuration Files Location

  • /etc/hostname.sxge<instance>
  • sxge.conf, e.g. /kernel/drv/sxge.conf

• Important configuration commands

  • Configure Jumbo Frames
    Edit /kernel/drv/sxge.conf, remove comment before accept-jumbo = 1; and reboot :
    
    

    ...
    # To enable jumbo support,
    accept-jumbo = 1;
    #
    # To disable jumbo support,
    # accept-jumbo = 0;
    #
    # Default is 0.
    ...

  • Configure VLAN tagging (802.1q)
    • VLAN tags get enabled by using "magic instance numbers", consisting of the original instance and the VLAN tag.
    • boot persistant config: create /etc/hostname.hcge<vlan-tag><instance number>
    • The hostname in this file should be found in /etc/hosts
    • The VLAN tag must be the same on the switch the Hydra NEM is connected to

                                            instance # (last 3 digits, value=0 -> uses sxge0 interface)
                                            |||
                       /etc/hostname.sxge123000
                                         |||
    VLAN tag (digit 4 and higher, value=123)
    
    # cat /etc/hostname.sxge123000
    va64-x6440b-gmp03
    # grep va64-x6440b-gmp03 /etc/hosts
    129.156.213.163 va64-x6440b-gmp03
    # ifconfig sxge123000
    sxge123000: flags=sxge123000 mtu 1500 index 2
                inet 129.156.213.163 netmask fffff800 broadcast 129.156.215.255
                ether 0:21:28:29:90:69
    

FAQ

1. Question: Can we use Link Aggregation with sxge ?
   Answer:  Yes, Static Link Aggregation is supported with the 2 sxge ports of the same NEM. It is configured on the blade OS. LACP can not be used (needs to be off). On the Switch side a static link aggregation (channel-goup mode on) should be configured for the two NEM ports. See User's Guide on Page 63. It has examples for Solaris dladm and Linux Bonding (round robin mode).

Information Gathering

• Information needed to assign to an other support group or level:
  • Short failure description including error or warning messages
  • Short customer situation including timeframes and business impact
  • Platform, part# and revision of NIC and PCI slot (if obtainable)
  • Explorer output (or path to it's location)
  • Network overview (detailed sketch, cable lengths, networking hardware)
  • Where is a test system connected (e.g. a "slow" client)
  • Short description of your measures to find and solve the problem before the handover

Installation

• See User's Guide- it contains a section per supported OS
• The SFP+ ports can be used in 1x 10GBE, 2x10GBE or 1x 40GBE configuration

De-installation

•  See User's Guide.

Troubleshooting

• get a problem object and defect (what is wrong with what)
• did it work once ?
• if so, what did change ?
• collect information as described at Information Gathering
• check ifconfig -a output if each interface has an unique MAC address
• in most cases eeprom local-mac-address\? should be true (SPARC)
• if several interfaces in same network are available on the system (e.g. an IPMP group): try to swap OK with not-OK cable connection
• try to connect an OK system directly for an back-to-back test
• try a different, tested cable
• check switch link settings, compare with the system's.
  
  port status of NEM0 and NEM1 :
  
  -> cd /CH/NEM0/SP/cli
  -> start
  > show -p 0
  > show -p 1
  > show conf
  > exit
  
  -> cd /CH/NEM0/SP/cli
  -> start
  > show -p 0
  > show -p 1
  > show conf
  > exit
  
  to compare this with the witch side, collect from switch CLI similar to
  
  #show interfaces status
  #show interface (port-conneted-to-NEM0)
  #show interface (port-conneted-to-NEM1)
  
  
   
• check if packet counter increases via e.g.
  
  # kstat -p sxge:0 | grep packets64
  repeat twice
• ping an adjacent system, repeat the above
• snoop the interface,  inspect the output e.g
  
  # snoop -d sxge0 -s 128
• collect a binary snoop for later inspection from problem time, e.g.
  
  # snoop -d sxge0 -o /tmp/snoop.out -s 128
• check that all systems in the subnet have the same MTU size (e.g. all 1500 or all 9000)

Performance

• See Relase Notes of Chassis, NEM and Blade Version for Hardware Requirements.
• Ensure CPU capacity (cycles and cores) for sending and receiving frames. Recommendation: 2 CPUs per 10GBE NIC.
• Use latest Solaris version for improved TCP/IP stack.
• Use the Uperf traffic generator to measure throughput, start with a low thread number and increase as long as a performance gain is observed.

Lab

Use Global Lab / jsc

Contacts

• support_ethernet_ww_grp@oracle.com mail alias - use for technical research 
• support_ethernet Mail Archive

Proactive

Further Reading

• Sun Blade 6000 Virtualized 40 GbE Network Express Module Documentation (SOL NEM) Document Library
• sxge(7D) man page
• External Marketing Page : http://www.oracle.com/goto/networking
• Virtualized 40 GbE NEM Marketing Page : http://www.oracle.com/us/products/servers-storage/servers/netra/6000-40gbe-nem/overview/
• Internal 40 GbE NEM Marketing Page : http://my.oracle.com/site/pd/sss/products/networking/blades/40gbe-nem/ (FAQs, White Papers)
• TOI slides are attached, see below.

References

This solution has no attachment