Oracle Exadata Hardware
 
Exadata Database Machine X2-2 Hardware
 

PLA-Support>Sun Systems>SAND>Network>SN-SND: Sun Network Infiniband
 
_Old GCS Categories>Sun Microsystems>Operating Systems>Solaris Network
 

In this Document

Purpose

Details

Product Support Team

Maintenance

Alerts

Description

Features

Exadata Storage Software Unique Features

Versions

There are three types of switches :

Systems Overview :

Compatibility/Patches

Ethernet Devices (Solaris Database Servers only!)

Ethernet Switch

InfiniBand Switch

Configuration

FAQ

Information Gathering

Ethernet Interfaces (Solaris Database Servers only!)

Ethernet Switch

InfiniBand Switch

Installation

Cabling

Network Overview

Ethernet Network Details

Exadata Storage Servers

Database Servers

Client Access With Bonding (Default)

Client Access Without Bonding

InfiniBand Network Details

Single Rack Case - Without Spine Switch

Single Rack Case - With Spine Switch

Multiple Rack Case - Up to 8 Racks

De-installation

Troubleshooting

Performance

Lab

Reference

Manuals

Trainings

TOIs

Other Important Links

Contacts

Aliases

Proactive

References

Oracle Confidential PARTNER - Available to partners (SUN).
Reason: Solaris and Network Domain (SaND) Product Page, internal content

Applies to:

Purpose

Note: This page doesn't cover any Oracle Enterprise Linux related stuff as this is not supported by our group.

Details

Product Support Team

SN-SND: Sun Network Ethernet Switches                                                       Product id 7558

Maintenance

Name              Function      Phone                            Geo/Country
Dan Black          Expert       +1 781-442-1315
Gino Kawalski      Expert       +1 781-442-1123            AMER/BUR

Alerts

Description

Features

• Balanced Incremental Scaling for OLTP and Data Warehousing
• Scale to 8 Racks by Just Adding Cables
• Two Operating System Choices on the Database Servers
• Oracle Linux
• Solaris 11 Express (x86)–Coming Soon
• Customers will choose their preferred Database Server OS at installation time
• Exadata Storage Servers will continue to be Oracle Linux
• Database Machine comes with DBFS shared Linux file system
• Shared storage for ETL staging, scripts, reports and other application files
• Files stored as SecureFile LOBs in database tables stored in Exadata
• Protected like any DB data –mirroring, DataGuard, Flashback, etc.
• 5 to 7 GB/sec file system I/O throughput
• First Secure Database Machine
• Moves decryption from software to hardware (Over 5x faster)
• Near zero overhead for fully encrypted database
• Queries decrypt data at hundreds of Gigabytes/second
• Enterprise Manager
• Manage & administer Database and ASM
• Monitor the Exadata Database Machine Hardware
• Plug-ins to monitor the Exadata Storage Cells and other components in the Database Machine
• Embedded Integrated Lights Out Manager (ILOM)
• Remote management and administration of hardware
• Comprehensive CLI
• Local Exadata Storage cell management
• Distributed shell utility to execute CLI across multiple cells
• Auto Service Request (ASR)
• File SRs automatically for common hardware faults

Exadata Storage Software Unique Features

• Exadata Smart Scans
• 10X or greater reduction in data sent to database servers
• Exadata Storage Indexes
• Eliminate unnecessary I/Os
• Hybrid Columnar Compression
• Efficient compression increases effective storage capacity and increases user data scan bandwidths by a factor of up to 10X
• Exadata Smart Flash Cache
• Breaks random I/O bottleneck by increasing IOPs by up to 20X
• Doubles user data scan bandwidths
• I/O Resource Manager (IORM)
• Enables storage grid by prioritizing I/Os to ensure predictable performance
• Quality of Service (QoS)
• Actively meet and maintain SLAs
• Memory Guard to protect existing current transactions from memory-based failures

Versions

There are three types of switches :

1) DCS 36 (NM2-36p) Infiniband switch (additional information in <Document 1396556.1> )

• provides the infiniband switched network between the database and storage cell servers
• hosts the sub-net manager for the Infiniband fabric
• a full rack has three DCS 36, the half and quarter rack configurations both have two DCS 36

2) Cisco 4948 Catalyst Gigagbit Ethernet switch

• provides connection within the rack for all the net0 and ILOM management ports of the servers and infiniband switches
• each rack has one Cisco 4948
• see Product Page for Cisco Ethernet 4948 Support page for Engineered Systems <Document 1485292.1>

3) Avocent KVM switch

• provides connection for the in-rack vdu and keyboard to the database and storage cell servers
• each rack has one Avocent KVM
• this is not a network switch (supported by the hardwarteam)

Systems Overview :

(components perrack)

Compatibility/Patches

Ethernet Devices (Solaris Database Servers only!)

Please refer to the ethernet device driver pages for patch infomation:

• igb
• ixgbe

Ethernet Switch

• Cisco Catalyst 4948 (371-4784)

Note:

We don't change software versions or do firmware upgrades, so we just use it as it gets delivered.

InfiniBand Switch

• Sun Datacenter Switch DCS36 X2821A-Z (541-3495)

• Firmware versions:

• Examine the Firmware version:

[root@ibswitch2 ~]# nm2version



NM2-36p version: 1.1.3-2

Note:

Expected firmware: 1.3.3-2. The original systems were released with firmware version 1.0.1-1 – the current version is 1.3.3-2. Please follow the instructions below if the firmware is downrev'd.

• Upgrade the Firmware version:

• Locate the firmware

• EIS-DVD (28SEP10 or later)

It's in directory ./sun2/patch/DC-SWITCH

• My Oracle Support

Patch 12373676: EXADATA V2 INFINIBAND SWITCH SOFTWARE UPDATE 1.1.3-2B (198.6 MB)

• Unpack the firmware on a Solaris/Linux server or laptop

[root@laptop]# mkdir fdir



[root@laptop]# cd fdir



[root@laptop]# gzcat <location>/SUN_DCS_36p_1.3.3.tar.gz



[root@laptop]# gzcat spfw_upgrade_1.3.3.tar.gz | tar xvBpf - (creates the packages directory)

• Copy the install script to each switch

[root@laptop]# scp spfw_upgrade_1.3.3.sh root@ibswitch2:/tmp

• Run the install script on each switch

[root@ibswitch2 /tmp]# ./spfw_upgrade_1.1.3.sh ftp://<laptopuser>:<password>@192.168.1.x/<full path to packages>







========================================================================



Packages not belonging to the switch firmware being installed are found!



These packages are:



rhpl wireless-tools



========================================================================



The package rhpl does not belong to the switch firmware version which is



being installed!



It is recommended that the package is uninstalled. Do you agree to remove



rhpl? [Y/n]: Y

• Completing the firmware update

• Edit the /etc/opensm/opensm.conf & change log_flags to 0x03
• Confirm the firmware version

[root@ibswitch2 ~]# nm2version



NM2-36p version: 1.3.3-2

Configuration

Please follow the latest Exadata EIS Checklist.

FAQ

Information Gathering

Ethernet Interfaces (Solaris Database Servers only!)

Please get the following data/information for any ethernet interface issue on Solaris based Exadata Database Servers:

• Explorer output

or

• Collect the following files/outputs, and put them in an tar file:

/etc/path_to_inst



/etc/release



/etc/system



/kernel/drv/<driver>.conf



/var/adm/messages 







# dladm show-phys



# dladm show-ether



# dladm show-dev



# dladm show-aggr



# dladm show-aggr -L



# ifconfig -a



# netstat -rn



# kstat -p



# modinfo | grep <driver>



# pkginfo -l



# prtdiag -v



# prtconf -vp



# fmadm faulty



# fmdump -eV

Refer to the specific ethernet device driver pages for further informations:

• igb
• ixgbe

Ethernet Switch

• Cisco Catalyst 4948 (371-4784)

Please get the following data/information for any Cisco ethernet switch issues (Management & ILOM connections):

show running-config



show tech-support



show logging

Note:

Troubleshooting is just about simple configuration checks, and hardware replacements.

InfiniBand Switch

• Sun Datacenter Switch DCS36 - X2821A-Z (541-3495)

Please get the following data/information for any InfiniBand switch issues:

• From each switch:

/var/log/messages



/var/log/opensm.log



/var/log/opensm-subnet.lst







# /usr/local/bin/nm2version



# /usr/local/bin/env_test



# /usr/local/bin/listlinkup



# /usr/sbin/ibcheckerrors -v



# /usr/bin/ibdiagnet -skip dup_guids -pm



 (collect all the files it creates in /tmp: ibdiagnet.db, ibdiagnet.fdbs, ibdiagnet.log, ibdiagnet.lst, ibdiagnet.mcfdbs, ibdiagnet.pkey, ibdiagnet.pm, ibdiagnet.sm, ibdiagnet_ibis.log)

• From any database or storage server:

# /opt/oracle.SupportTools/ibdiagtools/verify-topology -v 



 (collect the output and the files it creates in /tmp: ib_hosts.lst, ib_switches.lst, ibnetdiscover.map)

Refer to the Sun Datacenter Infiniband Switch 36 page for additional details.

Installation

Please follow the latest Exadata EIS Checklist.

Cabling

Except for external connections everything is pre-wired at factory. For verification you can check the Cabling Tables for Oracle Exadata Database Machine.
The switches are factory installed.

Network Overview

Network Diagram for Exadata X2-2 WITHOUT Bonding:		Network Diagram for Exadata X2-2 WITH Bonding:

Note: Oracle Exadata Database Machine X2-8 Full Rack has BONDIB0, BONDETH0, BOND2, and BOND3 on the InfiniBand private network.

• Management network: This required network connects to your existing management network, and is used for administrative work for all components of Oracle Exadata Database Machine. It connects the servers, ILOM, and switches connected to the Ethernet switch in the rack. There is one uplink from the Ethernet switch in the rack to your existing management network, one uplink from the KVM switch in the rack to your existing management network, and one uplink for each power distribution unit (PDU).

   

  ---

  Note: Network connectivity to the PDUs is only required if the electric current will be monitored remotely.

  ---

  Each database server and Exadata Storage Server has two network interfaces for management. One provides management access to the operating system through the NET0 Ethernet interface, and the other provides access to the Integrated Lights Out Manager through the ILOM Ethernet interface. Oracle Exadata Database Machine is delivered with the NET0 and ILOM interfaces connected to the Ethernet switch on the rack. The NET0 interface on the database servers should not be used for client or application network traffic. Cabling or configuration changes to these interfaces on Exadata Storage Servers is not permitted.

   

• Client access network: This required network connects the database servers to your existing client network and is used for client access to the database servers. Applications access the database through this network using Single Client Access Name (SCAN) and Oracle RAC Virtual IP (VIP) addresses. Database servers support Linux channel bonding of NET1 with NET2 to provide higher bandwidth or availability for client connections to the database. Oracle recommends channel bonding for the client access network.

• Additional networks (optional): Database servers may be configured to connect to one or two additional existing networks through the NET2 and NET3 ports. If channel bonding is used for the client access network, then only one additional port (NET3) is available.

• InfiniBand private network: This network connects the database servers and Exadata Storage Servers through the BONDIB0 interface to the Infini Band switches on the rack. Oracle Database uses this network for storage and Oracle RAC cluster interconnect traffic. This non-routable network is fully contained in Oracle Exadata Database Machine, and does not connect to your existing network. This network is automatically configured during installation.

Note:

All networks must be on distinct and separate subnets from each other.

Ethernet Network Details

Exadata Storage Servers

• ILOM: Provides access to the Integrated Lights Out Manager (ILOM) using the management network.
• NET0: Provides access to the operating system using the management network.

Database Servers

• ILOM: Provides access to the Integrated Lights Out Manager (ILOM) using the management network.
• NET0: Provides access to the operating system using the management network.

Client Access With Bonding (Default)

• NET1: Provides access to the database using the client access network, and is bonded with NET2 to create BONDETH0.
• NET2: Provides access to the database using the client access network, and is bonded with NET1 to create BONDETH0.

   

  ---

  Note: Oracle RAC SCAN and VIP addresses are defined on the BONDETH0 interface.

  ---

• NET3: Optional use, e.g. may be used for connectivity to a dedicated tape backup network or connectivity to a disaster recovery network.

A bonded network configuration is used when there is a need to aggregate multiple network interfaces together for client connections to the database. This is the default configuration.

When channel bonding is configured for the client access network during initial configuration, the Linux bonding module is configured for active-backup mode (mode=1). If a different bonding policy is preferred, then you may reconfigure the bonding module after initial configuration. For configuration details, refer to the Linux Ethernet Bonding Driver HOWTO in the /usr/share/doc/kernel-doc-2.6.18/Documentation/networking/bonding.txt directory on any Oracle Exadata Database Machine database server.

Note:

When using a bonded network configuration, you must provide a network switch capable of supporting the chosen bonding mode. For example, if mode 4 is configured (IEEE 802.3ad Dynamic link aggregation), then you must supply and configure the network switch capable of supporting this bonding mode. Requirements for specific bonding policies are documented in the Linux Ethernet Bonding Driver HOWTO.

Client Access Without Bonding

• NET1: Provides access to the database using the client access network. The Oracle RAC SCAN and VIP addresses are defined on this interface.
• NET2: Optional use, e.g. may be used for connectivity to a dedicated tape backup network or connectivity to a disaster recovery network.
• NET3: Optional use, e.g. may be used for connectivity to a dedicated tape backup network or connectivity to a disaster recovery network.

InfiniBand Network Details

Single Rack Case - Without Spine Switch

• Available in V2 Quarter Rack, V2 Half Rack, X2-2 Quarter Rack
• Leaf switches – L1, L2
• Connect port1 from DB & Exadata Storage Servers to leaf switch “L1”
• Connect port2 from DB & Exadata Storage Servers to leaf switch “L1”
• Interconnect “leaf” switches with each other (7 links wide)

Single Rack Case - With Spine Switch

• Available in V2 Full Rack, X2-2 Half Rack, X2-2 Full Rack, and X2-8 Full Rack
• Use 3rd switch (S) as spine switch for expansion to multiple racks
• Connect each leaf switch to spine switch (1 links wide)

Multiple Rack Case - Up to 8 Racks

• Valid up to 8 Racks

• Only Full Racks and X2-2 Half Racks can be inter-connected
  • Half-racks must have spine switch

• Leaf switches – L1, L2 (2 per rack)
  • Remove inter-switch links between leaf switches

• Spine switches – S (1 per rack)
  • Connect each leaf switch to every spine switch

De-installation

Troubleshooting

• Ethernet Switch Replacement - Cisco Catalyst 4948
• When replacing an infiniband switch, or any unit in the rack, take care that no stress is placed on the infiniband cables.
  These cables are easily damaged, and must not be subjected to a bend radius of less than 50mm.

Note:

Oracle is doing the hardware replacement (part number 371-4784) directly, no call will be logged with Cisco.

1. Backup configuration

Switch>enable



Password: 



Switch#config terminal



Switch(config)#ip ftp username <user>     (ftp server login name)



Switch(config)#ip ftp password <password> (ftp server password)



Switch(config)#end



Switch#copy running-config ftp://<ftp server ip>//var/tmp/config.txt

Note:

If you can't access the switch anymore, you'll either need the configuration saved at installation time or any later backup done by the customer.

1. Replace switch

1. Restore configuration

Switch>enable



Password: welcome1



Switch#config terminal



Switch(config)#interface vlan 1



Switch(config-if)#ip address <ip addr> 



Switch(config-if)#end



Switch(config)#no ip routing



Switch(config)#ip default-gateway <ip addr>



Switch(config)#ip ftp username <user>     (ftp server login name)



Switch(config)#ip ftp password <password> (ftp server password)



Switch(config)#end



Switch#copy ftp://<ftp server ip>//var/tmp/config.txt running-config



Switch#copy running-config startup-config

• Infiniband Switch Replacement - Sun Datacenter Switch DCS36

Please refer to the Exadata Infiniband Replacement procedure.

Performance

Lab

v-exadata-a-gw-gmp03.uk (Linux Gateway - GMP Exadata lab) :
Login:      ssh v-exadata-a-gw-gmp03.uk.sun.com

Reference

Manuals

• Exadata Database Machine Documentation - Internal
• Oracle Enterprise Manager System Monitoring Plug-ins

   

Trainings

• PRT/INT Exadata V2 - Installation & Maintenance (July 2010) - Internal [ *.pdf ]
• PRT/INT Exadata X2 - Installation & Maintenance(November 2010) - Internal [ *.pdf ]

TOIs

• Exadata V2 Field Services TOI: [ Part1: Hardware Overview | Part2: Installation Process | Part3: Site Planning | Part4: EIS Installation ] - Internal [ *.pdf ]
• TSC x64 Exadata V2 Hardware TOI- Internal [ *.pdf ]

   

• Infiniband TOI - Session 1 by Muppalla Sridhar (June 2011) - Internal [ *.pdf ](see attachment)
• Infiniband TOI - Session 2 by Muppalla Sridhar (June 2011) - Internal [ *.pdf ] (see attachment)
• Infiniband TOI - Session 3 by Muppalla Sridhar (June 2011) - Internal [ *.pdf ] (see attachment)
• Infiniband TOI - Session 4 by Muppalla Sridhar (June 2011) - Internal [ *.pdf ] (see attachment)

   

• Solaris on Exadata Support Readiness Training Session 1
  • Exadata pre-installs, configuration scripts (David Brean) - Internal [ *.pdf ]
  • Core NUMA IO (Nicolas Droux) - Internal [ *.pdf ]
  • RDSv3 (Giri Adari) - Internal [ *.pdf ]

   

Other Important Links

• Oracle Exadata TSC Process Guide
• TSC Network: Exadata Switches Product Page
• TSC x64: Exadata Product Page
• FSG: Exadata (fsg.uk)
• Onestop: Oracle Exadata Database Machine
• Exadata EIS Checklist [ *.pdf ]
• exadata.us.oracle.com

Contacts

Aliases

• sig.ns Sun Generic Networking mail alias
• exadata-support@sun.com Sun Field/TSC Exadata Technical Alias
• tsc-global-EV2@sun.com Sun TSC Exadata Alias
• EXADATA_WW@ORACLE.COM Oracle Exadata alias
• Sun Mail Archive

Proactive

References

This solution has no attachment