Oracle SuperCluster T5-8 Full Rack
 
SPARC SuperCluster T4-4 Full Rack
 
Oracle SuperCluster M7 Hardware
 
Oracle SuperCluster Specific Software
 
Oracle SuperCluster T5-8 Half Rack
 
Oracle SuperCluster M8 Hardware
 
SPARC SuperCluster T4-4 Half Rack
 
SPARC SuperCluster T4-4
 
Oracle SuperCluster M6-32 Hardware
 
Oracle SuperCluster T5-8 Hardware
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 

In this Document

Purpose

Scope

Details

Hardware

1Gb Management Network

Infiniband Network

10Gb Client Network

M6, M7 & M8 PDoms

LDoms

Solaris Operating System

DB Domains & DB Zones

Application Domains

Zones

Internal ZFS Appliance

Patching

Overview

Pro-Active maintenance

Reactive maintenance

References

Applies to:

Purpose

This document provides guidance on the sort of changes that can or should not be performed on an Oracle SuperCluster

Scope

This is intended as a reference for customers, Oracle Support and Oracle ACS teams to ensure no changes are made that impact the stability or supportability of SuperCluster. It applies to all varieties of SuperCluster (T4-4, T5-8, M6-32, M7, M8, half, full, small, base, etc). This document provides guidelines only and is not exhaustive. If you have any doubt or questions, raise an SR and ask for guidance before making any significant configuration changes.

Details

Oracle SuperCluster is an engineered system (often referred to as an integrated system). As such, it has been designed, built & optimized by Oracle for performance, scalability, fault tolerance & stability. This is achieved by engineering the software & hardware together and testing it together. Doing so relies on limited & fixed configurations to provide best practice, best performance, better test coverage & better stability (avoiding edge cases). Changing certain aspects of system configuration can reduce the benefits of an engineered system and may also be unsupported. This document gives examples of configuration changes that are not supported on SuperCluster. This document provides guidelines only and is not exhaustive. If you have any doubt or questions, raise an SR and ask for guidance before making any significant configuration changes.

Hardware

Generally, there should be no changes, additions or removal of any hardware with the exception of:

• Optional 8Gb & 16Gb Emulex or QLogic FC cards can be installed in spare PCI slots in the servers and used to connect to SAN storage
• Memory & CPU upgrades are only allowable on T5-8 SuperCluster using the T5-8 half to full rack upgrade kit
• Multi-racking via Infiniband with other engineered systems is generally allowed e.g.
  • Multi-racked with other SuperCluster, Exadata, Exadata expansion, Exalogic
    • Except it is not supported to connect an OVCA/PCA to SuperCluster via Infiniband
• Connecting Sun/Oracle branded systems via Infiniband is allowed e.g.
  • Sun media server, Sun ZS4 ZFS appliance
    • Connecting other Sun/Oracle systems to SuperCluster over Infiniband is not recommended. If a customer encounters problems on their SuperCluster, they may be asked by support to demonstrate the problems without other systems connected over Infiniband
  • You must not connect non-Sun/Oracle branded systems or non-Sun/Oracle HCAs via Infiniband
• The Cisco 4948 management network switch can be replaced or removed (but you must not change the management network configuration on any of the SuperCluster components)
• PDUs may be replaced if moving to a data center with different power attributes
• Some other hardware additions/upgrades may be allowed but you must first obtain engineering exception approval. This can be arranged by Oracle pre-sales consultants and must be approved before system installation

1Gb Management Network

Generally, there should be no changes to the management network (several SuperCluster tools rely on management network operation and make assumptions about it's configuration). The following rules apply:

• All major components in the rack must be able to communicate over the management net e.g.
  • All ILOMs, Exadata storage, ZFS appliance, Infiniband switches, Solaris domains, DNS & NTP
• Should be a single subnet for all components within the rack
• Should ideally be a single subnet for all racks in a multi-rack config
  
  • If multi-racked with racks with different subnets, then the management networks must be route-able
• No disabling or removal of components from the management net (including ILOMs & Solaris domains)
  • However, removing the management net interface from local zones is allowed
• No disabling of DNS or NTP on the management net (both services must be visible & useable)
• You must not change link, interface, address, IP, etc. properties or ndd settings at the OS level on any components within the rack
• No aggregates
  • Only link based IPMP in active/standby mode are allowed
• No VLAN tags within the rack
  • The management network on the data center side of the switch can be tagged, which requires additional configuration of the Cisco 4948 switch
• You can change management network IP addresses post install, but the above rules all still apply

Infiniband Network

Generally, there should be no changes to the Infiniband network.

• You must not change link, interface, address, IP, etc. properties or ndd settings at the OS level on any components within the rack
  • This includes not being able to change the active/standby or active/active configuration
  • This includes not being able to configure probe-based IPMP (only link-based IPMP is supported)
• No changes to partition membership for standard partitions FFFF, 8501, 8502, 8503, 8511 & 8512
• New partitions are allowed
  • Support additional Solaris Cluster instances
  • Support for multi-tenancy security segregation
• IP addresses on Exadata partition FFFF must be on a single subnet
• IP addresses on ZFS appliance partition 8503 must be on a single subnet
• It's recommended to always use a 22-bit netmask
• Solaris 11 must operate in Reliable Connected (RC) mode
• Solaris 10 must operate in Unreliable Datagram (UD) mode
• Multiple IPMP groups on the same Infiniband subnet is OK on SuperCluster
  • See new comment in MOS note 1020182.1

10Gb Client Network

The client network is much more flexible. Most configurations and operations supported by Solaris is allowed on SuperCluster

• However, it doesn't make much sense to remove the client network from a dedicated domain or zone
  • Note that root domains (intentionally) do not have the client access network configured, except for the control domain.
• VLANs are supported in all domain types
• Aggregates (with LACP) are supported in dedicated domains only
  
  • Aggregates in root domains and IO domains are not supported
  • Aggregates with DLMP are not supported
  • The default IPMP group should be manually removed and replaced with appropriate vlan, aggr, vnic, etc objects. Best performed by ACS during system installation
• Can change link, interface, address, IP, etc. properties or ndd settings (whatever is necessary to accommodate customer data center network properties)
• Can have domains & zones on different client subnets
• Can have multiple client networks in a single domain or zone e.g. Client network & backup network
  • Use spare ports on existing NICs
  • If no spare ports, raise a hardware exception to add more cards or create a larger domain that owns more IO slots
• You must not use the NICs on any optional FC HBAs for network connectivity (only SAN)
• Oracle recommends that only HA configurations are used e.g. IPMP, LACP, DLMP

M6, M7 & M8 PDoms

• You must not change any aspect of PDoms post-install
  • This can only be performed via a reset and re-install (an ACS service)

LDoms

• You can (and often do) run a workload in the primary/control domain
  • Most often a Solaris 11 DB domain, a Solaris 11 application domain
• You must not run any application workloads or zones in a root domain (workloads can only run in dedicated DB domains, application domains or IO domains)
  • However, it is OK to run OEM agents in a root domain
• You must not make any configuration changes using the ldm command i.e.
  • Generally, you should not use ldm add-, set-, remove-, migrate-, create-, destroy, etc. commands
  • There are some exceptions that you can use though:
    
    • 'ldm set-io' to set VLAN tags on virtual functions in IO domains as per document 2063071.1 - SuperCluster: Enabling 802.1Q VLAN
    • 'ldm add-spconfig' to save the current configuration to the Service Processor
    • 'ldm set-variable' to set an OBP variable for a domain
• You must not change the LDom names (as displayed by 'ldm ls')
• You must not create or modify domains using Enterprise Manager or Ops Center
  • Dynamic domain creation can only be performed using the IO domain virtual assistant BUI on version 2.0 (or later) SuperClusters
• CPU & memory can be reassigned between dedicated domains only by using the /opt/oracle.supercluster/bin/osc-setcoremem command
• CPU & memory can be reassigned between IO domains only by using the IO domain virtual assistant BUI interface
• IO domains can be be created, modified & destroyed only by using the IO domain virtual assistant BUI on version 2.0 (or later) SuperClusters
• Once IO domains ldoms have been created using SVA, one may not ever again use Soalris LDOM command line to create or modify or delete anything within the LDOM set up.
• You must not patch or upgrade the LDoms software independently of Solaris or outside of the SuperCluster QFSDP, unless under instructions from Oracle Support
• LDoms Live Migration is not supported on SuperCluster because all domains include physical IO hardware for which Live Migration is not available

Solaris Operating System

•  You must not change anything that would conflict with ssctuner e.g.
  • Some /etc/system settings, sd.conf, ssd.conf, ndd
  • Enabling intrd, poweradm, nxge services/drivers
• You must not apply Solaris updates, SRUs or kernel IDRs outside of the QFSDP (and SuperCluster IDR) without the express approval of Oracle support via an SR.
• You can apply userland (non-kernel) IDRs independently
• You must not change the layout/partitioning of the T4/T5/M6 internal disks
• You must not change the rpool layout or policies
• You can create additional filesystems/datasets in existing pool(s)
• You can create additional zpools on spare partitions on internal disks
  • Take extreme care not to overwrite a partition that may be in use as a vdisk by another domain
  • If you use a spare partition on internal disks, it may have to be relocated during a T5-8 half to full rack upgrade
• UFS filesystems are not supported anywhere in SuperCluster
• You can change hostname, management & client IP addresses, DNS, NTP, etc. post-install
• Deduplification of local ZFS filesystems is not supported
  • Note that deduplification of any appliance based filesystem is also not supported
• iSCSI static discovery is the only supported discovery method on Solaris domains. Send targets and iSNS are not supported methods

DB Domains & DB Zones

• You can run some applications in DB domains or DB zones e.g.
  
  • EM and Ops Center agents or backup agents/clients
  • Applications running in a DB domain should preferably be run in a zone to separate them from the DB processes
You must not run anything that has the potential to interfere with RAC or DB performance
• If you're running 3rd party applications in DB domains/zones, you may be asked to disable or remove the application if you experience instance or node evictions
• You must not run Solaris Cluster in DB domains
• You must not change default scheduling policies
• You can add & remove Solaris packages
  • DB domains now based on solaris-minimal-server anyway
• You can security harden DB domains
  • Be careful not to impact RAC. Refer to SuperCluster security hardening white papers on MOS
• DB zones in DB domains must be created using the Exadata Java One Command (JOC) tool (which is dependent upon the ssc-exavm package)
• You must not run RAC in a Database Domain in both the Global Zone and in Non-Global Zones
  • A DB domain hosting DB Zones must not host any DB instances in the global zone
• You can run versions of the DB older than 11gR2 in a DB domain or DB zone, but they cannot use the Exadata storage. They must use the ZFS appliance or external storage
• You must not change the of DB zones names (as displayed by 'zoneadm list')
• You can change hostname, management & client IP addresses, DNS, NTP, etc. post-install

Application Domains

Generally, application domains are much more flexible. They can run any application supported by Solaris. The rules in the previous sections still apply though (Solaris, LDoms, PDoms, networking).

Zones

• Physical memory capping (zone.capped-memory)) is not supported on any SuperCluster zone
• Virtual memory capping (zone.swap-max) is not supported in DB domains (including both dedicated and IO domains)
• Virtual memory capping (zone.swap-max) is supported on SuperCluster only in application domains
  • Memory caps should not be set for zones or projects hosting mission-critical applications. Caps should only be set for non-mission-critical applications where it is important to prevent them from consuming excessive amounts of swap at the expense of mission-critical applications running in the same domain
• Other zone memory control features can be used e.g.
  • zone.max-shm-memory
• You can use all CPU resource control features (pool, capping, etc)
• The same rules for DB domains apply to DB zones
• The same rules for application domains apply to application zones
• Kernel zones are not supported on SuperCluster
• Solaris 10 branded zones are supported only in application domains
  • Note that Solaris 10 branded zones provide the only support for Solaris 10 on SuperCluster M7 & M8
• Immutable zones are supported only in application domains

Internal ZFS Appliance

• Only NFS and iSCSI storage is supported
  • No CIFS, FC, etc.
• iSCSI storage is not supported for general purpose storage
  
  • iSCSI storage is only supported for infrastructure purposes, as setup by the SuperCluster tools
    • Limited to dedicated, root & IO domain boot disks, zone root filesystems for DB & application zones, Solaris Cluster quorum device
  • NFS must be used for general purpose use
• You must not change storage or SuperCluster pools layouts or policies
• You must not destroy the IPS-repos filesystems (it's needed for IO domain creation & patching)
• You can create additional NFS shares
• You can use the 2 spare on-board NICs in each head to connect to a client or separate backup net
  • The NICs will be identified as igb2 & igb3 (or ixgbe2 & ixgbe3) on each head. igb2/ixgbe2 may have a 1Gb management network cable connected, this can be removed.
• Deduplification of any appliance based filesystem is not supported
  • Note that deduplification of host based local ZFS filesystems is also not supported
• Data encryption or compression on the appliance is not supported
  • Use host based encryption or compression (on the host domains) on iSCSI LUNs if data encryption is required
• iSCSI LUNs with thin provisioning (sparse = true) are not supported
• ILOM/BIOS firmware should only be updated by Oracle support personnel via an SR
• Do not overload the internal ZFS appliance with high IOPS as it is used primarily for infrastructure purposes:
  • On T4 SuperClusters it provides iSCSI storage for zones
  • On T5 & M6 SuperClusters it provides iSCSI storage for IO domains & zones
  • On M7 & M8 SuperClusters it provides iSCSI storage for all domains and zones
  • Hence intensive operations such as large NFS copies or RMAN backup to the internal ZFS appliance can cause domain & zone hangs and sometimes lead to RAC node evictions
• With M7 & M8 SuperCluster, Oracle recommends that the internal ZFS appliance is not used for any other purpose than system/infrastructure use (storage for domains & zones, system log files, quorum for the Exadata Storage Servers). We no longer recommend that it is used for pre-11gR2 Oracle Database, file sharing, database backups, ETL or other similar purposes

Patching

Overview

There are two types of maintenance – Proactive Maintenance to keep up to date and prevent issues and Reactive Maintenance to address issues already encountered. Maintenance updates are provided quarterly for the proactive maintenance of SuperCluster. Reactive Maintenance should be undertaken in consultation with Oracle support who will advise if an update should be applied to a component.

Pro-Active maintenance

This is performed using the SuperCluster Quarterly Full Stack Download Patch (QFSDP), released on MOS every January, April, July & October. They form the core part of a pro-active patching strategy for SuperCluster. They contain the latest versions TESTED ON SUPERCLUSTER for all major SuperCluster components - Solaris 10 & 11, DB & GRID, Exadata Storage Server, ZFS Appliance, platform & ILOM firmware, Infiniband switch firmware, Solaris Cluster (3.3 & 4.x), Exa-family, PDU, Infiniband HCA firmware. They are expected to be installed by anyone - customers, partners, ACS, Oracle Platinum Patching team, etc. Scripts are provided to assist with installation. Note that not every component is updated each quarter. Refer to MOS doc Contents of each SuperCluster Quarterly Full Stack Download Patch (QFSDP) [2056975.1]for details.

It's recommended to keep systems no more than 6 months out of date i.e. No older than 2 QFSDPs. The QFSDP must be installed as a unit so all components are updated together - There's no dim-sum or staggered/staged patching allowed without the express approval of Oracle support via an SR. Users should follow the included checklists and use the provided tools to perform patch installation.

Reactive maintenance

Reactive patching IS allowed for exceptional situations. Usually to address critical issues with no easy/viable workaround that are not fixed in the latest QFSDP. In which case, the individual component will be patched with a one-off fix for the specific bug. Customers must accept the risks of running a patch/version that may not have been fully tested on SuperCluster in combination with other components. Reactive Maintenance should only be undertaken in consultation with Oracle support (via an SR) and in most cases the system should be updated with the next QFSDP to properly address the issue.

References

This solution has no attachment