Oracle SuperCluster M6-32 Hardware
 
SPARC SuperCluster T4-4 Half Rack
 
Oracle SuperCluster T5-8 Half Rack
 
Oracle SuperCluster T5-8 Full Rack
 
SPARC SuperCluster T4-4 Full Rack
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>SPARC SuperCluster>DB: SuperCluster_EST
 

When a SuperCluster Infiniband switch is rebooted, database nodes may evict on domains with a large number of RDS connections. The problem is due to a timeout threshold being reached before all the RDS connections can be re-established. There are a number of factors which impact the number of RDS connections.  This Doc provides information for users to determine their current number of RDS connections and prediction formulae to determine the likely number of RDS connections after applying this QFSDP.  If the prediction formulae indicate that applying this QFSDP would exceed the current safe threshold of RDS connections, then file an SR and await further advice. The prediction formulae should also be used before deploying additional Databases on a system.

In this Document

Symptoms

Changes

Cause

Solution

References

Applies to:

Symptoms

One of more database nodes may evict due to diskmon split brain for RAC or diskmon may fence off cells for non-RAC.

If this has occurred, indications similar to the following can be found in the diskmon.trc file after the rebooted switch comes online:

If the impacted database is in a zone, the zone may or may not transition to the state "shutting_down" and possibly hang.

This may be observed using the zoneadm command:

To fully recover from a hung zone, the domain containing the database zones needs to be rebooted.

Changes

When a SuperCluster Infiniband switch is rebooted, database nodes may evict on domains with a large number of RDS connections. The numbers stated are on a per LDom (global zone)  basis not the cumulative across all LDoms(global zones)

There are a number of factors which impact the number of RDS connections, in particular Network Resource Management (a.k.a. Exadata QOS / TOS), which is enabled in Database 12.1.0.2, Solaris 11.2 SRU1, and 12.1.2.x.x Exadata Storage Cells. These, or later, versions are included in the SuperCluster QFSDP from July 2015 onwards. Other factors include the number of Databases, the number of Exadata Storage Cells they use, etc.

This Doc provides information for users to determine their current number of RDS connections and prediction formulae to determine the likely number of RDS connections after applying the July 2015 QFSDP or later.

This July 2015 QFSDP or later will be suitable for most SuperCluster customers to install as is.  In particular, enhancements contained in the July 2016 QFSDP and further enhancements contained in Oct 2016 QFSDP will make these QFSDPs suitable for the vast majority of customers.

If the prediction formulae indicate that applying this QFSDP would exceed the current safe threshold of RDS connections, then file an SR and await further advice.

The prediction formulae should also be used before deploying additional Databases on a system.

Cause

The problem is due to a timeout threshold being reached before all the RDS connections can be re-established on SuperCluster domains with a large number of RDS connections.  

Solution

Incremental improvements have been made in Solaris 11.3 SRU7 (in July 2016 QFSDP) and SRU11 (in Oct 2016 QFSDP).

The safe RDS connection limit is:

• 200 for systems below Solaris 11.3 SRU7
• 700 for systems running Solaris 11.3 SRU7 (July 2016 QFSDP level)
• 1,800 for systems running Solaris 11.3 SRU11 (Oct 2016 QFSDP level)
• 3,400 for systems running Solaris 11.3 SRU14 (Jan 2017 QFSDP level)

1. To determine the number of RDS connections currently on a System, run the shell script below (conn-cnt.sh) in the global zone of all database domains. If the reported number of RDS connections is below the safe RDS connection limit in each domain, then your system is not currently at risk. Proceed to step 2.

If the number of RDS connections is above the safe RDS connection limit in any domain, log a service request for further assistance.

$ cat conn-cnt.sh

2) Before applying the July 2015 QFSDP or later, creating more database zones, adding storage expansion or migrating database versions - for example, to 12.1.0.2 - use the following formulae to predict the likely resultant number of RDS connections. If the prediction is above the safe RDS connection limit, log a service request for further assistance.

C is the number of IP addresses specified in file  /etc/oracle/cell/network-config/cellip.ora. Note: There may be more than one ip address per line and each should be counted.

N is the node count in the cluster. Run 'olsnodes' and count the number of nodes: Run as root or grid user:

For 12.1.0.2 databases, the formula is:

  B(B*N*2 + C*7)

For 11g and 12.1.0.1 database versions, the formula is:

  B(B*N*2 + C*2)


Sum the above for all database zones under a single domain. If there are no  zones, it's just the sum of all databases running in the domain itself.

For example, rds_count=Zone_1_12.1.0.2[B(B*N*2 + C*7)] + Zone_2_11.2.0.4[B(B*N*2+*C*2)] + Zone_3_12.1.0.1[B(B*N*2+*C*2)].

This bug introduced by the putback for BugID: 16024464 - QoS - Segregate RDS traffic based on SL (/PSARC/2013/237 - IB QOS for RDSv3)
<Bug 22380320> - Multiple worker threads are required for connection scaling (was <Bug 21417505> - rebooting of ib switch causing local zones going to shutting down state) was addressed in Solaris 11.3 SRU7 (in the July 2016 QFSDP) and the safe RDS connection limit
has been further enhanced in Solaris 11.3 SRU11 with further enhancements due in Solaris 11.3 SRU14.

Safe RDS Connection Limits:

• 200 for systems below Solaris 11.3 SRU7
• 700 for systems running Solaris 11.3 SRU7 (July 2016 QFSDP)
• 1,800 for systems running Solaris 11.3 SRU11 (Oct 2016 QFSDP)
• 3,400 for systems running Solaris 11.3 SRU14 (Jan 2017 QFSDP)

NOTE: 

If a customer system has already exceeded the limit of safe RDS connection limit, there are 2 choices (in the following order of preference):

1) Avoid Infiniband switch reboots
2) Disable the NRM / QOS feature on all 12c RAC nodes

Option 1 is preferable since it doesn't involve any disruptive changes that have to be enabled again once the bug is fixed (which might be forgotten or overlooked). However, it still leaves the customer vulnerable should an Infiniband switch fail. If the customer has experienced leaf switch failures or node evictions in the recent past, then consider implementing option 2.

Disabling NRM / QOS is the last choice and not generally recommended since it requires setting a hidden parameter. It requires both the compute nodes and the Exadata storage cells to be rebooted at the same time to get rid of existing RDS QOS connections, otherwise they'll persist.  20 minutes must be left between IB switch reboots.

1) Disable NRM / QOS on all 12 RAC nodes as follows:

References

This solution has no attachment