| Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback |
| Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition | ||
|
|
||
 |
||||||||||||||||||||||||||||||||||||||||||||||||||
Solution Type Predictive Self-Healing Sure Solution 1682501.1 : Setting up the Subnet Manager in a multi-rack cabling configuration containing Exalogic/Big Data Appliance and Exadata/SuperCluster
In this Document
Applies to:Exalogic Elastic Cloud X3-2 Hardware - Version X3 to X3 [Release X3]Oracle Exadata Hardware Oracle Exalogic Elastic Cloud Software - Version 2.0.6.2.4 to 2.0.6.2.4 Exadata X4-2 Quarter Rack Oracle Exalogic Infrastructure Information in this document applies to any platform. PurposeWe are aligning the Subnet Manager configuration for multi-racked Engineered Systems that contain Exalogic (EL)/Big Data Appliance (BDA) and Exadata (ED)/SuperCluster (SC). These new requirements will allow for the Subnet Manager to fail over and automatically recover in the case of an EL/BDA rack level failure. ScopeThe scope of this document is for all multi-rack configurations that contain both Exalogic (EL)/Big Data Appliance (BDA) and Exadata (ED)/SuperCluster (SC). New systems should be configured per these requirements. Existing configurations should be changed to meet these requirements. DetailsThe basic guidance is to run the (Master) Subnet Manager on the Sun Network QDR InfiniBand Gateway Switches (aka NM2-GW switches) in multi-rack configurations containing both Exalogic (EL)/Big Data Appliance (BDA) and Exadata (ED)/SuperCluster (SC). Specifically, if an Exalogic virtual is included in a multi-rack cabling, then it is required that the Master Subnet Manager runs on one of the Exalogic NM2-GW switches. If the multi-rack cabling does not include an Exalogic virtual, then additional Subnet Manager configurations might be possible. For specific use cases please check with Oracle InfiniBand Support team. Below is a chart showing these recommendations:
The process for changing the configuration on the NM2-36p switches is below. On a pair of NM2-36P switches located in one of the Exadata (ED)/SuperCluster (SC) attached to the Exalogic (EL)/Big Data Appliance (BDA): 1) Disable the Subnet Manager: disablesm 2) Set the Subnet Manager priority to 2: setsmpriority 2 3) Set the controlled handover to false: setcontrolledhandover FALSE 4) Configure the smnodes list The smnodes list needs to contain the IP addresses of all switches which have Subnet Manager enabled so that partition configuration can be synchronized across all these switches. Ensure that all switches with the Subnet Manager enabled appear in the smnodes list output by running the following command on all the Exalogic NM2-GW switches and Exadata NM2-36P leaf switches with the Subnet Manager enabled: smnodes list If you have switch(es) that need to be removed run the command: smnodes delete x.x.x.x (where x.x.x.x is the IP address of the switch you want to remove from the smnodes file) en If you have switch(es) that you need to add run the command: smnodes add x.x.x.x (where x.x.x.x is the IP address of the switch you want to add to the smnodes file) Output should be the same on all switches eligible to run the Subnet Manager. Note: For additional guidance refer to the Multi-Rack Cabling EIS Checklist and to MOS Notes 1598479.1 and 2177177.1. 5) Enable the Subnet Manager: enablesm This will allow the Subnet Manager to failover to an ED/SC rack if the EL/BDA rack experiences a rack level failure and migrate the partition keys (pkeys) to the new rack. When the configuration from this MOS Note is properly implemented then Master Subnet Manager will relocate back to the EL/BDA rack if this rack becomes operational again - no need for manual action. This is already stated above. 6) Check the configuration: setsmpriority list This will tell you the current priority. This will also tell you whether or not controlled handover is set TRUE. Here is a sample configuration of a 4 rack setup containing 2 Exalogic Full Racks and 2 Exadata Full Racks: Exalogic 1 spine - smpriority = 1 controlledhandover = False subnet manager disabled GW1 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW2 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW3 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW4 - smpriority = 5 controlledhandover = TRUE subnet manager enabled Exalogic 2 spine - smpriority = 1 controlledhandover = False subnet manager disabled GW1 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW2 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW3 - smpriority = 5 controlledhandover = TRUE subnet manager enabled GW4 - smpriority = 5 controlledhandover = TRUE subnet manager enabled Exadata 1 spine - smpriority = 1 controlledhandover = False subnet manager disabled IBA - smpriority = 2 controlledhandover = FALSE subnet manager enabled IBB - smpriority = 2 controlledhandover = FALSE subnet manager enabled Exadata 2 spine - subnet manager disabled IBA - smpriority = 2 controlledhandover = FALSE subnet manager disabled IBB - smpriority = 2 controlledhandover = FALSE subnet manager disabled NOTE: [For the smnodes list to take effect on all four IB switches (2 - Exadata; 2 - Exalogic), had to run 'smpartition start;smpartition commit' on the master Exalogic GW switch.] References<BUG:17482244> - CANNOT ESTABLISH NEW CONNECTIONS UNTIL SM IS MANUALLY RESTARTEDAttachments This solution has no attachment |
||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||