Exalogic Elastic Cloud X4-2 Eighth Rack
 
Oracle Exalogic Elastic Cloud Software
 

PLA-Support>Eng Systems>Exalogic/OVCA>Oracle Exalogic>MW: Exalogic Core
 

In this Document

Symptoms

Cause

Solution

Created from <SR 3-6507832511>

Applies to:

Symptoms

You have an Exalogic system where information about nodes added to Subnet Manager partitions, which is expected to be identical across all the switches in the Exalogic rack (all NM2-GW gateway switches and any NM2-36P spine switch that is present within the rack configuration), appears to be inconsistent between some switches as the result of not being propagated across all the switches. As a result, information about changes to partitions is not present in all switches which could give rise to unexpected behavior in the event that one of the switches, whose partition list is incomplete/out of sync, assumes the role of the MASTER Subnet Manager (as identified by the "getmaster" command available on switches).

To determine whether or not the partition information is consistent across all switches, run the "smpartition list active" command and compare the output. As the output is long and since the output of the  "smpartition list active" cannot be easily redirected to a file (it pauses the output and requires the space/return key to be pressed in a "more" like fashion) the following script can be created as checkPartitionConsistency.sh and run:

On its conclusion the script prints, for each switch, an "md5sum" value based on the content of the switch's active partition list and the md5sum value should be expected to be the same across all switches:

In the case the partition information is already found to be consistent across each switch then no further action is needed from this note and further analysis of the problem will be necessary.

Cause

However, in the event the md5sum values returned for any switch is inconsistent with other switches, this suggests:

1. There is an inconsistency in the IP address list reflected in the "smnodes list" output for one or more of the switches
   1. this can occur if the same set of "smnodes add" commands have not yet been run on every one of the switches
      
      
2. The difference is the result of an unexpected error encountered when an "smnodes add" command was run on one of the switches, such that the switch executing the command was unable to validate a Subnet Manager could be reached at the IP address provided.
   This can occur when:
   
   1. the Subnet Manager (opensm) and Infiniband Partition Daemon (partitiond) services are not running on the switch being added
   2. the switch to be added does not have a valid configuarion, for example /conf/configvalid does not contain the value 1
   3. the switch to be added is part of an extended Infiniband Fabric that includes multiple Engineered Systems (e.g. Exalogic/Exadata/SuperCluster) and the IP address provided could not be reached from the network environment of the switch executing the "smnodes add" command
      e.g. the IP address belongs to a different subnet that either may not be connected to or routable from the swith on which the "smnodes add" command is being run.

Solution

For every switch of each type present in your IB fabric, whether a gateway (NM2-GW) or spine switch (NM-36P):

1. Verify whether or not the Subnet Manager is running:
   
   # service opensmd status
   opensm (pid 32296) is running...
   
   # service partconfigd status
   partitiond-daemon is running
   
   
   
2. For any switch where the Subnet Manager is found not to be running, utilize the following steps to start it:
   
   
   1. Check that /conf/configvalidcontains the value 1 across all switches
      • if it does not, then use the following approach to reset it:
        
        # echo '1' > /conf/configvalid
        
        
        
   2. Start/re-start the Subnet Manager (opensm) and Infiniband Partition Daemon (partitiond) services, via the "enablesm" command:
      
      # enablesm
      Starting IB Subnet Manager.                                [  OK  ]
      Starting partitiond daemon.                                [  OK  ]
      
      
      
3. Confirm that the output from "smnodes list" on every switch, shows the IP address of each switch in the system
   
   # smnodes list
   10.141.135.37
   10.141.135.38
   10.141.135.40
   
   
   
4. In the case that "smnodes list" on a switch does not show the IP address of each switch in your Exalogic rack, please run additional "smnodes add" commands to add the IP address of each of the missing switches to the list originally returned by that switch:
   
   # smnodes add 10.141.135.39
   
    
   
   
5. In the event that you receive an error, such as:
   
   # smnodes add 10.141.135.39
   Could not communicate with 10.141.135.39. Node not added
    
   Then:
   
   
   • Ensure the IP address is reachable from the current switch (# ping <Switch IP>)
   • Connect to the targeted switch via ssh and:
     
     • ensure the Subnet Manager is running (enablesm)
     • confirm that /conf/configvalid is 1 (echo "1" > /conf /configvalid)
       
       

   Note that, beyond the above suggestions, additional problems may be present that prevent the "smnodes add" command from completing as expected for a given switch and that the resolution of such problems may vary and therefore falls outside of the scope of this note. However, to ensure the "smnodes list" output is consistent between nodes as required, it will be necessary to further troubleshoot and resolve these problems until the same set of "smnodes add" commands have been run successfully against each of the switches expected to be present in the "smnodes list" output.

This solution has no attachment