Oracle Server X5-2
 
Oracle Exadata Storage Server Software
 
Exadata X4-8 Hardware
 
Oracle Server X5-2L
 

PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
 

Introduction

In this Document

Goal

Solution

Introduction

Overview of the Elastic Configuration Process

Detailed Explanation of Steps

Detailed Example for a New Rack

Procedure if Re-imaging (rather than using applyConfig) for a New Rack

Adding an X5 Node into an Existing X5 Rack

Adding an X5 Node into a pre-X5 Older Rack

Scripts and Log Information

Applies to:

Goal

This document provides details on configuring Exadata DB nodes and cells using the elastic configuration process.

Solution

Introduction

This document provides an overview of the elastic configuration process. Elastic configuration is new in Exadata version 12.1.2.1.0, and replaces the older applyconfig in previous versions. Elastic configuration applies to all rack configurations; i.e. racks ordered with a standard number of DB nodes and cells (such as a traditional quarter or half rack,) as well as rack configurations that feature additional DB nodes and cells. This is now the standard methodology for all new deployments, and applies to X5, X5-2L and X4-8b (with X5 storage cells) servers. The same process is also used to add additional DB servers or cells to an existing configuration.

Overview of the Elastic Configuration Process

The elastic configuration process will allow initial IP addresses to be assigned to database servers and cells, regardless of the exact customer configuration ordered. The customer specific configuration can then be applied to the nodes.

Every Exadata system has a pre-defined method for the cabling of nodes to IB switch ports. Therefore, there is a fixed mapping from each node's location in the rack to the ports of the IB switches. Assuming the rack is always populated following this map, a node's rack unit location can be identified by querying the IB fabric to determine the IB switch port the node is connected to. Once found, that information is used to determine the rack unit location of the node. Utilizing this information, nodes can be allocated IP addresses based on their rack unit location; with nodes lower in the rack getting lower IP addresses.

Below is a detailed look at each of the steps. Specific detailed steps can be found in the “Detailed Steps” section further down.

Detailed Explanation of Steps

1. Elastic Self Re-IP
   
   
   • Nodes shipped from the factory will have an elastic config marker “/.elasticConfig” placed in the root (/) directory
     
     
   • As the nodes are powered up, the elastic configuration code finds the marker file, and performs the following actions –
     
     
     • Runs ibnetdiscover, and chooses an IB switch as the reference.
       
       
     • The IB switch port number is determined (for the first port on the first IB card)
       
       
     • An IP address in the range of 172.16.x.x is allocated on eth0 for each server in the rack
       
       
       • The IP address is determined by using the port # + 36. For example, if the port# returned is 8, then the IP address will be 172.16.2.44. (8 + 36)
       • IP conflicts are checked for using “arping.” If an IP address conflict is found, (i.e. the IP address is being used elsewhere on the customer network,) the code will add 36 to the last octet of the IP address, and try to reassign that value, and so on.
       • Addresses in the range of 172.16.2.x will be tried first. However, depending on the IP's in use, the IP assigned could be anywhere between 172.16.2.0 and 172.16.2.254
         
         
     • Once a good (non-duplicate) IP address is found –
       
       
       • Eth0 is configured with the new 172.16.x.x IP address
       • IP addresses are un-configured from all IB interfaces to avoid conflicts with any existing IB addresses already in use in the IB fabric
       • The IB node description is set to the format  –
         
         
         • “Node8  elasticNode 172.16.2.44, 172.16.2.44 ETH0”
           
           
     • Hostnames are assigned based on IB port number. For example, a host connected to IB switch port 8 will be designated "node8". This can also be seen in the node description of ibhosts.
       
       It is prudent at this point to log into the rack and issue an ibhosts command. All targetted nodes should be identified with elasticNode, and the description will show a 172.16.x.x IP address.
       
       At this point, the self re-IP of elastic configuration is complete. This will make it possible for OEDA to run ibhosts and get information for all elastic nodes in the rack, along with their IP addresses.
       
       
       
2. Applying the User Configuration
   
   
   • Before applying the user configuration, the switch to OVM (only for racks that will have virtualization,) and reclaimdisk steps must be completed. (additional details for these steps can be found in the more detailed procedures)
     
     
   • The “applyElasticConfig” script is shipped and installed as part of the OEDA software. This script in turn will call the elasticConfig utility located in the /opt/oracle.SupportTools/firstconf directory
     
     
   • When the OEDA “applyElasticConfig.sh” script is called, it will find the nodes to be reconfigured by running ibhosts,  and filtering the returned list on the node description of “elasticNode”
     
     
   • OEDA will then perform the following actions -
     
     
     • Create a preconf.csv file
       
       Based on the rack type, OEDA dynamically creates an Infiniband IB switch port to rack unit map file, (for each elastic node OEDA determines the rack unit for the node based on the IB switch port it is connected to.)
       
       OEDA can now generate the preconf.csv file, with the lower lines corresponding to the lower rack units, and typically the lower IP addresses from user input assigned to the lower rack units (users can override this in OEDA before generate step in OEDA).
       
       
     • Modify the “ibSwitchPortToRackUnitMap”  file by marking each rack unit line as "elastic". It is expected to be in the following format, and is based on the port numbers on the Infiniband switch:

• • 
    
    • Using the ETH0 IP from the ibhosts node description. SCP the input files generated previously to each of the elastic nodes in the rack
      
      
    • SCP the “ibSwitchPortToRackUnitMap”  file to each node
      
      
    • Call the elasticConfig.sh script on each node under /opt/oracle.SupportTools/firstconf,  and use the files generated previously as input.
      
      
  • ElasticConfig.sh on each node then performs the following actions –
    
    
    • For db servers, check that reclaimdisks completed and the system is now a single OS (Linux or OVS/DOM0).
      
      
    • Validates that the count of lines in the preconf file is same as the count of elastic nodes it sees in the ibhosts output
      
      
    • Sorts the nodes of one type (db and cells) in the preconf.csv by their rack unit location/order based on the IB switch port the node is connected to
      
      
    • Generates a new single line preconf.csv file only for the node it is running on
      
      
    • Calls ipconf to generate the new cell.conf and apply it. It will reboot the node for the final user configuration to take effect
      
      
      • ipconf -preconf <this node preconf.csv> [optional pkey.conf] -generateall <--- generates the new cell.conf
      • remove the /.elastiConfig
      • ipconf -force -newconf <new cell.conf> -reboot

Detailed Example for a New Rack

1. Obtain the latest OEDA
   
   
2. Run the OEDA configuration tool to generate configuration files
   
   
3. Apply power to all servers
   
   
1. DB nodes and cells will automatically assign to themselves unused IP addresses on eth0 in the 172.16 range
2. Node hostnames will be of the format "node2", "node 8" etc.
   
   
4. Connect to the first DB node by connecting a laptop to the ILOM Serial Management port (9600,8,N,1)
   
   
5. (first dbnode) run ibhosts to get list of eth0 172.16 IPs for all servers
   
        ibhosts output will look similar to the format below -

All nodes should appear in ibhosts with a node description as -

"hostname elasticNode IP address ETH0"

A further check can be made using "ip addr" to ensure a 172.16 IP address has been assigned to ETH0.

Note: ETH0 will be seen in the "ibhosts" output only during initial elastic configuration. Once the elastic configuration process has completed "HCA-1" will be seen.

6. (all dbnodes, only if OVM) switch_to_ovm.sh (systems will reboot)

After switching to OVM, the 172.16 IP addresses will be on vmeth0 (rather than ETH0 as before.)

7. (all dbnodes) reclaimdisks.sh -free -reclaim

       If OVM was chosen, this step will lock in the image as OVM, so going back and forth will not be possible

 
8. (first dbnode) copy OEDA config files to /root/install (not /u01, which will not exist in dom0, so need common location)
     use either USB flash drive or temporary network connection

9. (first dbnode) cd /root/install/linux-x64

10. (first dbnode) ./applyElasticConfig.sh -cf customer_name-configFile.xml (systems will reboot as each node is completed.)

11. Connect cables for management and client networks

12. (all dbnodes and cells) reboot

13. (first dbnode) Run ibhosts and verify that all nodes show the correct IP addresses and hostnames. There should also be no
      nodes with "elasticNode" in the description.

14. (first dbnode) run OEDA installation tool to deploy

Procedure if Re-imaging (rather than using applyConfig) for a New Rack

1. Obtain the latest OEDA
   
   
2. Run the OEDA configuration tool to generate configuration files
   
   
3. Apply power to all servers
   
   
1. DB nodes and cells will automatically assign to themselves unused IP addresses on eth0 in the 172.16 range
2. Node hostnames will be of the format "node2", "node 8" etc.
   
   
4. Perform a PXE reimage of all the nodes to assign correct IP addresses. Refer to MOS Note 1959143.1 for addition information on image version 12.1.2.1.1 and above.
   
   
5. (all dbnodes, only if OVM) switch_to_ovm.sh (systems will reboot)
   
   
6. (all dbnodes) reclaimdisks.sh -free -reclaim
   
   
7. (new dbnodes) Run ibhosts and verify that all nodes show the correct IP addresses and hostnames. There should also be no
    nodes with "elasticNode" in the description.
   
   
8. (new dbnodes) run OEDA installation tool to deploy
   
   

Adding an X5 Node into an Existing X5 Rack

1. Obtain the latest OEDA
   
   
2. Run the OEDA configuration tool to generate configuration files for the servers to be added
   
   
3. Apply power to the new servers(s)
   
   
4. Connect to one of the new nodes. Either an ssh connection from an existing node in the rack (a 172 IP address can be brought up on an existing node,) or by connecting a laptop to the ILOM Serial Management port (9600,8,N,1)
   
   
5. (One a new node) run ibhosts to get list of the ETH0 172.16 IPs for all new servers
   
   
6. (All new DB nodes, only if OVM) switch_to_ovm.sh (systems will reboot)
   
   
7. (All new DB nodes) reclaimdisks.sh -free -reclaim
   
   
8. (first new dbnode or cell) copy OEDA config files to /root/install (not /u01, which will not exist in dom0, so need common location)
    use either USB flash drive or temporary network connection
   
   
9. (first new dbnode or cell) cd /root/install/linux-x64
   
   
10. (first dbnode) ./applyElasticConfig.sh -cf customer_name-configFile.xml (systems will reboot as each node is completed.)
    
    ** The customer_name-configFile.xml in this case should contain the original nodes, plus the new nodes to be added
    
    
11. Connect cables for management and client networks
    
    
12. (new dbnodes and cells) reboot
    
    
13. (first new dbnode) Run ibhosts and verify that all nodes show the correct IP addresses and hostnames. There should also be no nodes with "elasticNode" in the description.
    
    

Adding an X5 Node into a pre-X5 Older Rack

The minimum software version for an X5 server is 12.1.2.1.0. The elastic configuration procedure outlined above was not designed specifically to add X5's into older racks. Alternative methods such as reimaging the servers with a preconf file populated with MAC addresses is the suggested method for this. ElasticConfig may work however as long as the port mappings are the same.

Scripts and Log Information

The ./applyElasticConfig.sh script bundled with OEDA calls "/opt/oracle.SupportTools/firstconf/elasticConfig.sh"

Log file information is available under the OEDA directory at

Additional log file information is available at -

This solution has no attachment