Oracle Exalogic Elastic Cloud Software
 
Oracle Virtual Assembly Builder
 
Exalogic Elastic Cloud X3-2 Hardware
 

PLA-Support>Eng Systems>Exalogic/OVCA>Oracle Exalogic>MW: Exalogic Core
 

Configuring Exalogic N/W Routing to ensure packets arriving via a given interface will have their responses sent out using that same interface instead of always sending via the default gateway

In this Document

Symptoms

Cause

Solution

References

Created from <SR 3-5567966411>

Applies to:

Symptoms

When client is off subnet and connecting through management network (net0) and the default router is over bond1, the packets which are arriving via a given interface will have their responses sent out using the default gateway instead of that same interface where they are arriving from.

Cause

Exalogic Environment is by default configured with 3 logical network interfaces configured: management network (typically eth0), client access network (typically bond1), and private network (typically bond0). The default route for the system uses the client access network and the gateway for that network. All outbound traffic that is not destined for an IP address on the management or private networks is sent out via the client access network. This poses a problem for some connections to the management network in some Exalogic environments.

Due to this routing setup the ARP table entry for default does not get updated. Tracing the eth0 traffic shows that a ping from the windows client 10.180.xx.xx to the net0 management address 10.xx.xxx.128 will trigger an ARP REQUEST for the default router from bond1 with the source mac address of net0. This causes random connectivity failures for the client if the default gw ARP entry has timed out.

One of the situations, this behavior can be seen in a Exalogic environment is, if the eth0 (management) and bond1 (client) networks are tied on same physical LAN segment outside of Exalogic, then the compute node's/vServer ARP entries get mixed up.

This is not a mis-configuration or incorrect network setup , but depending the requirement, this type of network configuration might have been configured in this Exalogic environment.

Sample network tracing packet 520 below illustrates this problem:

This issue is caused due to the ARP network routing configuration in Exalogic environment,  which is making the network packets that are arriving via a given interface have their responses sent out using the default gateway instead of that same interface where they are arriving.

Solution

The configuration changes outlined in this note will modify that behavior so that any connections that come in via the management network interface will have their responses sent out via the management network interface. Similarly, packets arriving via the client access network will have responses sent out via the client access network. The private network needs no such arrangements since all network traffic on the private network is direct communication between the two end points without any routers involved.

The Below configuration changes need to be done on each and every Compute Node/vServer  in the Exalogic rack to achieve the modified behavior.

This below method of configuring rules and routes is referred as " iproute2" and is the best way to solve network routing issues in these situations.

Configuration for boot-time routing configuration:

Several specially-configured files must exist in /etc/sysconfig/network-scripts on each Exalogic Compute Node/vServer in order to support the boot-time configuration of the modified routing tables.

For each Ethernet interface that have configured IP addresses (not for slaves), each Exalogic Compute Node/vServer must have route-ethX and rule-ethX files configured. Rule/route table number should start with 220 for eth0, 221 for eth1, etc.

Assume eth0 has IP address 10.149.49.12, netmask 255.255.252.0 (network 10.149.48.0/22) and gateway 10.149.48.1.

For each bond over Ethernet interface, Exalogic compute node must have route-bondX and rule-bondX. Rule/route table number should start with 210 for bond1, 211 for bond2, etc.

Assume bond1 has IP address 10.204.78.15, netmask 255.255.255.0 (network 10.204.78.0/24) and gateway 10.204.78.1.

For bond over IB interfaces, the route and rule files are already there for Exalogic Compute Node/vServer, and in general should not have IP route and rule files for a single bond0 interface.

NOTE: Even if the following section is used to configure the routing in real-time, it is highly recommended to reboot the Exalogic Compute Nodes/vServers to validate that the boot-time configuration will result in the desired configuration.

Configuration for real-time routing configuration:

To configure the rules on the running system, the /sbin/ip command is used to create the same configuration that will be performed at boot-time. The following commands will result in the same configuration as outlined in the examples above.

NOTE: Even if the preceding commands are used to configure the routing in real-time, it is highly recommended to reboot the Exalogic Compute Nodes/vServers to validate that the boot-time configuration will result in the desired configuration.

Removing real-time configuration (for troubleshooting purposes):

While performing configuration or troubleshooting, if you need to remove the rules and routes, use the below details .The following commands are for removing rules and routes (ordered in reverse intentionally).

Verification and status commands:

To verify the routes and rules, use these commands.

Checking rules:

The rules can be checked via the same /sbin/ip command using the "rule list" argument. This command output shows all rules on the system.

 Checking routes:

In the configuration outlined in this note, we leave the default routing table as it is and add two new routing tables. The two new routing tables are used when the rules dictate their use. The default route table has no indication that two alternate tables exist. The only indication that alternate routing tables are in use is by reviewing the rules (shown above).

To see the default route table, use:

To see the supplemental routing tables identified by the rules in the "ip rule list" output earlier, append the table name to the end of the "ip route list" command, like this:

root@xxxxxx# ip route list table 220

10.149.48.0/22 dev eth0 scope link

default via 10.149.48.1 dev eth0

root@xxxxxx# ip route list table 210

10.204.78.0/24 dev bond1 scope link

default via 10.204.78.1 dev bond1

root@xxxxxx#

Removal and backout:

The best way to clean up and return to default behavior is to remove the files added to /etc/sysconfig/network-scripts and reboot the node.

References

This solution has no attachment