Configuring Exadata Database Server Routing

Asset ID:	1-71-1306154.1
Update Date:	2018-02-16
Keywords:

Solution Type Technical Instruction Sure

Solution 1306154.1 : Configuring Exadata Database Server Routing

Applies to:

Oracle Exadata Hardware - Version 11.1.0.7 and later
Linux OS - Version Oracle Linux 5.8 to Oracle Linux 5.8 [Release OL5U8]
Exadata X3-2 Hardware
Linux x86-64

Goal

The procedure provided here will configure network routing on database servers in Exadata Database Machine so that packets arriving via a given interface will have their responses sent out using that same interface (instead of always sending via the default gateway).

Solution

Overview

For all Exadata Database Machine database servers that have fresh images starting with version 11.2.3.2.1 or later, the configuration contained in this note is already in place by default. The configuration notes here apply to database servers that were imaged prior to version 11.2.3.2.1. If the imagehistory command reports any version prior to 11.2.3.2.1, then the steps here may be applied. If all versions reported by the imagehistory command on your database server are 11.2.3.2.1 or later, the steps included in this note were applied at deployment time.

Note: Due to some changes to the Linux kernel, any system that was deployed prior to 11.2.3.2.1 should follow this note to add or update the files mentioned in this note. Failure to update these files may result in networking problems following an upgrade to any version 11.2.3.2.0 or later since that's when an updated kernel was included for database servers. To avoid the issue, update the rule* and route* files as described in this note.

Database servers are deployed with 3 logical network interfaces configured: management network (typically eth0), client access network (typically bond1 or bondeth0), and private network (typically bond0 or bondib0). The default route for the system typically uses the client access network and the gateway for that network. All outbound traffic that is not destined for an IP address on the management or private networks is sent out via the client access network. This poses a problem for some connections to the management network in some customer environments.

The configuration changes outlined in this note will modify that behavior so that any connections that come in via the management network interface will have their responses sent out via the management network interface. Similarly, packets arriving via the client access network will have responses sent out via the client access network. The private network needs no such arrangements since all network traffic on the private network is direct communication between the two endpionts without any routers involved. If the database server has additional auxiliary networks configured, additional files should be set up for the additional interface(s) as well.

Configuration for boot-time routing configuration

Several specially-configured files must exist in /etc/sysconfig/network-scripts on each database server in order to support the boot-time configuration of the modified routing tables.

For each Ethernet interface that have configured IP addresses (not for slaves) the database server must have route-ethX and rule-ethX files configured. Rule/route table number should start with 220 for eth0, 221 for eth1, etc.

Assume eth0 has IP address 10.149.49.12, netmask 255.255.252.0 (network 10.149.48.0/22) and gateway 10.149.48.1.

/etc/sysconfig/network-scripts/rule-eth0 contains
from 10.149.49.12 table 220
to 10.149.49.12 table 220

/etc/sysconfig/network-scripts/route-eth0 contains
10.149.48.0/22 dev eth0 table 220
default via 10.149.48.1 dev eth0 table 220

For each bond over Ethernet interface, the database server must have route-bondethX and rule-bondethX. Rule/route table number should start with 210 for bondeth0, 211 for bondeth1, etc.

Assume bondeth0 has IP address 10.204.78.15, netmask 255.255.255.0 (network 10.204.78.0/24) and gateway 10.1.78.1.

/etc/sysconfig/network-scripts/rule-bondeth0 contains
from 10.204.78.0/24 table 210
to 10.204.78.0/24 table 210

/etc/sysconfig/network-scripts/route-bondeth0 contains
10.204.78.0/24 dev bondeth0 table 210
default via 10.204.78.1 dev bondeth0 table 210

For bond over IB interfaces, the route and rule files are already there for X2-8 database servers (which have multiple IB bonded interfaces). X2-2 database servers should not have ip route and rule files for their single bondib0 interface.

NOTE: Even if the following section is used to configure the routing in real-time, it is highly recommended to reboot the database server to validate that the boot-time configuration will result in the desired configuration.

Configuration for real-time routing configuration

To configure the rules on the running system, the /sbin/ip command is used to create the same configuration that will be performed at boot-time. The following commands will result in the same configuration as outlined in the examples above.

/sbin/ip rule add from 10.149.49.12 table 220
/sbin/ip rule add to 10.149.49.12 table 220
/sbin/ip route add 10.149.48.0/22 dev eth0 table 220
/sbin/ip route add default via 10.149.48.1 dev eth0 table 220

/sbin/ip rule add from 10.204.78.0/24 table 210
/sbin/ip rule add to 10.204.78.0/24 table 210
/sbin/ip route add 10.204.78.0/24 dev bondeth0 table 210
/sbin/ip route add default via 10.204.78.1 dev bondeth0 table 210

NOTE: Even if the preceding commands are used to configure the routing in real-time, it is highly recommended to reboot the database server to validate that the boot-time configuration will result in the desired configuration.

Removing real-time configuration (for troubleshooting purposes)

While performing configuration or troubleshooting, you may need to remove the rules and routes. The following commands are for removing rules and routes (ordered in reverse intentionally).

/sbin/ip route del default via 10.149.48.1 dev eth0 table 220
/sbin/ip route del 10.149.48.0/22 dev eth0 table 220
/sbin/ip rule del to 10.149.49.12 table 220
/sbin/ip rule del from 10.149.49.12 table 220

/sbin/ip route del default via 10.204.78.1 dev bondeth0 table 210
/sbin/ip route del 10.204.78.0/24 dev bondeth0 table 210
/sbin/ip rule del to 10.204.78.0/24 table 210
/sbin/ip rule del from 10.204.78.0/24 table 210

Verification and status commands

To verify the routes and rules, use these commands.

Checking rules

The rules can be checked via the same /sbin/ip command using the "rule list" argument.

root@dbhost# ip rule list
0: from all lookup 255
32762: from all to 10.204.78.0/24 lookup 210
32763: from 10.204.78.0/24 lookup 210
32764: from all to 10.149.49.12 lookup 220
32765: from 10.149.49.12 lookup 220
32766: from all lookup main
32767: from all lookup default
root@dbhost#

This command output shows all rules on the system.

Checking routes

In the configuration outlined in this note, we leave the default routing table as it is and add two new routing tables. The two new routing tables are used when the rules dictate their use. The default route table has no indication that two alternate tables exist. The only indication that alternate routing tables are in use is by reviewing the rules (shown above).

To see the default route table, use:

root@dbhost# ip route list
10.204.78.0/24 dev bondeth0 proto kernel scope link src 10.204.78.15
192.168.10.0/24 dev bondib0 proto kernel scope link src 192.168.10.8
10.149.48.0/22 dev eth0 proto kernel scope link src 10.149.49.12
default via 10.149.52.1 dev bondeth0
root@dbhost#

To see the supplemental routing tables identified by the rules in the "ip rule list" output earlier, append the table name to the end of the "ip route list" command, like this:

root@dbhost# ip route list table 220
10.149.48.0/22 dev eth0 scope link
default via 10.149.48.1 dev eth0
root@dbhost# ip route list table 210
10.204.78.0/24 dev bondeth0 scope link
default via 10.204.78.1 dev bondeth0
root@dbhost#

Removal and backout

The best way to clean up and return to default behavior is to remove the files added to /etc/sysconfig/network-scripts and reboot the node.
/bin/rm -f /etc/sysconfig/network-scripts/rule-eth0
/bin/rm -f /etc/sysconfig/network-scripts/route-eth0
/bin/rm -f /etc/sysconfig/network-scripts/rule-bondeth0
/bin/rm -f /etc/sysconfig/network-scripts/route-bondeth0
reboot

References

<BUG:11725389> - TRACK112230: MARTIAN SOURCE REPORTED ON DB NODES BONDETH0 INTERFACE
<NOTE:1286796.1> - rp_filter for multiple private interconnects and Linux Kernel 2.6.32+
<BUG:14793169> - CHANGE CONTENT OF RULE-BONDETH0 FILE CREATED BY IPCONF
<BUG:17723513> - EXADATA - ROUTING ERRORS AFTER PLATINUM PATCHING FROM 11.2.3.1.0 TO 11.2.3.2.1

Attachments

This solution has no attachment