Setup Listener on Infiniband Network using both SDP and TCP Protocol

Asset ID:	1-71-1580584.1
Update Date:	2016-08-08
Keywords:

Solution Type Technical Instruction Sure

Solution 1580584.1 : Setup Listener on Infiniband Network using both SDP and TCP Protocol

Applies to:

Exadata Database Machine V2 - Version All Versions and later
Oracle Exadata Hardware - Version 11.2.0.1 and later
Exadata Database Machine X2-2 Hardware - Version All Versions and later
Exadata Database Machine X2-8 - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
Generic Linux

Goal

Exadata Environment, though the same can be used for Non-Exadata Deployments that are using Infiniband
and the Linux platform supporting SDP protocol.

Configuring Sql*Net Listener using SDP/TCP protocol on Infiniband Network. If the intention is to make usage of SDP on various environments, i.e. multiple RACKS/Clusters be sure to implement this change on all nodes.

Please Note Oracle recommends the usage of TCP instead of SDP for Exalogic to Exadata connectivity.

Although the following instructions in this MOS note outline steps to deploy either protocol, it is highly recommended to use TCP for maximum stability.

The stability benefits provided by TCP outweigh any performance increase obtained from the use of SDP. For a variety of workloads in several scenarios, we have observed extremely good performance with TCP. In case of link failovers, TCP also resumes better than SDP due to less dependency on hardware layers.

Solution

Step 1. Enable SDP

Steps are to be performed on each of the DB compute nodes.
SDP protocol comes with RPM Package # libsdp
In my example test environment we are using libsdp-1.1.99-11.el5 which is provided with the 2.6.32-400.11.1.el5uek kernel image.

PLEASE NOTE
If you are on X4 or X5 Hardware utilizing active/active IB, then please be sure to apply fix for bug 17551223 and confirm with Oracle Support if need be. Alternatively the fix is included in 11.2.0.4 BP6. Reference --> Bug 17551223 - UNABLE TO START VIP ON IB NETWORK ON EXADATA X4-2

If APM is enabled at the kernel layer via modprobe and this is integrated with Exalogic please take steps to disable this as per -> SDP Connection in inter-connected Exalogic and Exadata stopped working (Doc ID 1588546.1)

a). Enable SDP Protocol

File name and location slightly differs between releases!
For 12.1.2.1.0 or higher the we make the below change in file# /etc/rdma/rdma.conf
For all other releases prior to 12.1.2.1.0 using OEL5 kernel we make the below changes to file# /etc/infiniband/openib.conf

change from
~~~~~~~~~~~~
# Load SDP module
SDP_LOAD=no

change too
~~~~~~~~~~~
# Load SDP module
SDP_LOAD=yes

For SOLARIS Operating System!
Note: There is no requirement to enable SDP on Solaris DB Nodes, it is enabled by default check with the following and then proceed to step #2

Ensure rds SMF Service is running

% svcs -a|grep rds

online Sep_15 svc:/system/rds:default

Else enable it with

% svcadm enable rds

b). Change loadable kernel module (LKM) and enable required drivers via modprobe!

For 12.1.2.1.0 and higher

Edit /etc/modprobe.d/exadata.conf
Add the following options at the bottom of the file ---->

options ib_sdp sdp_zcopy_thresh=0 sdp_apm_enable=0 recv_poll=0

For 11.2.3.3.X to 12.1.1.1.X

Edit /etc/modprobe.conf
Add the following options at the bottom of the file ---->

options ib_sdp sdp_zcopy_thresh=0 sdp_apm_enable=0 recv_poll=0

For earlier releases prior to 11.2.3.3.0

Edit /etc/modprobe.conf
Add the following options at the bottom of the file ---->

options ib_sdp sdp_zcopy_thresh=0 recv_poll=0

c). Ensure we have correct protocol settings, SDP should be loaded "On-Demand" and TCP should be used by default, Having both defined has been known to cause CPU Spikes.

File name and location slightly differs between releases!
For 12.1.2.1.0 or higher the we make the below change in file# /etc/libsdp.conf
For all other releases prior to 12.1.2.1.0 using OEL5 kernel we make the below changes to file# /etc/ofed/libsdp.conf

Change this to be as follows -->

# to exclude SDP add "use tcp" rules before these defaults.
use tcp server * *:*
use tcp client * *:*
#

d). Reboot the node for the changes to take effect

Shutdown Clusterware!

% $GRID_HOME/bin/crsctl stop crs
<wait until the above completes>
% sync
% reboot

Step 2. Modfiy /etc/hosts files for the Application VIP IB Hostnames.

Extremely important you use separated IP#'s to what is already currently used for the Private bondib0 network.
In my example we have a 2 node Cluster and I've created extra 2 IP#'s dedicated for IB listeners.

% tail -6 /etc/hosts

### Hosts for SDP Listener confiugured on IB Network ###
192.168.1.21 exadbmel01-ib.au.oracle.com exadbmel01-ib
192.168.1.22 exadbmel02-ib.au.oracle.com exadbmel02-ib

Step 3. Add IB network and listener resources to Clusterware

By default clusterware deployments will have resource ora.net1.network for which dependent SCAN/VIP Client Network uses. Following Steps will configure net2 resource group on the Infiniband Network Fabric instead of Client Network.

Please note the srvctl commands slightly differ if you are running Active/Standby IB in V2,X2,X3 versus Active/Active in X4 ensure you specify the correct command
relevant to the deployment your running!

<as root>

For V2,X2 & X3 Hardware Running Active/Standby IB Interfaces please run ->
% srvctl add network -k 2 -S 192.168.1.0/255.255.255.0/bondib0

For X4, X5 hardware using Active/Active IB Interfaces please run ->
% srvctl add network -k 2 -S 192.168.1.0/255.255.255.0/ib0\|ib1

% srvctl config network -k 2
Network exists: 2/192.168.1.0/255.255.255.0/bondib0, type static

% crsctl start res ora.net2.network

For V2,X2 & X3 Hardware Running Active/Standby IB Interfaces please run ->
% srvctl add vip -n exadbmel01 -A exadbmel01-ib/255.255.255.0/bondib0 -k 2
% srvctl add vip -n exadbmel02 -A exadbmel02-ib/255.255.255.0/bondib0 -k 2

For X4, X5 hardware using Active/Active IB Interfaces please run ->
% srvctl add vip -n dm01db01 -A exadbmel01-ib/255.255.255.0/ib0\|ib1 -k 2
% srvctl add vip -n dm01db02 -A exadbmel02-ib/255.255.255.0/ib0\|ib1 -k 2

% srvctl start vip -i exadbmel01-ib
% srvctl start vip -i exadbmel02-ib

<as grid owner>
% srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522
% srvctl start listener -l LISTENER_IB

% srvctl status listener -l LISTENER_IB
Listener LISTENER_IB is enabled
Listener LISTENER_IB is running on node(s): exadbmel01,exadbmel02

Step 4. Configure Sql*Net Connect Strings

Introduction of additional clusterware networks along with RDBMS Parameter listener_networks in 11.2 allow you to define both the currently used SCAN/VIP Client network on your public network for both local and remote registration along with an additional network (net2) to be used for SDP Listener over IB Communication for example.

Local and Remote Listener Changes Needed. This is just an example for the changes on 2 nodes you will need to make changes for additional Nodes in the cluster.

E.g. of TNSNAMES.ORA Changes

LISTENER_IBREMOTE_VIP1 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = exadbmel02-ib.au.oracle.com)(PORT = 1522))
   ))
LISTENER_IBREMOTE_VIP2 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = exadbmel01-ib.au.oracle.com)(PORT = 1522))
   ))
LISTENER_IBLOCAL_VIP1 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel01-ib.au.oracle.com)(PORT = 1522))
))
LISTENER_IBLOCAL_VIP2 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel02-ib.au.oracle.com)(PORT = 1522))
))
LISTENER_IPLOCAL_VIP1 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = exa01-vip.au.oracle.com)(PORT = 1521))
))
LISTENER_IPLOCAL_VIP2 =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = exa02-vip.au.oracle.com)(PORT = 1521))
))
LISTENER_IPREMOTE =
(DESCRIPTION =
        (ADDRESS_LIST =
                (ADDRESS = (PROTOCOL = TCP)(HOST = scan-exa.au.oracle.com)(PORT = 1521))
))

SDP/IB Connect String for actual database

AMIB =
(DESCRIPTION =
        (LOAD_BALANCE=on)
        (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel01-ib)(PORT = 1522))
        (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel02-ib)(PORT = 1522))
        (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel03-ib)(PORT = 1522))
        (ADDRESS = (PROTOCOL = SDP)(HOST = exadbmel04-ib)(PORT = 1522))

        (CONNECT_DATA =
                (SERVER = DEDICATED)
                (SERVICE_NAME = AM)
        ))

Step 5. Modify RDBMS Listener_Networks

Please note I'm only showing the example for Instance 1 and 2, you will need to do the same on remaining instances with required adjustments

SQL> alter system set listener_networks=
2 '((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL_VIP1)
3 (REMOTE_LISTENER=LISTENER_IBREMOTE_VIP1))',
4 '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL_VIP1)
5 (REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both sid='AM1';

System altered.

SQL> alter system set listener_networks=
2 '((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL_VIP2)
3 (REMOTE_LISTENER=LISTENER_IBREMOTE_VIP2))',
4 '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL_VIP2)
5 (REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both sid='AM2';

System altered.

Step 6. Finally Restart Listener

% srvctl stop listener -l LISTENER_IB
% srvctl start listener -l LISTENER_IB

References

<BUG:17551223> - UNABLE TO START VIP ON IB NETWORK ON EXADATA X4-2
<BUG:19168690> - IB NETWORK AND VIP WORK NOT FINE AFTER PULL-OUT ONE OF THE IB PLUG
<BUG:18962892> - VIP ON ACTIVE/ACTIVE IB DOES NOT FAILBACK AFTER NETWORK RESTARTED.
<NOTE:1588546.1> - SDP Connection in inter-connected Exalogic and Exadata stopped working
<NOTE:1063571.1> - How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure
<NOTE:1548030.1> - SDP Connection From Application Server hosted on Exalogic To an Exadata System Does Not Respond
<BUG:13578047> - SDP STOPPED WORKING - NODE REBOOT RECOVERED THE ISSUE

Attachments

This solution has no attachment