ODA Instance Not Available After Enabling IPTABLES

Asset ID:	1-72-2081287.1
Update Date:	2015-11-30
Keywords:

Solution Type Problem Resolution Sure

Solution 2081287.1 : ODA Instance Not Available After Enabling IPTABLES

Applies to:

Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

After starting IPTABLES (i.e. firewall) on Node 0 of the rack, none of the RAC/CRS services on that node are working.
After trying to disable IPTABLES RAC/CRS, one node is still having problems with service startup.
After rebooting the node, the IPTABLES were still enabled and CRS still would not start.
Local connections to the instance on node 0 works, but client connections are not available.
LSNR service appears to be not working.

CRS / CLUSTER

[grid@oda-01 ~]$ crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[grid@oda-02 ~]$ crsctl check cluster -all
**************************************************************
oda-01:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oda-02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

LISTENER

[grid@oda-01 ~]$ lsnrctl

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 20-NOV-2015 10:00:00
Copyright (c) 1991, 2014, Oracle. All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused

LSNRCTL> stop
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
startTNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused

LSNRCTL>
Starting /u01/app/12.1.0.2/grid/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/app/12.1.0.2/grid/network/admin/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/ooda-01/listener/alert/log.xml
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12555: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00525: Insufficient privilege for operation
Linux Error: 1: Operation not permitted

LISTENER.LOG

<msg time='2015-11-20T10:00:05 ' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='oda-01'
host_addr='10.210.8.20'>
<txt>Trace information written to /u01/app/grid/diag/tnslsnr/oda-01/listener/trace/ora_23811_140460045407648.trc
</txt>
</msg>

That file does not exist

[grid@oda-01 ~]$ cd /u01/app/grid/diag/tnslsnr/oda-01/listener/trace/

[grid@oda-01 trace]$ pwd
/u01/app/grid/diag/tnslsnr/oda-01/listener/trace

[grid@oda-01 trace]$ ls -l
total 45644

Changes

Starting IPTABLES on one or both of the ODA nodes.

Cause

After setting IPTABLES the clusterware is down in the node and the ora.LISTENER.lsnr will also stop.

The Oracle Database Appliance (ODA) is a two-node engineered system using an interconnect configured to use specific IP addresses. Using IPTABLEs on the engineered ODA interconnect is not supported. IPTABLES on RAC is not supported.

Solution

From <Document 554781.1> RAC instabilities due to firewall (netfilter/iptables) enabled on the cluster interconnect:

"...

To disable iptables on all run levels (for next boot):

# chkconfig iptables off
# chkconfig --list iptables

iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

To stop iptables on current running system:

# service iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]

If you still need the iptables to control other interfaces, you should flush all the rules:

# iptables --flush

and make sure you only create rules that do not affect the cluster interconnect interfaces, protocols and network address space.

..."

Please refer to <Document 554781.1> for more generic and detailed information regarding IPTABLES on RAC.

Do not introduce IPTABLES or Firewall for the RAC interconnect.
As an engineered system the ODA interconnect has hard coded IP addresses which should not be altered.

References

<NOTE:554781.1> - RAC instabilities due to firewall (netfilter/iptables) enabled on the cluster interconnect
<NOTE:369699.1> - Pre-11.2: Root.sh Unable To Start CRS On Second Node
<NOTE:981357.1> - 11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network

Attachments

This solution has no attachment