ORA-27303: SKGXP IPC libraries must be the same version. [local: RDS,remote: UDP] on Exadata

Asset ID:	1-72-1574772.1
Update Date:	2013-08-08
Keywords:

Solution Type Problem Resolution Sure

Solution 1574772.1 : ORA-27303: SKGXP IPC libraries must be the same version. [local: RDS,remote: UDP] on Exadata

Applies to:

Exadata Database Machine X2-2 Full Rack - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

On both nodes the databases are not ale to come up simultaneously.

Node 1: alert_agilepr1.log
======================

Tue Aug 06 17:03:07 2013
Errors in file /u01/app/oraprod/diag/rdbms/agilepr/agilepr1/trace/agilepr1_dia0_66399.trc:
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:proto mismatch failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcon
ORA-27303: additional information: Protocol of this IPC does not match remote (169.254.133.88). SKGXP IPC libraries must be the same version. [local: RDS,remote: UDP]
Tue Aug 06 17:03:07 2013
Errors in file /u01/app/oraprod/diag/rdbms/agilepr/agilepr1/trace/agilepr1_diag_66389.trc:
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:proto mismatch failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcon
ORA-27303: additional information: Protocol of this IPC does not match remote (169.254.133.88). SKGXP IPC libraries must be the same version. [local: RDS,remote: UDP]
Tue Aug 06 17:03:08 2013

Node 2: alert_agilepr2.log
====================

Errors in file /u01/app/oraprod/diag/rdbms/agilepr/agilepr2/trace/agilepr2_lmon_129974.trc:
ORA-27550: Target ID protocol check failed. tid vers=1, type=1, remote instance number=1, local instance number=2
NOTE: initiating MARK startup
LMON (ospid: 129974): terminating the instance due to error 481
System state dump requested by (instance=2, osid=129974 (LMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oraprod/diag/rdbms/agilepr/agilepr2/trace/agilepr2_diag_129962.trc
Dumping diagnostic data in directory=[cdmp_20130806171052], requested by (instance=2, osid=129974 (LMON)), summary=[abnormal instance termination].
Instance terminated by LMON, pid = 129974

To check IPC communication between the nodes right now, use skgxpinfo to see how it is linked since version 11.2.0.3.0

dcli -g dbs_group -l oraprod /u01/app/oraprod/product/11.2.0.3/agile/bin/skgxpinfo

dbtddb01: rds
dbtddb02: udp <==== node 2 having UDP usage

Running below command will double check the Exadata node where IPC library is linked with UDP usage:

dcli -g dbs_group -l oraprod 'nm /u01/app/oraprod/product/11.2.0.3/agile/lib/libskgxp11.so | grep rds_enabled'

dbtddb01: 00000000001df7f0 b skgxp_rds_enabled

Changes

Executed relinking using the below on DB home earlier and later started receiving the warning "WARNING: Oracle executable binary mismatch detected."

Running this might broke the RDS link

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk rac_on ioracle

Cause

RAC instances can’t see communicate with each other.

IPC error is due to mismatch in cluster interconnect protocol being in use by these 2 nodes. Node2 used UDP and Node1 uses RDS now. So due to this mismatch the other node is down.

Exadata does not support mixing of UDP and RDS ports.

Solution

- Shutdown all the instances running on this ORACLE_HOME which was configured for the instance having UDP port
- Execute as oracle user,

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk ipc_rds ioracle

- Startup the instance using srvctl

References

<NOTE:220970.1> - RAC: Frequently Asked Questions
<NOTE:751343.1> - RAC Support for RDS Over Infiniband

Attachments

This solution has no attachment