ORA-00600: [ipc_create_que_1] in Exafusion enabled environment like database version 12.2

Asset ID:	1-72-2282831.1
Update Date:	2018-04-10
Keywords:

Solution Type Problem Resolution Sure

Solution 2282831.1 : ORA-00600: [ipc_create_que_1] in Exafusion enabled environment like database version 12.2

Applies to:

Exadata Database Machine X2-8 - Version All Versions and later
Oracle Database - Enterprise Edition - Version 12.2.0.1 and later
Generic Linux
Applicable to software versions that support Exafusion feature
Oracle database version 12.1.0.2.0 BP11 or later

Symptoms

Database alert log reports following internal error, although the running RDBMS version is 12.2 where exafusion is enabled by default.

ORA-00600: internal error code, arguments: [ipc_create_que_1], [1], [8], [], [], [], [], [], [], [], [], []

Changes

Upgrade to RDBMS 12.2 or exafusion is enabled

Cause

Exafusion is a feature that allows database processes to read and send Oracle Real Applications Cluster (Oracle RAC) messages directly over the Infiniband network bypassing the overhead of entering the OS kernel, and running the normal networking software stack. The feature is available starting Oracle database version 12.1.0.2.0 BP11 or later and is enabled by default starting 12.2.

Typically error ORA-00600: [ipc_create_que_1] is received in a environment where Exafusion is enabled (via exafusion_enabled=1 or underscore parameter _exafusion_enabled) and the running database software or storage software does not support the feature.

However there are cases where the internal error can occur even on a supported version like 12.2 database. The corresponding trace file show error stack as:

IPCLW:[0.0]{E}[PINI]:IB: [0]failed to ibv_create cq - size 2048, errno 12
IPCLW:[0.1]{E}[PINI]:RC: [0]XRC RCV CQ creation failed for port 0x40001ddd22e0. <<<<
IPCLW:[0.2]{E}[PINI]:PUB: [0]Unable to create CREPLY port.
IPCLW:[0.3]{E}[PINI]:PUB: [0]Failed Pini at IP 192.168.10.1 cookie 70369245229856 rval 3
MQL:Deferred initialization for MQL Context failed
2017-06-16T09:30:23.133534+04:00
Incident 808255 created, dump file: /u01/app/oracle/diag/rdbms/xldr/xldr1/incident/incdir_808255/xldr1_ora_394763_i808255.trc
ORA-00600: internal error code, arguments: [ipc_create_que_1], [1], [8], [], [], [], [], [], [], [], [], []

[MEM]: Memory leak! 3152 bytes not deallocated in following chunks:
[MEM]: [0] Hdr: 0x40001ddd22b0, Ptr: 0x40001ddd22e0 Size: 3152, Type: [MTYP=PGA][ATTR=NONE][OBJT=0x03], Allocated @ ipclw_pub.c:7454
[MEM]: Memory leak! 3152 bytes not deallocated in following chunks:
[MEM]: [0] Hdr: 0x40001ddd22b0, Ptr: 0x40001ddd22e0 Size: 3152, Type: [MTYP=PGA][ATTR=NONE][OBJT=0x03], Allocated @ ipclw_pub.c:7454

Trace file reveals that error was triggered by OS user orarom (in this case, it may be other user as well). The resource limit section tells memlock setting for orarom is only 64 KB. It appears that OS user orarom is logging into database with insufficient OS resource limits set.

SO: 0x2b1438458, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x2b1438458, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
(process) Oracle pid:119, ser:108, calls cur/top: (nil)/0x200757908

...
Process Group: DEFAULT, pseudo proc: 0x2d1663bf8
O/S info: user: orarom, term: UNKNOWN, ospid: 210874 <<<<
OSD pid info:
KGL-UOL (Process state object)

...

----- Process Resource Limits -----
***************** Dumping Resource Limits(s/h) *****************
core file size 0 KB/0 KB
data seg size UNLIMITED/UNLIMITED
scheduling priority 0 KB/0 KB
file size UNLIMITED/UNLIMITED
pending signals 1005 KB/1005 KB
max locked memory 64 KB/64 KB <<<<
max memory size UNLIMITED/UNLIMITED
open files 4 KB/4 KB
POSIX message queues 800 KB/800 KB
real-time priority 0 KB/0 KB
stack size 32 MB/UNLIMITED
cpu time UNLIMITED/UNLIMITED
max user processes 1005 KB/1005 KB
virtual memory UNLIMITED/UNLIMITED
file locks UNLIMITED/UNLIMITED

Solution

Add user orarom to /etc/security/limits.conf file with appropriate memlock value. May be existing OS user oracle or grid can be taken as reference.

It is required to restart EM agent, if the orarom user is used by EM.

Note: We have come across few cases where even after increasing the limits, the error ipc_create_que_1 repeats. During such scenarios, we want customers to collect following details, create an SR with Oracle Support Exadata to troubleshoot further. Disabling Exafusion can get rid of the error but you will be losing the benefit of the feature.

- Contents of /etc/pam.d/sshd and change history (by change history we mean, if the file was modified/changed ever and what changes were made)
- Contents of /etc/security/limits.conf (and change history)
- Contents of /etc/bashrc (and change history)
- Contents of ~oracle/.bashrc , and all other files called from .bashrc
- Copy and paste the following lines in a file called list.sh and provide its output:

for PID in `grep 'Max locked' /proc/*/limits | grep -v unlimited | awk -F/ '{print $3}'`
do
   echo ========
   echo $PID
   grep 'Max locked' /proc/$PID/limits
   cat /proc/$PID/comm
   cat /proc/$PID/cmdline
   echo
done

- OS/kernel version of the client host (if clients are remote, or if clients are ssh'ing to the DB host to run sqlplus)
- How the clients connect to the database host and database (exact command lines, including what files are being sourced other than the regular .bashrc)
- Login shell info, whether the customer uses the default bash or something else (csh etc), and what files are sourced (.rc files), or whether any limits changes are manually made in a shell script, etc.

As updated by Atsushi Morimura from dev, if customer comes back with the above information in an SR, Oracle Support engineer can then file a bug to investigate further

References

<NOTE:2037900.1> - EXAFUSION details and supported versions
<BUG:25839547> - ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [IPC_CREATE_QUE_1], [1], [8]

Attachments

This solution has no attachment