Exadata: ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process when logging into database

Asset ID:	1-72-1555516.1
Update Date:	2013-06-17
Keywords:

Solution Type Problem Resolution Sure

Solution 1555516.1 : Exadata: ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process when logging into database

Applies to:

Exadata Database Machine X2-2 Half Rack - Version All Versions and later
Information in this document applies to any platform.

Symptoms

After applying a Bundle Patch, any user not in the oinstall/dba group is unable to connect to the database.

ASM alert log shows:

ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 18662
Wed Feb 13 15:35:26 2013
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 19970
Wed Feb 13 15:36:29 2013
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 21395

Database alert log shows:

Wed Feb 13 11:00:09 2013
Errors in file /u01/app/oracle/diag/rdbms/cidcq1/cidcq13/trace/cidcq13_m000_11310.trc:
ORA-01114: IO error writing block to file (block # )
ORA-01114: IO error writing block to file 2 (block # 165782)
ORA-29701: unable to connect to Cluster Synchronization Service
ORA-29701: unable to connect to Cluster Synchronization Service

Cssd log shows:

2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodClsaAuthInit: failed on clsaauthstart ret clsaretOSD (8), endp 0x7f650ceed780 [0000000006feb853] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_dmq1db03_)(GIPCID=cd431c46-da13a126-8071))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_dmq1db03_)(GIPCID=da13a126-cd431c46-24005))', numPend 5, numReady 1, numDone 0, numDead 3, numTransfer 0, objFlags 0x0, pidPeer 24005, flags 0x603710, usrFlags 0x14000 }
2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodClsaAuthInit: slos op : mkdir
2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodClsaAuthInit: slos dep : Permission denied (13)
2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodClsaAuthInit: slos loc : authprep6
2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodClsaAuthInit: slos info: failed to make dir /u01/app/11.2.0.3/grid/auth/css/dmq1db03/A6108554
2013-02-13 10:22:48.917: [GIPCXCPT][1083693376] gipcmodMuxTransferAccept: internal accept request failed endp 0x8a6510, child 0x7f650ceed780, ret gipcretAuthFail (22)
2013-02-13 10:22:48.917: [ GIPCMUX][1083693376] gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0x8a6510

Changes

Applied 11.2.0.2 BP14.

Cause

The os user that is attempting to connect is not the ORACLE_HOME owner, or a member of the binary owned group (oinstall), and this user doesn't have write permission on $GRID_HOME/auth directory.

Solution

It was found that node 1 had the correct write directory permissions for $GRID_HOME/auth, but the other nodes in the cluster did not.

Run the following command to change permission on the affected nodes:

chmod 775 <GRID_HOME>/auth

Attachments

This solution has no attachment