ODA: Unable to start Clusterware Listener on ODA Configurations due to PRCR-1079 CRS-5016 CRS-2674

Asset ID:	1-72-2135798.1
Update Date:	2016-05-09
Keywords:

Solution Type Problem Resolution Sure

Solution 2135798.1 : ODA: Unable to start Clusterware Listener on ODA Configurations due to PRCR-1079 CRS-5016 CRS-2674

Applies to:

Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.2 [Release 11.2 to 12.1]
Information in this document applies to any platform.

Symptoms

1) Unable to start Clusterware Listener on ODA Configurations due to the next errors:

grid@asmcloud58:~/ [grinf12102] srvctl start listener
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5016: Process "/u01/app/12.1.0.2/grid/bin/lsnrctl" spawned by agent "ORAAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/asmcloud58/crs/trace/crsd_oraagent_grid.trc"
CRS-5016: Process "/u01/app/12.1.0.2/grid/bin/lsnrctl" spawned by agent "ORAAGENT" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/diag/crs/asmcloud58/crs/trace/crsd_oraagent_grid.trc"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'asmcloud58' failed

2) The associated “ora.LISTENER.lsnr” resource is shown as OFFLINE in the second node:

ora.LISTENER.lsnr
ONLINE ONLINE asmcloud57 STABLE
ONLINE OFFLINE asmcloud58 STABLE

3) crsd_oraagent_grid.trc log reports:

.
.
.
2016-05-09 10:38:42.624557 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624566 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624572 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624582 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)Copyright (c) 1991, 2014, Oracle. All rights reserved.
2016-05-09 10:38:42.624589 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624597 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624603 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624612 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2016-05-09 10:38:42.624619 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624627 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)TNS-12541: TNS:no listener
2016-05-09 10:38:42.624634 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624641 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:) TNS-12560: TNS:protocol adapter error
2016-05-09 10:38:42.624648 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624656 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:) TNS-00511: No listener
2016-05-09 10:38:42.624663 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624670 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:) Linux Error: 111: Connection refused
2016-05-09 10:38:42.624677 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)
2016-05-09 10:38:42.624709 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1
2016-05-09 10:38:42.624829 :CLSDYNAM:3050223360: [ora.LISTENER.lsnr]{1:2027:18184} [check] execCmd ret = 1
2016-05-09 10:38:42.625127 : AGFW:3067033344: {1:2027:18184} ora.LISTENER.lsnr asmcloud58 1 state changed from: STARTING to: OFFLINE
2016-05-09 10:38:42.625365 : AGFW:3067033344: {1:2027:18184} Agent sending last reply for: RESOURCE_START[ora.LISTENER.lsnr asmcloud58 1] ID 4098:162417
.
.
.

Cause

Old associated sockets were not cleaned or released.

Solution

1) Shutdown CRS on the affected node as root user:

# crsctl stop crs

2) Remove the old sockets in the affected node as root user as follows:

# rm -rf /var/tmp/.oracle

3) Start CRS on the affected node as root user:

# crsctl start crs

4) The listener shows now the correct status:

ora.LISTENER.lsnr
ONLINE ONLINE asmcloud57 STABLE
ONLINE ONLINE asmcloud58 STABLE

5) Confirm the listener can be restarted as grid user as follows:

grid@asmcloud57:~/ srvctl stop listener
grid@asmcloud57:~/ srvctl start listener
grid@asmcloud57:~/ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): asmcloud58,asmcloud57

Community Discussions ODA

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

Attachments

This solution has no attachment