'srvctl start database' Fails With PRCR-1013 PRCR-1064 CRS-2680 CRS-5802

Asset ID:	1-72-2309987.1
Update Date:	2017-09-29
Keywords:

Solution Type Problem Resolution Sure

Solution 2309987.1 : 'srvctl start database' Fails With PRCR-1013 PRCR-1064 CRS-2680 CRS-5802

Applies to:

Exadata X3-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

SRVCTL cannot STOP or START database on one node after upgrading db from 11.2.0.4 to 12c.

SQLPLUS can start the database.

Status shows not running:

$ srvctl status database -d <node_name>
Instance <instancename> is not running on node node01
Instance <instancename> is running on node node02
Instance <instancename> is running on node node03

Attempts to start result in errors similar to the following:

$ srvctl start instance -d ints -i INTS1
PRCR-1013 : Failed to start resource ora.<database_name>.db
PRCR-1064 : Failed to start resource ora.<database_name>.db on node dbadm01
CRS-2680: Clean of 'ora.<database_name>.db' on 'dbadm01' failed
CRS-5802: Unable to start the agent process

CRS_alert.log will show errors like the following:

2017-09-13 11:09:05.223 [CRSD(280048)]CRS-5828: Could not start agent '/u01/app/12.1.0.2/grid/bin/oraagent'. Details at (:CRSAGF00126:) {1:19801:39244} in /u01/app/oracle/diag/crs/mydbadm01/crs/trace/crsd.trc.

2017-09-13 11:09:20.322 [CRSD(280048)]CRS-5828: Could not start agent '/u01/app/12.1.0.2/grid/bin/oraagent'. Details at (:CRSAGF00123:) {1:19801:39244} in /u01/app/oracle/diag/crs/mydbadm01/crs/trace/crsd.trc.

2017-09-13 11:09:20.322 [CRSD(280048)]CRS-5828: Could not start agent '/u01/app/12.1.0.2/grid/bin/oraagent'. Details at (:CRSAGF00126:) {1:19801:39244} in /u01/app/oracle/diag/crs/mydbadm01/crs/trace/crsd.trc.

2017-09-13 11:09:20.324 [CRSD(280048)]CRS-2758: Resource 'ora.hcmprl.db' is in an unknown state.

crsd.trc will show the following errors:

2017-09-13 11:09:20.322683 : AGFW:484337408: {1:19801:39244} Agfw Proxy Server received the message: RESOURCE_MODIFY_ATTR[ora.hcmprl.db 3 1] ID 4355:22953678
2017-09-13 11:09:20.322715 : AGFW:484337408: {1:19801:39244} Agfw Proxy Server rejecting message RESOURCE_MODIFY_ATTR[ora.hcmprl.db 3 1] ID 4355:22953678
2017-09-13 11:09:20.322722 : AGFW:484337408: {1:19801:39244} X_AGFW_RejectMsg : Could not find the resource: ora.hcmprl.db 3 1
(File: clsAgfwSrvResource.cpp, line: 1574

2017-09-13 11:09:20.323143 : CRSPE:473831168: {1:19801:39244} CRS-2678: 'ora.hcmprl.db' on 'mydbadm01' has experienced an unrecoverable failure

2017-09-13 11:09:20.323577 : CRSPE:473831168: {1:19801:39244} CRS-0267: Human intervention required to resume its availability.

2017-09-13 11:09:20.323607 :UiServer:4026529536: {1:19801:39244} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2678: 'ora.hcmprl.db' on 'mydbadm01' has experienced an unrecoverable failure]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.hcmprl.db 1 1]
WAIT:
TextMessage[0]
]
2017-09-13 11:09:20.324109 : CRSD:473831168: {1:19801:39244} {1:19801:39244} Resourceora.hcmprl.db has failed into unknown state!
2017-09-13 11:09:20.324125 :UiServer:4026529536: {1:19801:39244} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-0267: Human intervention required to resume its availability.]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.hcmprl.db 1 1]
WAIT:
TextMessage[0]
]
2017-09-13 11:09:20.324135 : CRSPE:473831168: {1:19801:39244} Sequencer for [ora.hcmprl.db 1 1] has completed with error: CRS-5802: Unable to start the agent process

2017-09-13 11:09:20.324224 : CRSPE:473831168: {1:19801:39244} Starting resource state restoration for: START of [ora.hcmprl.db 1 1] on [mydbadm01] : Op:0x7f1cac532a80, Cmd:0x7f1cac398750, SeqId:16940 restart: , state change= 0, local restart=0

Status of the database resource will show one or more of them as UNKNOWN status.

ora.<database_name>.db
1 ONLINE UNKNOWN dbadm01 STABLE
2 ONLINE ONLINE dbadm02 Open,STABLE
3 ONLINE ONLINE dbadm03 Open,STABLE

Changes

Customer upgraded database from 11.2.0.4 to 12c.

Cause

Able to start DB using SQL*Plus, but not from srvctl command. The issue was caused by incorrect file ownership after upgrade. The sticky bit on agent folder was missing:

Solution

Solution 1:

./srvctl stop instance -d racdb -i racdb1

./srvctl remove instance -d racdb -i racdb1

./srvctl add instance -d racdb -i racdb1 -n racnd01

./srvctl enable instance -d racdb -i racdb1

./srvctl start instance -d racdb -i racdb1

Solution 2:

Setting the sticky bit will fixed the problem:

$ chmod 1777 /407/apps/u01/oragi/11.2.0/grid/log/rac-node1/agent/crsd/oraagent_oracle

$ ls -ld /407/apps/u01/oragi/11.2.0/grid/log/rac-node1/agent/crsd/oraagent_oracle
drwxrwxrwt. 2 oracle oinstall 4096 Aug 22 10:52 /407/apps/u01/oragi/11.2.0/grid/log/rac-node1/agent/crsd/oraagent_oracle

If this fails to resolve the issue, please collect a TFA covering a two hour period around the problem (see Doc ID 1513912.1) then open a Service Request with Oracle Support.

Attachments

This solution has no attachment