ODA : odacli Database Service Creation Fail Due to the ora.storage Resource is Offline

Asset ID:	1-72-2293172.1
Update Date:	2017-08-06
Keywords:

Solution Type Problem Resolution Sure

Solution 2293172.1 : ODA : odacli Database Service Creation Fail Due to the ora.storage Resource is Offline

Applies to:

Oracle Database Appliance X6-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Issue Summary : odacli database creation failed.

Please check the below SR Customer had confirmed solution worked :

SR 3-15456985741: ODA: Database service creation: Fail to run command Failed to start the filesystem

[root@ora2 log]# odacli describe-job -i b7c16b0d-a856-4934-95f2-b3cd26d057d8

Job details
----------------------------------------------------------------
ID: b7c16b0d-a856-4934-95f2-b3cd26d057d8
Description: Database service creation with db name: ifsuat
Status: Failure
Created: August 1, 2017 11:11:54 AM CEST
Message: Fail to run command Failed to start the filesystem on datifsuat

Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database Service creation August 1, 2017 11:11:54 AM CEST August 1, 2017 11:12:09 AM CEST Failure
Database Service creation August 1, 2017 11:11:54 AM CEST August 1, 2017 11:12:09 AM CEST Failure
Setting up ssh equivalance August 1, 2017 11:11:54 AM CEST August 1, 2017 11:11:54 AM CEST Success
Creating volume datifsuat August 1, 2017 11:11:54 AM CEST August 1, 2017 11:12:04 AM CEST Success
Creating ACFS filesystem for DATA August 1, 2017 11:12:04 AM CEST August 1, 2017 11:12:09 AM CEST Failure
Starting FileSystem August 1, 2017 11:12:07 AM CEST August 1, 2017 11:12:09 AM CEST Failure

Filename :

/opt/oracle/dcs/log/dcs-agent.log

2017-08-01 09:11:53,083 DEBUG [dw-100721 - POST /databases] [] c.o.d.a.r.s.s.DiskGroupUtils: DiskGroupUtils : isDiskGroupExists
2017-08-01 09:11:53,083 DEBUG [dw-100721 - POST /databases] [] c.o.d.c.u.CommonsUtils: run: cmd= '[/u01/app/12.1.0.2/grid/bin/crsctl, check, cluster]'

2017-08-01 09:12:09,036 DEBUG [Starting FileSystem] [] c.o.d.c.u.CommonsUtils: Output : PRCA-1089 : Unable to retrieve volume and disk group for volume device path /dev/asm/datifsuat-31 PRCA-1051 : No volume resource registered with CRS for specified diskgroup or volume name or volume device
2017-08-01 09:12:09,036 WARN [Starting FileSystem] [] c.o.d.a.r.s.s.FileSystemUtils: Failed to add filesystem resource to the clsuterware
2017-08-01 09:12:09,036 ERROR [Starting FileSystem] [] c.o.d.a.r.s.s.FileSystemActions: Fail to run command Failed to start the filesystem on datifsuat
2017-08-01 09:12:09,037 ERROR [Starting FileSystem] [] c.o.d.c.j.JsonRequestProcessor: RPC request invocation failed on request: {"classz":"com.oracle.dcs.agent.rpc.service.storage.FileSystemActions","method":"startFileSystem","params":[{"type":"com.oracle.dcs.agent.rest.storage.FileSystem","value":{"updatedTime":null,"id":1501578714266,"name":"datifsuat","description":null,"createTimeStamp":null,"type":"FileSys","devName":"datifsuat","fsType":"acfs","fsUser":"oracle","mountPoint":"/u02/app/oracle/oradata/ifsuat","diskGroupName":"DATA"}}],"revertable":false,"threadId":100725}
! java.lang.RuntimeException: Fail to run command Failed to start the filesystem on datifsuat
! at com.oracle.dcs.agent.rpc.service.storage.FileSystemActions.startFileSystem(FileSystemActions.java:35) ~[dcs-agent-12.1.2.10-SNAPSHOT.jar:jenkins-ol-dcs-agent-release-31]
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
! at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]

==>>> ASM instance alert logfile

SQL> ALTER DISKGROUP Data ADD VOLUME datifsuat SIZE 104857600K /* ASMCA */
Tue Aug 01 11:12:00 2017
ERROR: failed to create volume resource ora.DATA.DATIFSUAT.advm error <<<<<<<<<<<<<<<<<<<< Here is the Volume Name
Tue Aug 01 11:12:00 2017
SUCCESS: ALTER DISKGROUP Data ADD VOLUME datifsuat SIZE 104857600K /* ASMCA */
Tue Aug 01 11:12:00 2017
NOTE: volume resource ora.DATA.DATIFSUAT.advm is online

Cause

the root cause is the GI init resource ora.storage is offline.

[root@ora2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
......
ora.storage
1 ONLINE OFFLINE STABLE <<<<<<<

Solution

Apply below steps to restart the ora.storage resource.

1. stop the resource.

root#crsctl stop resource ora.storage -f -init
CRS-2673: Attempting to stop 'ora.storage' on 'odalarge1'
CRS-2677: Stop of 'ora.storage' on 'odalarge1' succeeded

2. start it

root#crsctl start resource ora.storage -f -init
CRS-2672: Attempting to start 'ora.storage' on 'odalarge1'
CRS-2676: Start of 'ora.storage' on 'odalarge1' succeeded

3. check the current status

root#crsctl stat res ora.storage -t - ini
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.storage
1 ONLINE ONLINE odalarge1 STABLE <<<<<<

Note : Please make sure target and state are online.

Please remove the volume once the ora.storage is online and retry the database creation.

Attachments

This solution has no attachment