Exadata - ACFS State Is Offline On One Of The Nodes

Asset ID:	1-72-1684026.1
Update Date:	2014-08-13
Keywords:

Solution Type Problem Resolution Sure

Solution 1684026.1 : Exadata - ACFS State Is Offline On One Of The Nodes

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 to 11.2.0.4 [Release 11.2]
Exadata X3-2 Full Rack - Version All Versions to All Versions [Release All Releases]
Linux x86-64

Symptoms

On an Exadata Database Machine, you notice that the ACFS resource "ora.registry.acfs" is not coming up on one of your nodes after a reboot:

oracle:@exapdb01 ~ > crsstat acfs

Resource Name                            Resource Type Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
ora.registry.acfs                        registry       OFFLINE      OFFLINE
ora.registry.acfs                        registry       OFFLINE      OFFLINE

oracle:@exapdb01 ~ > crsctl start res ora.registry.acfs

CRS-2672: Attempting to start 'ora.registry.acfs' on 'exapdb02'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'exapdb01'
CRS-5016: Process "/u01/app/11.2.0.4/grid/bin/acfsregistrymount" spawned by agent "/u01/app/11.2.0.4/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.4/grid/log/exapdb02/agent/crsd/orarootagent_root/orarootagent_root.log"
CRS-2674: Start of 'ora.registry.acfs' on 'exapdb02' failed
CRS-2676: Start of 'ora.registry.acfs' on 'exapdb01' succeeded
CRS-4000: Command Start failed, or completed with errors.

oracle:@exapdb01 ~ > crsstat acfs

Resource Name                            Resource Type Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
ora.registry.acfs                        registry       ONLINE       ONLINE       exapdb01
ora.registry.acfs                        registry       ONLINE       OFFLINE

Cause

While ACFS is not supported before 12.1.0.2 on Exadata (see Can ACFS be installed on Exadata? - <Note:1326938.1>) it may have been installed by default. In this case the ACFS resource "ora.registry.acfs" will typically start and stop along with all other cluster resources.

It is possible that after a cluster reboot the ACFS resource may not start. This can happen when the service is stopped prior to rebooting. Upon reboot, the node will return to the same state leaving the ACFS resource stopped. If manually starting the ACFS resource does not work for each node as shown above, you can run into an issue where the ACFS driver may no longer be configured properly.

Solution

ACFS is not supported before 12.1.0.2 on Exadata storage.

There are two ways to resolve this issue:

(1) - Reconfigure the ACFS resource and then stop the resource on all nodes

    (a) Verfy that the ACFS Driver is installed and loaded using the following note:

       How to Verify that the ACFS Driver is installed and loaded - <Note:1319263.1>

    (b) Reconfigure the ACFS resource "ora.registry.acfs" using the following note:

   “ora.registry.acfs” ("ora.drivers.acfs") Resource Was Not Configured Therefore RAC ACFS Filesystem Is Not Mounting During The Reboot. - <Note:1486208.1>

oracle:@exapdb01 ~ > crsctl status res ora.registry.acfs -t

--------------------------------------------------------------------------------
NAME           TARGET STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.registry.acfs
               ONLINE ONLINE       exapdb01
               ONLINE ONLINE       exapdb02

oracle:@exapdb01 ~ > crsctl stop res ora.registry.acfs

CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exapdb01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exapdb02'
CRS-2677: Stop of 'ora.registry.acfs' on 'exapdb02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exapdb01' succeeded

oracle:@exapdb01 ~ > crsctl status res ora.registry.acfs -t

--------------------------------------------------------------------------------
NAME           TARGET STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.registry.acfs
               OFFLINE OFFLINE     exapdb01
               OFFLINE OFFLINE      exapdb02

(2) - Remove ACFS from your Exadata Database Machine database nodes (<Note 1326938.1>):

    (a) If you are on 11.2.0.2, use the method explained in:

         ACFS TAINTED KERNEL MESSAGES -- FAQ - <NOTE:1082851.1>

    (b) If you are on 11.2.0.3, use the method explained in:

         Oracle Restart ora.registry.acfs state is UNKNOWN, unable to stop/start HAS - <NOTE:1498114.1>

References

<NOTE:1326938.1> - Can ACFS be installed on Exadata ?
<NOTE:1319263.1> - How to Verify that the ACFS Driver is installed and loaded
<NOTE:1486208.1> - “ora.registry.acfs” ("ora.drivers.acfs") Resource Was Not Configured Therefore RAC ACFS Filesystem Is Not Mounting During The Reboot.

Attachments

This solution has no attachment