ODA Unable to start ASM or Unable to start GI, TFA Hang, ORA-09925:Linux-x86_64 Error: 28: No space left on device w/ Huge Number Of Files Created

Asset ID:	1-72-2295143.1
Update Date:	2017-10-10
Keywords:

Solution Type Problem Resolution Sure

Solution 2295143.1 : ODA Unable to start ASM or Unable to start GI, TFA Hang, ORA-09925:Linux-x86_64 Error: 28: No space left on device w/ Huge Number Of Files Created

Applies to:

Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Unable to start ASM or Unable to start GI
TFA or other data collection tools nearly hang or do not respond or return outputs
The node shows out of space messages or unable to write .AUD files
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Huge Number Of Files Created Under /u01/app/grid/diag/clients/user_grid/host_XX on one node. (Usually the oakd master node)

Here is one example GI cannot startup:

crsctl stat res -init

showing:

NAME=ora.crsd
TYPE=ora.crs.type
TARGET=ONLINE
STATE=INTERMEDIATE on oda-n2-base

Also ASM and storage showing online but we find no running ASM process

ps -ef|grep smon
< -- No ASM detected

Log into asm results in the following errors

"No space left on device"
"ORA-09925: Unable to create audit trail file":

$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Fri ...2017
Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925

$ df -k -- Confirms /u01 has 21% FREE

$ df -k

Filesystem      1K-blocks        Used       Available   Use%   Mounted on

/dev/xvda2 57060636    42932984      11222412 80%     /
     tmpfs      65879408             0      65879408      0%     /dev/shm
/dev/xvda1        462820         67036        371464    16%     /boot
/dev/xvdb1 95988492      71225684      19880120     79% << /u01

$ df -i -- Confirms that the we have used 100% of the Inodes

$ df -i

Filesystem    Inodes      IUsed     IFree   IUse% Mounted on

/dev/xvda2   3637248      61310    3575938     2%    /
     tmpfs 16469852          1   16469851     1%     /dev/shm
/dev/xvda1    121920         46     121874     1%    /boot
/dev/xvdb1   6111232    6111232          0   100% << /u01

Changes

Cause

Excessive file generation which are not monitored or purged can result in running out of Inodes.

On the ODA this can also be due to INTERNAL Bug 25614866 generating a high number traces in /u01

Once Inodes are 100% for /u01 we are unable to start ASM as it shows as

ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925

Solution

1) Confirm the directory running out of Inodes using $ df -i

2) Clear the excessive number of files

2081655.1 ODA (Oracle Database Appliance): Cleanup trace and log files Using purgeODALog Tool
1681062.1 How to clean up the old staged patches on ODA
2154918.1 ODA: "/opt" Filesystem is Running Out of Space 99% Full in ODA Configurations

3) confirm Inodes are available for the previous full directory

[root@oda-n2-base ~]# df -i

Filesystem      Inodes   IUsed   IFree    IUse% Mounted on

/dev/xvda2     3637248 61310 3575938     2% /
     tmpfs    16469852   1 16469851     1% /dev/shm
/dev/xvda1      121920      46    121874     1% /boot
/dev/xvdb1     6111232 5343835    767397    88% /u01     << down from 100%

4) Confirm that connections to ASM can work and that the GI can restart successfully.

References

<BUG:25614866> - HUGE NUMBER OF FILES CREATED UNDER /U01/APP/GRID/ORADIAG_GRID/DIAG/CLIENTS/USER
<BUG:21347974> - ODA: GRID INFRASTRUCTURE GENERATES MORE TRACE FILES
<NOTE:2081655.1> - ODA (Oracle Database Appliance): Cleanup trace and log files Using purgeODALog Tool
<BUG:25664107> - DIAGNOSTIC PATCH FOR BUG 25614866
<NOTE:1681062.1> - How to clean up the old staged patches on ODA
<NOTE:2154918.1> - ODA: "/opt" Filesystem is Running Out of Space 99% Full in ODA Configurations

Attachments

This solution has no attachment