Unable To Login To Database ORA-12170: TNS:Connect timeout occurred and root(/) Shows 100% Space Used

Asset ID:	1-72-1918160.1
Update Date:	2016-12-06
Keywords:

Solution Type Problem Resolution Sure

Solution 1918160.1 : Unable To Login To Database ORA-12170: TNS:Connect timeout occurred and root(/) Shows 100% Space Used

Applies to:

Exadata X3-2 Quarter Rack - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.
***Checked for relevance on 09-Sep-2016***

Symptoms

Unable to login to database and it is failing with below error.

ORA-12170: TNS:Connect timeout occurred

root(/) shows 100% space used

[root@dr-ex-p-marwa-mgmt /]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
30G 29G 0 100% /

Cause

The 'du' command shows only 32k is consumed by directory tmp:

[root@dr-ex-p-marwa-mgmt /]# du -sh tmp

32K tmp

But, 'lsof' command shows two (deleted) files in directory tmp are holding about 13 GB space by oracle processes.

[root@dr-ex-p-marwa-mgmt /]# lsof | grep '(deleted)$' | sort -rnk 7
ob2rman.p 6607 oracle 3w REG 252,0 12292517888 1654786 /tmp/OB2DBG_16476_2014-08-18-3_OB2RMAN_dr-ex-p-marwa-mgmt.yahajj.net_6607_dr-dwh-1.txt (deleted)
ob2rman.p 6609 oracle 4w REG 252,0 1640636416 1654900 /tmp/OB2DBG_16476_2014-08-18-3_OB2RMAN_dr-ex-p-marwa-mgmt.yahajj.net_6609_dr-dwh-1.txt (deleted)

Files are already shown as deleted but space is not released, because space is held due to running oracle processes 6607 and 6609

[root@dr-ex-p-marwa-mgmt /]# ps -ef | grep 6607
oracle 6607 6606 99 10:59 ? 04:49:53 /opt/omni/bin/perl -I/opt/omni/lib/perl -X -CA /opt/omni/lbin/ob2rman.pl -standby -backup -full -readstdin
oracle 6609 6607 24 10:59 ? 01:12:38 /opt/omni/bin/perl -I/opt/omni/lib/perl -X -CA /opt/omni/lbin/ob2rman.pl -standby -backup -full -readstdin

Solution

Kill/stop the below processes to release space

[root@dr-ex-p-marwa-mgmt /]# ps -ef | grep 6607
oracle 6607 6606 99 10:59 ? 04:49:53 /opt/omni/bin/perl -I/opt/omni/lib/perl -X -CA /opt/omni/lbin/ob2rman.pl -standby -backup -full -readstdin
oracle 6609 6607 24 10:59 ? 01:12:38 /opt/omni/bin/perl -I/opt/omni/lib/perl -X -CA /opt/omni/lbin/ob2rman.pl -standby -backup -full -readstdin

On killing the processes 13 GB space is released

[root@dr-ex-p-marwa-mgmt /]# ps -ef | grep 6607
root      85654 74735 0 16:04 pts/3    00:00:00 grep 6607
[root@dr-ex-p-marwa-mgmt /]# df -h .
Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
                       30G   16G   13G 55% /

Note : If you delete a file, and space does not get freed, it's usually either because the file is still kept open or there are other hardlinks to it. Deleting the file won't free the space until you delete the processes that have open handles against that file.

Attachments

This solution has no attachment