Oracle Communications Diameter Signaling Router (DSR)
 

PLA-Support>Sun Systems>CommsGBU>Global Signaling Solutions>SN-SND: Tekelec DSR
 

When a filesystem is reporting excessive utilization and no large files or large numbers of small files can be found to be the culprit, the problem may be in part due to a file or files that were open suddenly were terminated abnormally and deleted.  Frequently this occurs when a user unintentionally creates a file that grows rapidly and fills the remaining space in the filesystem.  It may terminate, and the user may try to delete (remove) the file to clear up space.  The file will no longer show up when the directory is listed ('ls') but the space did not free up.

In this Document

Goal

Solution

Created from <SR 3-10891408831>

Applies to:

Goal

This document will outline how to find open (deleted) files that are still consuming space in the filesystem, and allow the user to delete them.

Solution

First, check filesystem using standard methods to find either large files or large numbers of small "artifact" files that are causing the problem, and delete them conventionally.  Check that the filesystem utilization has returned to normal.

If standard methods do not reveal the culprit or if the filesystem has not returned to normal utilization levels after the previous deletes, check to see if an open file is in limbo and unable to be completely deleted:

This command may show a number of temporary files with small or zero filesizes.  This is normal and can be ignored.  Look for large filesizes with obvious causes.  Examples:

[REMOVED EXTRA STUFF FROM OUTPUT, ADDED COLUMN HEADER, ADDED LINE =X=> INDICATORS]

The =1=> file is an example of a temporary file that can be ignored.  Note its size is 0. This file is system generated.
The =2=> file is ENORMOUS and clearly a user file that is in limbo.

The pcap file needs to be deleted to clear up the 153014272 bytes it is consuming.  Typically there are two methods to delete a file in this condition.  First, if it was due to a process launched by a user, logging out of the server can allow the file to close and delete.  Sometimes that is not an option, as is the case here.  The tcpdump user is invoked when an output file is created by the tcpdump utility, regardless of which user launched the utility.  To resolve this scenario, we need to terminate [via 'kill -9 <PID>'] the process provided by the 'lsof' command in the resulting output.  As an example, we'll use from the above:

Since the file is already deleted, when the process (PID) is terminated the filesystem utilization should release the space. 

This solution has no attachment