Diameter Signaling Router (DSR) - Server Disk Space Shortage Warning on /var due to Excessive httpd Logging

Asset ID:	1-72-2265433.1
Update Date:	2017-09-25
Keywords:

Solution Type Problem Resolution Sure

Solution 2265433.1 : Diameter Signaling Router (DSR) - Server Disk Space Shortage Warning on /var due to Excessive httpd Logging

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 7.3.0 to DSR 8.0.0 [Release DSR 7.0 to DSR 8.0]
Information in this document applies to any platform.

Symptoms

Server Disk Space Shortage Warning for /var directory on one of the DSR nodes in DSR7.3.

Disk space usage should not reach too high, disc usage should be below the set threshold.

Run following command to verify the issue:

df -hk

And check the /var status in output:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-plat_var 976M 736M 190M 80% /var

Changes

Cause

Issue is caused by enhanced logging in following files:

/var/log/httpd/ssl_request_log
/var/log/httpd/ssl_access_log
/var/log/messages

There are huge number of reverse DNS lookup queries logged in /var/log/messages on the problematic SOAM server:

Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#64454: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#52859: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#61814: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#58250: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#56824: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#52733: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#59340: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#53462: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#54161: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)
Apr 23 04:10:01 SOAMNAME named[4675]: client xxx.xxx.x.x#59042: query: <IP Address>.in-addr.arpa IN PTR + (yyy.yyy.y.y)

Logging level on these can be decreased. Following log files have higher logging levels which are related to bug 25752225 (fixed in 8.1).

/var/log/httpd/ssl_request_log
/var/log/httpd/ssl_access_log

Solution

Carry out the following steps to resolve the issue:

Delete the oldest 2 ssl_request_log and ssl_access_log files manually from /var/log/httpd folder to clear condition
Turn off DNS query logging on problematic SOAM using following steps:

Log in as root to SOAM Command Line
Run Command:
rndc querylog

Check in the /var/log/messages that following statement is seen in the log file to ensure the query logging is off :

query logging is now off

Command mentioned in step 2 is not persistent and if the system is rebooted, the command may be needed to be run again.

References

<BUG:25752225> - [SOAM:HTTPD] /VAR/ PARTITION USAGE WARNINGS DUE TO EXCESSIVE HTTPD LOGGING
<NOTE:2200931.1> - Server Disk Space Shortage Warning Space Alarm In DSR

Attachments

This solution has no attachment