Alarm 32500 or 32312 received - Server Disk Space Shortage Warning/Error alarm for /var/camiant/log

Asset ID:	1-72-2217144.1
Update Date:	2017-01-13
Keywords:

Solution Type Problem Resolution Sure

Solution 2217144.1 : Alarm 32500 or 32312 received - Server Disk Space Shortage Warning/Error alarm for /var/camiant/log - qpUpgradeHelper.log

Applies to:

Oracle Communications Policy Management - Version POLICY 10.5.0 to POLICY 12.1.0 [Release POLICY 10.0 to POLICY 12.0]
Information in this document applies to any platform.

Symptoms

Alarm page in GUI reporting the following -

Severity: Minor - Alarm ID: 32500 - Description: Server Disk Space Shortage Warning

Severity: Major - Alarm ID: 32312 - Description: Server Disk Space Shortage Error

Syscheck reporting the following -

For alarm 32500 between 80% and 89%

* fs: FAILURE:: MINOR::5000000000000001 -- Server Disk Space Shortage Warning
* fs: FAILURE:: Space used in "/var/camiant/log" exceeds the recommended limit 80%. 85% used.

for Alarm 32312 between 90% and 100%

* fs: FAILURE:: MAJOR::3000000000001000 -- Server Disk Space Shortage Error
* fs: FAILURE:: Space used in "/var/camiant/log" exceeds the set limit 90%. 100% used.

Use df -h to check the partition sizes, and ls -lh in the partition

[root]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-plat_root
992M 384M 557M 41% /
tmpfs 24G 0 24G 0% /dev/shm
/dev/mapper/vgroot-plat_tmp
992M 34M 908M 4% /tmp
/dev/mapper/vgroot-plat_usr
3.9G 2.0G 1.7G 55% /usr
/dev/mapper/vgroot-plat_var
992M 343M 599M 37% /var
/dev/mapper/vgroot-plat_var_tklc
3.9G 1.7G 2.1G 45% /var/TKLC
/dev/sda1 251M 77M 162M 33% /boot
/dev/mapper/vgroot-camiant_var
257G 5.3G 238G 3% /var/camiant
/dev/mapper/vgroot-camiant_log
4.0G 3.9G 0 100% /var/camiant/log
/dev/mapper/vgroot-camiant_opt
2.0G 790M 1.1G 42% /usr/TKLC/camiant
/dev/mapper/vgroot-camiant_tmp
7.9G 146M 7.4G 2% /var/camiant/tmp
/dev/mapper/vgroot-camiant_upgrade
1008M 460M 497M 49% /var/camiant/upgrade
[root@vzw-cmp-1a log]#

[root@vzw-cmp-1a log]# ls -lh /var/camiant/log/qpUpgradeHelper.log
-rw-r----- 1 root root 3.1G Dec 22 11:07 /var/camiant/log/qpUpgradeHelper.log
[root@vzw-cmp-1a log]#

Cause

One possible cause for this is the /var/camiant/log/qpUpgradeHelper.log file. A known issue causes this file to continuously grow.

Solution

[WORKAROUND]

The qpUpgradeHelper.log file can be safely removed using rm -f.

# rm -f /var/camiant/log/qpUpgradeHelper.log

[PERMANENT FIX]

The following cron can be added to automatically delete the file. From the command line as root, run the following two commands -

# echo '0 4 * * 0 root rm -f /var/camiant/log/qpUpgradeHelper.log > /dev/null 2>&1' > /etc/cron.d/delete-log
# service crond reload

This will create a cron file that will automatically delete the file at 04:00 every Sunday.

This issue is fixed in PCRF version 12.1.2 and above.

[VERIFICATION]

Once the workaround or fix has been implemented, verify the alarm has cleared on the Alarm page in the GUI, or re-run the following syscheck command from the command line to verify in syscheck

# syscheck -v

[ROLL BACK]

To roll back the cron, delete the previously created file with the following command. This will delete the cron entry completely.

# rm -f /etc/cron.d/delete-log

Attachments

This solution has no attachment