Diameter Signaling Router (DSR) : Server Disk Space Shortage Warning :Lot Of Pam

Asset ID:	1-72-2385985.1
Update Date:	2018-04-18
Keywords:

Solution Type Problem Resolution Sure

Solution 2385985.1 : Diameter Signaling Router (DSR) : Server Disk Space Shortage Warning :Lot Of Pam_unix Messages Under /var/log/secure File

Applies to:

Oracle Communications Diameter Signaling Router (DSR) - Version DSR 7.2.0 and later
Tekelec

Symptoms

We were getting Server Disk Space Shortage Warning/Error on both Sites Active SOAM's.

[admusr@dsrSO-a ~]$ sudo syscheck
Running modules in class disk...
* fs: FAILURE:: MINOR::5000000000000001 -- Server Disk Space Shortage Warning
* fs: FAILURE:: Space used in "/var" exceeds the recommended limit 80%. 90% used.

[admusr@dsrSO-a ~]$ sudo syscheck
Running modules in class disk...
* fs: FAILURE:: MAJOR::3000000000001000 -- Server Disk Space Shortage Error
* fs: FAILURE:: Space used in "/var" exceeds the set limit 90%. 93% used.

Changes

Cause

As we know /var directory has been given as 976M Size and out of which we do see almost 5 Secure messages files under /var/log which utilizes most of the space from /var directory and crosses the threshold level and causing alarm to be appeared on the SOAM's.
Normally these secure-mmddyy file size remains under 10M, but here we see the Secure files which are getting generated has around 160-170M Size.
When i checked the contents in these files, i can see continuous messages as below :

[admusr@dsrSO-a ~]$ sudo tail -200 /var/log/secure-20180408
Apr 8 04:00:58 dsrSO-a sshd[23784]: Accepted publickey for awadmin from X.X.X.X port 53403 ssh2
Apr 8 04:00:58 dsrSO-a sshd[23784]: pam_unix(sshd:session): session opened for user awadmin by (uid=0)
Apr 8 04:00:58 dsrSO-a sshd[23787]: Accepted publickey for awadmin from X.X.X.X port 53405 ssh2
Apr 8 04:00:58 dsrSO-a sshd[23787]: pam_unix(sshd:session): session opened for user awadmin by (uid=0)
Apr 8 04:00:58 dsrSO-a sshd[23795]: Received disconnect from X.X.X.X: 11: disconnected by user
Apr 8 04:00:58 dsrSO-a sshd[23784]: pam_unix(sshd:session): session closed for user awadmin
Apr 8 04:00:58 dsrSO-a sshd[23813]: Received disconnect from X.X.X.X: 11: disconnected by user
Apr 8 04:00:58 dsrSO-a sshd[23787]: pam_unix(sshd:session): session closed for user awadmin
Apr 8 04:00:58 dsrSO-a sshd[23785]: Accepted publickey for awadmin from X.X.X.X port 53404 ssh2

Solution

We went through the /var/log/secure messages and we can see continuous flow of messages, as mentioned above, which indicates that there are ISO's still placed under Main Menu: Status & Manage -> Files , were not yet UnDeployed.

The ISO deployment audit checks that the files present in the IsoDeployment table is present in the file management area. If the file is not preset in the file management area, then the same is tried to be scp'd from the Active Parent.

The audit is happening locally based on these two - Entries in the IsoDeployment table. - Files in the file management area . So each server checks the two variables needed for the audit. You need to delete the entries from the table. You shall be able to do that by using the undeploy ISO feature on the NOAM GUI. The IsoDeployment table is a replicated table. Once the changes are made on the NOAM, the same will be replicated down to all the servers.

We performed the procedure of UnDeploy ISO from NOAM GUI, once done, we can see that the messages (pam_unix(sshd:session): session opened for user awadmin) stop getting generated.

Note : You can follow the DSR Upgrade Guide , Appendix A.2 for UnDeploy ISO

https://docs.oracle.com/cd/E68457_01/docs.72/E73083_rev_03.pdf

Attachments

This solution has no attachment