Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-2200930.1
Update Date:2017-08-07
Keywords:

Solution Type  Problem Resolution Sure

Solution  2200930.1 :   Critical Alarm 6140 Generated For High TPS That Resulted in Front End Core and System Was Placed in Evaluation Mode. How To Restore Normal License.  


Related Items
  • Oracle Communications Subscriber Data Management (SDM)
  •  
Related Categories
  • PLA-Support>Sun Systems>CommsGBU>Broadband Network Solutions>SN-SND: Tekelec SDM
  •  




In this Document
Symptoms
Cause
Solution
References


Created from <SR 3-13581138311>

Applies to:

Oracle Communications Subscriber Data Management (SDM) - Version SDM 9.3 and later
Tekelec

Symptoms

 WebCI presents 6140 Critical Alarm that indicates Service Hss on a Slot is out of Service.  In rare cases, the Front End processes will not communicate to license server where the server is placed in a Evaluation mode. Evaluation mode means the server is entering into traffic limitation mode due to licensing. TPS is set to 50 sessions which will cause impact of Sh Diameter traffic.

Cause

 The SPR software is designed to restart process conditions whenever high CPU or  hung processes are encountered. This is a normal event to protect system from irregular operation.

Solution

Should you experience 6140 critical alarm in WebCI reporting Service Hss is out of service on a slot, execute the following steps to ensure that the slot has not entered the Evaluation Mode.

1) SSH into the Front End (FE) Server that was reported in Alarm 6140.

2) In /blue/var/log, you want to perform the following command: cat HssServer.* | grep "limitation mode"

Example Output

[root@bur-spr-1a log]# /blue/var/log

[root@bur-spr-1a log]# cat HssServer.* | grep "limitation mode"

HssServer.20161103_09h59m49s.out:OCDiameter[INFO]: Entering traffic limitation mode due to licensing. Current traffic rate is limited to 9000 TPS. ./HssServer.20161103_09h59m49s.out:OCDiameter[INFO]: Exiting traffic limitation mode. Current traffic rate is

9000 TPS. ./HssServer.20161103_09h59m49s.out:OCDiameter[INFO]: Entering traffic limitation mode due to licensing. Current traffic rate is limited to 9000 TPS.

 

3) If Entering traffic limitation mode due to Licensing IS NOT SEEN, and if you are experiencing Diameter too busy errors ,then proceed with the procedure, Otherwise, STOP and do not continue.

 --------------------------------------------------------------------------------------------------------------------------------------------------------

4) If Entering traffic limitation mode due to Licensing OR Diameter too Busy IS SEEN, then proceed with the procedure outlined below:

5) On the Front End server, execute Hss stop command

[root@bur-spr-1a log]# Hss stop
Hss: Stopping...
Hss: Updating Service State to STOPPING...
Hss: error="0" affected="0"
HssServer: Stopping...
HssServer: Stopped
Hss: Updating Service State to STOPPED...
Hss: error="0" affected="0"
Shutting down HP OpenCall License Server daemon:
[ OK ]
Hss: Stopped

 

6) Validate the hss process has stopped by executing the following command:

ps -ef | grep -i hss | grep -v grep

Example:  if not running there will be no output. The following is the output if hss is running.

[root@bur-spr-1a log]# ps -ef | grep -i hss | grep -v grep
root 5835 1 0 13:26 ? 00:00:00 blueHSS -orl /NodeManager2/HssServer -i -name HssServer -id -s
[root@bur-spr-1a log]#

 

7) If process has not been halted, then Manual Kill process

kill -9 <HssServer Process id>

Example:

kill -9 5835

 

8) Run step 6 again to determine if process is now stopped. If it has not, then contact Oracle Support for assistance. 

9) Start hss process by executing hss start command,

Example:
[root@bur-spr-1a log]# Hss start
Hss: Starting...
Shutting down oclicsd
Shutting down HP OpenCall License Server daemon:
[ OK ]
......stopped
Shutting down gmfprovd: [ OK ]
Starting gmfprovd: [ OK ]
oclicsd.sh: Starting monit...
oclicsd.sh: bluemonit daemon at 1011 awakened
Starting oclicsd
......started

Hss: SCTP kernel module is already loaded: do nothing
configure_sctp.sh: configuring SCTP parameters ...
configure_sctp.sh: configuring SCTP parameters done
Hss: Updating Service State to STARTING...
Hss: error="0" affected="0"
HssServer: Starting...
HssServer: Starting with: blueHSS -orl /NodeManager2/HssServer -i -name HssServer -id -s
HssServer: Started (5835)
Hss: Updating Service State to STARTED...
Hss: error="0" affected="0"
Hss: Started
[root@bur-spr-1a log]#

 

 10) Verify license has been restored by using the following commands /opt/OC/bin/oclicoam show codewords and  /opt/OC/bin/oclicoam show max all.

Example

[root@bur-spr-1a log]# /opt/OC/bin/oclicoam show codewords
/opt/OC/bin/oclicoam show max all
0 09ve0F-z8z%F2-BCDE55-BBBc51-x8COA5-8BBB55
1 0LF601-z9qQF6-hijkFF-hhh6FB-Reiu0F-gZghFF-heBhFF-ghehFF-hhhhFF
2 0LAi01-z9qrF4-EFGHAA-EEEZAE-%D9T5A-F4FEAA-EEFEAA-FEDEAA-EEEEAA


[root@bur-spr-1a log]# /opt/OC/bin/oclicoam show max all
# Feature Amount Version ExpiryDate
# ------- ------- ---------- ----------
OcDiamAuth 1 1.3 Unlimited
OcDiamTps 800 1.3 Unlimited
[root@bur-spr-1a log]#

 11) You should see codewords and OCDiamTps restored. If so, then procedure is complete. If not, please open SR for Support investigation. 

References

<BUG:25048645> - SPR 9.3 FE CORE DID NOT CLEANLY EXECUTE SHUTDOWN.
<BUG:25048595> - FAILS TO ALARM WHEN DIAMETER LICENSE IS IN EVALUATION MODE.

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback