Oracle Communications Policy Management
 

PLA-Support>Sun Systems>CommsGBU>Broadband Network Solutions>SN-SND: Tekelec Policy
 

Server has qp_procmgr restart errors due to a known java bug in JDK8 (build 1.8.0-b132).

In this Document

Symptoms

Cause

Solution

References

Created from <SR 3-12801558321>

Applies to:

Symptoms

Product:
Oracle Communications Policy Management

Version:
POLICY 12.0.0

Behavior:
Active MPE blade restart (qp_procmgr).

Error:

AppEventLog.0

part=0 srcNode=00000.000 srcTimeStamp=12/31/1969 19:00:00.000 severity=*C timeStamp=06/02/2016 17:20:54.927 seqNum=91 task=camiantApp eventNumber=QP Critical process failed instance= eventData= errInfo=Critical process policy died additionalInfo=

Example:
qp_procmgr:
Thu Jun 2 17:20:57 EDT 2016
qp_procmgr died, reason: 194, death count=1
starting getCrashInfo in the background
Thu Jun 2 17:20:59 EDT 2016: QP_PROCMGR restarting. Death count=1
RESETTING OPERATIONAL ENVIRONMENT

rc_output.log has a full thread dump shortly after.

Example:
2016-06-02 17:22:44
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):

Tomcat_output.log shows problems with JAR file and has a full thread dump at the same time as server restarts.

Example:
Jun 02, 2016 5:21:00 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with JAR file [/usr/TKLC/camiant/tomcat/lib/comcol.jar], exists: [false], canRead: [false]
Jun 02, 2016 5:21:00 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with JAR file [/usr/TKLC/camiant/tomcat/lib/comcol.jar], exists: [false], canRead: [false]

2016-06-02 17:22:44
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):

Tomcat_output-2016-06-02-17-21.log shows JVM shutdown

************************************
* JVM shutdown has been initiated. *
************************************
Jun 02, 2016 5:20:54 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]

*** Trying to dump threads for PID = 8833:

2016-06-02 17:20:54
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):

Followed by the following errors.

Jun 02, 2016 5:20:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/li] appears to have started a thread named [IPC.ConnectionMonitor:100000] but has failed to stop it. This is very likely to create a memory leak.
Jun 02, 2016 5:20:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/li] appears to have started a thread named [IPCTCP.Selector] but has failed to stop it. This is very likely to create a memory leak.
Jun 02, 2016 5:20:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

Expected Behavior:
For qp_procmgr to not restart.

Cause

This is a known java bug and it fixed in 12.1 releases.

[ROOT CAUSE]:
JVM crash with JDK8 (build 1.8.0-b132) with G1 GC. This fix is included in 7u76 b01 and in 8u60 b01. The fix is covered in build 1.8.0_40-b25, as this is a later build

Known Bugs:
Java bug JDK-8044406
Bug 21543579
Bug 21615228
Bug 23522198

Example:
JRE version: Java(TM) SE Runtime Environment (7.0_72-b14) (build
1.7.0_72-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x522e65]
G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*,
HeapWord*, void const*)+0xd5
#
# Core dump written. Default location:
/var/camiant/db/comcol/run/proc/qp_procmgr/core or core.8832
#
# An error report file with more information is saved as:
# /var/camiant/db/comcol/run/proc/qp_procmgr/hs_err_pid8832.log
 

Solution

This issue is fixed by upgrading the code to the 12.1 releases or higher.
 

References

This solution has no attachment