After IXP Server Upgrade to PIC 9.0.3 and 9.0.4 Some Sessions are Not Recovering

Asset ID:	1-72-1681561.1
Update Date:	2015-12-15
Keywords:

Solution Type Problem Resolution Sure

Solution 1681561.1 : After IXP Server Upgrade to PIC 9.0.3 and 9.0.4 Some Sessions are Not Recovering

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 9.0.3 to 9.0.4 [Release 9.0]
Tekelec

Symptoms

After IXP server upgrade, some sessions are not recovering. No particular error message visible from ProTrace application GUI beside session end time not updating.

Changes

IXP server was just upgraded.

Cause

IxpForward might be stuck in deadthread state and is not recovering automatically.

This can be confirmed by look at IxpForward log which is stuck on following message:

1008:153244.048 TR-V Dead thread detected: 0xb6774b90 (MonitoringManager1) - Subsystem: [5618/CcMain.cpp:357]
1008:153245.050 TR-V See details in /var/TKLC/ixp/run/proc/IxpForward/deadthread.IxpForward.131008153245 [5618/CcMain.cpp:412]

Solution

Connect on IXP server as cfguser
Kill IxpForward process:
pm.kill -9 IxpForward
Verify IxpForward logs. New message should appear and no new "deadthread" message should be visible

Issue will not reoccur outside an upgrade context, so it is a definitive workaround.

Attachments

This solution has no attachment