How to Troubleshoot Session Lagging on PIC?

Asset ID:	1-75-2248741.1
Update Date:	2017-11-12
Keywords:

Solution Type Troubleshooting Sure

Solution 2248741.1 : How to Troubleshoot Session Lagging on PIC?

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 7.1 and later
Information in this document applies to any platform.

Purpose

This document describes main steps to troubleshoot a session being late (not on current time) on Oracle Communication Performance Intelligence Center.

It is important not to mix lagging session and stopped session. A session lagging is a session which do not have current data but which is still has "End date" updating.

Troubleshooting Steps

Understand Last in/Last out

First, it is important to understand IXP process log. Typically, it looks like:

1119:132400.03 DF1 Last in: 13:23:56 (3), last out: 13:23:35 (14). List size: 285558

Explanation of each field:

Last in: Data entering DTS buffer of DFP.
Last out: Data exiting DTS buffer (not of DFP!)
(3) : Difference between last in and current date. This is usually this counter that shows the lag.
(13): Difference between last out and current date
List size: Amount of frames or xDR per minutes
Valid for IxpBuild, IxpOperate, IxpStore

Procedure

ProTrace session list
ProTrace "Query Settings", click "Refresh session period"
Is it lagging?
- Yes: it is a Start/End date refresh issue. Verify content of log /var/log/nsp/NSP_CALCULATE_PERIOD.log
- No: No lagging issue on the system
Connect on ixp as cfguser
Check IxpStore log
1. Find where is the process:
  show_process_list -l | grep STOR | grep <storename>
  Output example:
  [cfguser@ixp0001-1a ~]$ show_process_list.pl -l|grep STOR | grep DIA
  YES 26031 STORAGE ixp0001-1c S_DIAMETER
2. Connect on server hosting the process (1c in example)
Display store log (id taken from step a) and verify if lag is present:
tr.tail -20 IxpStore26031
Is it lagging?
- Yes: Search for error in that log and search that error message in MOS. If no error message present, move to next step
- No: Then, it is a GUI issue (ProTrace). Verify step 3 again.
Find input stream(s) of store with show_process_info.pl
output example:
show_process_info.pl -a --id 26031
--------------------------------------------------------------------------------
Basic Dataflow processing information
--------------------------------------------------------------------------------
DFP ID : 26030
DFP name : 'S_DIAMETER'
DFP type : STORAGE
Process ID : 26031
Active : YES
--------------------------------------------------------------------------------
Input streams
--------------------------------------------------------------------------------
ID Type Location Name
25938 XDR Internal B_DIAMETER
--------------------------------------------------------------------------------
Output sessions
--------------------------------------------------------------------------------
ID Lifetime DHW pool Name
26027 72 ixp0001_Pool DIAMETER
Find which process is writing in stream (in example, B_DIAMETER):
[cfguser@ixp0001-1b ~]$ show_stream_info.pl -a --name B_DIAMETER
--------------------------------------------------------------------------------
Basic Stream Information
--------------------------------------------------------------------------------
Stream ID : 25938
Stream name : 'B_DIAMETER'
Data type : XDR
Extern IP1 :
Extern IP2 :
--------------------------------------------------------------------------------
Stream History
--------------------------------------------------------------------------------
ID IP Address Start timestamp End timestamp
14688 ixp0001-1c 02/14/2017 13:41:57 02/14/2017 13:51:19
14694 ixp0001-1c 03/24/2017 10:55:14 12/31/1969 19:00:00
--------------------------------------------------------------------------------
Last data on local machine
--------------------------------------------------------------------------------
Timestamp Micro Data size Compression Subpart
--------------------------------------------------------------------------------
Processings reading from the stream
--------------------------------------------------------------------------------
ProcID Type Server Name
26031 STORAGE ixp0101-1c S_DIAMETER
--------------------------------------------------------------------------------
Processings writing into the stream
--------------------------------------------------------------------------------
ProcID Type Server Name
26024 BUILDING ixp0101-1b B_DIAMETER
Warning:
- Build can be distributed on another server of the subsystem
- It can be an operate process that writes in the stream
Display build log and verify if lag is present:
tr.tail IxpBuild26024
Is it lagging?
- Yes: Verify all input streams in the build as it can be due to any source. It is important to check if critical state is checked for all streams
- No: Verify step 3 again.

Attachments

This solution has no attachment