Unable to Obtain Decoded Traces for Session (PDU appears deleted)

Asset ID:	1-72-1933464.1
Update Date:	2018-02-22
Keywords:

Solution Type Problem Resolution Sure

Solution 1933464.1 : Unable to Obtain Decoded Traces for Session (PDU appears deleted)

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 9.0.3 to 9.0.4 [Release 9.0]
Information in this document applies to any platform.

Symptoms

Decoded traces are not available in session for some particular duration of time period. Deleted is seen instead of PDU decode.

Changes

Cause

Issue verification

Check session lag

If there is huge lag observed (value between parenthesis) on the build for the duration decoded traces are missing, this might the reason for the missing traces.

ixp0001-1a# tr.cat IxpBuild102306
0918:042600.000 DF1 Last in: 22:40:51 (20709), last out: 22:40:48 (20712). List size: 58277. Input: 19 Mbps [31841/DfThread.cpp:748]
0918:042700.010 DF1 Last in: 22:41:28 (20732), last out: 22:41:25 (20735). List size: 65532. Input: 19 Mbps [31841/DfThread.cpp:748]
0918:042800.000 DF1 Last in: 22:42:05 (20755), last out: 22:42:03 (20757). List size: 48294. Input: 20 Mbps [31841/DfThread.cpp:748]
0918:042900.000 DF1 Last in: 22:42:42 (20778), last out: 22:42:39 (20781). List size: 54653. Input: 19 Mbps [31841/DfThread.cpp:748]
0918:043000.010 DF1 Last in: 22:43:18 (20802), last out: 22:43:15 (20805). List size: 60405. Input: 18 Mbps [31841/DfThread.cpp:748]

Check IxpBuild process overload

You must be receiving below messages in Build logs. You can also refer to output of "top" command.

918:095900.906 IXP3 Send alarm: Thread CPU overload (clear event) - Average CPU load (71%) < threshold (80%) - DFP MAP_HLR - Thread xDR storage [30587/DmAlarm.cpp:214]
0918:100000.070 DF1 Last in: 09:58:37 (83), last out: 09:58:35 (85). List size: 100773. Input: 29 Mbps [30601/DfThread.cpp:748]
0918:100001.007 IXP3 Send alarm: Thread CPU overload (warning event) - Average CPU load (96%) > threshold (80%) - DFP MAP_HLR - Thread Level 2 PDU processing [30587/DmAlarm.cpp:214]
0918:100100.067 DF1 Last in: 09:58:58 (122), last out: 09:58:56 (124). List size: 91773. Input: 23 Mbps [30601/DfThread.cpp:748]
0918:100109.191 TR-V Got signal 15 [30583/CcMain.cpp:492]

Verify off load setting

Now to verify if high lag on IxpBuild is the cause for missing decoded traces, login to master server of IXP subsystem having the session as cfguser and execute below command:

ixp0001-1a# igrep -p BldOffLoadLag LongParam

Generally its value is set to 300, 1800, or 3600.

If lag present on build for the duration of missing decoded traces is greater than this parameter value, then this is the reason why decoded traces are missing in session.

BldOffLoadLag is an internal parameter for PIC, which disables PDU storage for session when there is lag on Build greater than the specified limit in order to prevent complete data loss. (here it was defined as 300 seconds, hence whenever lag will be greater than 300sec, pdu storage will be automatically disabled)

Solution

BldOffLoadLag parameter value can be increased to max recommended value 3600. But again if lag will exceed this value, decoded traces will be again not available.

If IxpBuild have high load, recommend customer to decrease IxpBuild load by filtering/splitting the traffic on several IxpBuild.

BldOffloadLag value may not be adapted to your customer configuration as a 60s lag can be very common depending on usual load or network capacity. In this case you may want to change it to a higher value.

For example if you'd like to change it to 1800 seconds, type this command as cfguser on Active Master:

iset -fvalue=1800 LongParam where "name='BldOffloadLag'"

Expected output:

=== changed 1 records ===

Warning60.gif IxpBuild processes have to be restarted to take that parameter change into account.

Attachments

This solution has no attachment