Gn/Gp Builder Restarts Continuously when GTP Association Table Size is too Small

Asset ID:	1-72-1949059.1
Update Date:	2017-01-04
Keywords:

Solution Type Problem Resolution Sure

Solution 1949059.1 : Gn/Gp Builder Restarts Continuously when GTP Association Table Size is too Small

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 10.1.0 to 10.1.5.1 [Release 10.0]
Tekelec

Symptoms

IxpBuild process is restarting several times a day.

Check related DFP with Buildmonitor, number of xDR in progress (BldPeakInProgressXDR) is growing till it reaches around 100 000, then goes down to 0 and this continues over and over

Cause

Default GTP builder settings are limiting number of xDR in progress to 100 000, IxpBuild process crashes when it is reaching the limit.

In abterms files located in $PROC/IxpBuildxxxxx/, following lines can be found after "<signal handler called>":

5 <signal handler called>
6 0x00007fd1e0042976 in GprsGtpContextInfo::SecondaryContext(PLD_Gtp*) () from /opt/TKLCixp/prod/lib/plugins/build/libGprsGtp.so
7 0x00007fd1e0043383 in GprsGtpContextInfo::AddKey(unsigned int, GprsGtpKey, PLD_Gtp*) () from /opt/TKLCixp/prod/lib/plugins/build/libGprsGtp.so
8 0x00007fd1e005396d in GprsGtpAssociationTable::UpdateContextTable(unsigned char*, unsigned char*, bool) () from /opt/TKLCixp/prod/lib/plugins/build/libGprsGtp.so

Solution

In CCM go to Mediation -> IXP -> Dataflow Processings -> Related build process -> Builders -> GTP GPRS
Increase "Associate table size" parameter to 1000000 (1 million)
Apply changes on IXP

After modification, if IxpBuild is still crashing with same abterm stack, it probably means it reach 1000000 (1 million). Open a SR to have this value raised for PIC release below 10.2.

PIC 10.2 allows to set value up to 5000000 (5 millions).

It may happen that even setting "Associate table size" parameter to 1 000 000 is not sufficient, but CCM is not allowing larger values. Another situation when this may be required is when Gn correlation is low and to improve situation it is required to increase "Garbage time of Context duration", which in turn will increase observed number of xDR in progress.

In this case, it is possible to edit CCM limits by editing values directly in NSP database and after repeat the above steps 1-3 to increase the value.

Be careful on any sql query done on NSP database. It may cause inconsistencies.

Connect to the NSP oracle box as oracle user and connect to sqlplus:
[oracle@NSP ~]$ sqlplus nsp/<password>

SQL*Plus: Release 10.2.0.3.0 - Production on Mon Feb 16 22:57:26 2015
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning and Data Mining options

SQL>
Change the limits with the following SQL statement:
SQL> update CFG_XBPARAMTYPE set DATATYPE='LIM_INT(10,5000000)' where NAME='CONFIG:GTP Association Table size';

1 row updated.
Commit your changes:
SQL> commit;

Commit complete.
Disconnect:
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning and Data Mining options
Log on to the NSP GUI and increase the "Association Table size" up to 5 Millions for the build Dataflow processing that would require it.

References

<BUG:20058430> - GN TDR BUILD IS CRASHING WHEN REACHING 100000 XDR IN PROGRESS

Attachments

This solution has no attachment