PDUs are getting dropped on PMF and IXP due to change in COMCOL version to 7.3 from 6.5 as a part of TPD change to 7.4

Asset ID:	1-77-2337228.1
Update Date:	2018-03-02
Keywords:

Solution Type Sun Alert Sure

Solution 2337228.1 : PDUs are getting dropped on PMF and IXP due to change in COMCOL version to 7.3 from 6.5 as a part of TPD change to 7.4

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Hardware - Version 10.2.1 to 10.2.1 [Release 10.0]
Tekelec

Description

Due to change in TPD version from 7.0.4 to 7.4 and subsequent changes in COMCOL version from 6.5 to 7.3, it is observed that PDU messages are being dropped on PMF and IXP.

There will be a loss of monitoring traffic because of non updation of statistical sessions in IXP and PDU message drop on PMF.

Occurrence

Only affect PIC release installed on 10.2.1 or upgraded to 10.2.1 release.

Symptoms

One way to identify the issue is to search for the following error:

ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest record

On PMF, the symptoms can be found in pmiaMonitor.

[cfguser@<PMF-SITE> ~]$ tr.tail pmiaMonitor
+ cf.follow -10 /tekelec/TKLCmf/runtime/run/proc/pmiaMonitor/trace.dat
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:04:53.296, nsec=115349
[DBMsuBlock.C:215]
[7381/DBMsuBlock.C:357]
0817:050453.299 TR-V link 22329: pause the add on all links due to the
failure: ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest
record [IdbFragIter.cxx:440]
^^ t48=(1502960693.296 < endStamp=1502960693.297 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:04:53.296, nsec=115349
[DBMsuBlock.C:215]
[7381/MfLinkWriter.C:443]
0817:050503.318 TR-V link 22329: resume the add on all links
[7374/MfLinkWriter.C:496]
0817:050915.006 TR-V DbMsuBlock00 DBMsuBlock::add Failed to add a new
block: ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest record
[IdbFragIter.cxx:440]
^^ t48=(1502960955.002 < endStamp=1502960955.003 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:09:15.002, nsec=265740
[DBMsuBlock.C:215]
[7381/DBMsuBlock.C:357]
0817:050915.006 TR-V link 22329: pause the add on all links due to the
failure: ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest
record [IdbFragIter.cxx:440]
^^ t48=(1502960955.002 < endStamp=1502960955.003 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:09:15.002, nsec=265740
[DBMsuBlock.C:215]
[7381/MfLinkWriter.C:443]
0817:050925.318 TR-V link 22329: resume the add on all links
[7374/MfLinkWriter.C:496]
0817:051004.708 TR-V DbMsuBlock00 DBMsuBlock::add Failed to add a new
block: ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest record
[IdbFragIter.cxx:440]
^^ t48=(1502961004.701 < endStamp=1502961004.702 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:10:04.701, nsec=314174
[DBMsuBlock.C:215]
[7381/DBMsuBlock.C:357]
0817:051004.708 TR-V link 22329: pause the add on all links due to the
failure: ERROR: IDB_NOT_NEWEST: log timestamp precedes current newest
record [IdbFragIter.cxx:440]
^^ t48=(1502961004.701 < endStamp=1502961004.702 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
^^ 0: failed to add new block, 08/17/2017 05:10:04.701, nsec=314174
[DBMsuBlock.C:215]
[7381/MfLinkWriter.C:443]

On IXP, the symptoms can be found in Build logs.

1102:120947.302 DTS2 DtsProducerBuffer::Flush addRec failed : ERROR:
IDB_NOT_NEWEST: log timestamp precedes current newest record
[IdbFragIter.cxx:440]
^^ t48=(1509617335.000 < endStamp=1509617335.001 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
, Stream : 20040 [30952/DtsProducerBuffer.cpp:540]
1102:120947.302 IXP3 Send alarm: xDR Loss (major event) Stream =
B_MSS_BICC_30 [30952/DmAlarm.cpp:215]
1102:120947.303 IXP3 Send alarm: xDR Loss clear (clear event) Stream =
B_MSS_BICC_30 [30952/DmAlarm.cpp:215]
1102:120947.374 DTS2 DtsProducerBuffer::Flush addRec failed : ERROR:
IDB_NOT_NEWEST: log timestamp precedes current newest record
[IdbFragIter.cxx:440]
^^ t48=(1509617342.000 < endStamp=1509617342.001 [IdbFragIter.cxx:441]
^^ IdbFragIter::_writePrep() [IdbFragIter.cxx:527]
^^ IdbIter::addRecSelect(void *xRec) [IdbIter.cxx:2456]
, Stream : 20040 [30952/DtsProducerBuffer.cpp:540]
1102:120947.375 IXP3 Send alarm: xDR Loss (major event) Stream =
B_MSS_BICC_30 [30952/DmAlarm.cpp:215]
1102:120947.375 IXP3 Send alarm: xDR Loss clear (clear event) Stream =
B_MSS_BICC_30 [30952/DmAlarm.cpp:215]

Patches

This is fixed in maintenance release is available on MOS under Patch 27584919: Patch for PIC 10.2.1.1.0.

Note: To upgrade from 10.2.1.0.0 use the "Patch Installation Guide" on OHC under PIC 10.2.1. This must be run for all modules as this is a full release.

To upgrade from previous releases use the "Upgrade Guide" on OHC under PIC 10.2.1 (Please check the document for supported paths).

History

Changes

Changes in COMCOL version to 7.3 from earlier 6.5.

Cause

The issue is caused due to the change in the version of COMCOL to 7.3.

From COMCOL's version 7.3, it now uses 64 bit time stamp.

Solution

Oracle recommends not using PIC 10.2.1 until a fix is available. Current target for fix is 23 Feb 2018.

References

<BUG:26695879> - PDU IS GETTING DROP ON PMF.

Attachments

This solution has no attachment