UAM 0911 Dynamic Database Is Inconsistent

Asset ID:	1-72-2133013.1
Update Date:	2018-05-29
Keywords:

Solution Type Problem Resolution Sure

Solution 2133013.1 : UAM 0911 Dynamic Database Is Inconsistent

Applies to:

Oracle Communications EAGLE (Software) - Version EAGLE 43.x and later
Tekelec

Symptoms

On Eagle STP, UAM 0911 "Dynamic database is inconsistent" appears frequently

  ****16-02-25 11:32:57****
  7214.0911 ** SYSTEM Dynamic database is inconsistent
  Card 1201
  ****16-02-25 11:53:02****
  9461.0911 ** SYSTEM Dynamic database is inconsistent
  Card 3101
  ****16-02-25 12:33:08****
  9152.0911 ** SYSTEM Dynamic database is inconsistent
  Card 3113
  ****16-02-25 13:03:08****
  7611.0911 ** SYSTEM Dynamic database is inconsistent
  Card 3113

Cause

For the Time when DDB inconsistency is observed; take following logs from STP:

rtrv-ss7opts
rept-stat-mps
rept-stat-db:db=mps:display=all

aud-data:type=ddb:display=all

rtrv-log:snum=09111:enum-0922:num=100
Also; rtrv-log for all UAM and UIM during the time when appeared as well

rept-stat-mfc:mode=stats:sample=tot5m:service=vsccp
rept-stat-mfc:mode=stats:sample=avg1h:service=vsccp
rept-stat-mfc:mode=stats:sample=tot24h:service=vsccp

rept-imt-info:report=hipr2err
rept-imt-lvl1:r=summary:sloc=1201:eloc=1115
rept-imt-lvl1:r=full:sloc=1201:eloc=1115
clr-imt-stats:all=yes
Wait 15 mins
rept-imt-lvl1:r=summary:sloc=1201:eloc=1115
rept-imt-lvl1:r=full:sloc=1201:eloc=1115

DDB Inconsistency alarms are appearing in parallel with IP connection alarms.

Output of aud-data command shows that one SCCP card 3107 is inconsistent for MATED APPL Table:

> aud-data:type=ddb:display=all

stp 16-03-09 17:07:20 EST EAGLE5 45.0.1-64.70.35
aud-data:type=ddb:display=all
Command entered at terminal #21.
;

Command Accepted - Processing
Command Executed
stp 16-03-09 17:07:21 EST EAGLE5 45.0.1-64.70.35
DDB AUDIT REPORT
SYSTEM STATUS : INCONSISTENT
ACTIVE MTP CARDS : 75
NON RESPONDING CARDS : 0
RESPONDING CARDS : 75
CARDS WITH NO DATA : 0
CARDS WITH DATA : 75
CARDS FAILING QUIET PRD : 0
CARDS WITH DDB UPD IN PRG : 0
CARDS CONSIDERED FOR CKSM : 75
INCONSISTENT CARDS : 1: 3107
CONSISTENT CARDS : 74
AUDIT START TIME : 09/03/2016 17:07:20
QUIET PERIOD : 500 ms

RTE LINK SET LINK CM CARD CM CLSTR MATED APPL MTP GLOBLS
LOC STATUS CAUSE IDLE DDB UPD ADDN'L STATUS
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1105 CONSISTENT 904350 6505660
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1107 INCONSISTENT 1890 3659665
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1112 CONSISTENT 904360 6499313
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1201 CONSISTENT 249765 3651717
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1203 CONSISTENT 904420 6534829
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1205 CONSISTENT 904400 6542089
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1207 CONSISTENT 185055 3642316
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1212 CONSISTENT 904395 6551176
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1213 CONSISTENT 480615 3639258
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1215 CONSISTENT 904380 6551194
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1216 CONSISTENT 904340 9268188
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1217 CONSISTENT 904375 6552876
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1218 CONSISTENT 904385 9125846
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1301 CONSISTENT 72770 3625357
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1303 CONSISTENT 904405 6552817
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1304 CONSISTENT 904355 6555890
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1305 CONSISTENT 121265 3638266
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1308 CONSISTENT 904025 6558437
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1311 CONSISTENT 904400 6559966
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1312 CONSISTENT 904400 6560704
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
1313 CONSISTENT 294600 3628277
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1315 CONSISTENT 904390 6558940
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1316 CONSISTENT 904375 9282463
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1317 CONSISTENT 904385 6548881
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
1318 CONSISTENT 904385 9103588
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2101 CONSISTENT 171510 3643687
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2103 CONSISTENT 904355 6561891
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2104 CONSISTENT 904380 6565456
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2107 CONSISTENT 32115 3642594
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2111 CONSISTENT 904380 6568285
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2113 CONSISTENT 182925 3640632
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2115 CONSISTENT 904350 6567863
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2117 CONSISTENT 904365 6567748
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2201 CONSISTENT 231350 3635331
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2203 CONSISTENT 904330 6567464
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2205 CONSISTENT 904385 6564380
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2213 CONSISTENT 407935 3630102
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2215 CONSISTENT 904330 8080027
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2217 CONSISTENT 904340 8050585
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2218 CONSISTENT 904345 6543730
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2301 CONSISTENT 167605 3636833
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2303 CONSISTENT 904365 6542282
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2304 CONSISTENT 904355 6546221
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2305 CONSISTENT 42295 3632868
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2311 CONSISTENT 904375 6551103
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2312 CONSISTENT 904015 6553569
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
2313 CONSISTENT 160565 3659644
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2315 CONSISTENT 904350 6553773
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2317 CONSISTENT 904380 6549408
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
2318 CONSISTENT 904425 6548094
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3101 CONSISTENT 202185 3627157
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3103 CONSISTENT 904360 6559086
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3104 CONSISTENT 904395 6547214
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3105 CONSISTENT 904375 6561635
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'00288bce H'00000000
3107 CONSISTENT 209125 3658050
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3111 CONSISTENT 904370 6543579
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3112 CONSISTENT 904380 6562666
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3113 CONSISTENT 239130 3633922
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3115 CONSISTENT 904350 6554448
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3117 CONSISTENT 904345 6565165
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3201 CONSISTENT 5415 3656444
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3203 CONSISTENT 904375 4837315
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3205 CONSISTENT 904335 6572184
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3207 CONSISTENT 165720 3632310
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3212 CONSISTENT 904380 5813868
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3213 CONSISTENT 26670 3631548
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3215 CONSISTENT 904375 6499539
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3217 CONSISTENT 904365 6494968
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3301 CONSISTENT 645310 3620362
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3303 CONSISTENT 904370 6500736
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3305 CONSISTENT 904400 6504297
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3306 CONSISTENT 904350 6503049
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 H'0027ef0d H'00000000
3313 CONSISTENT 645295 3591113
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3317 CONSISTENT 904355 6494664
H'006d3535 H'03529b6a H'03222ed4 H'0000f823 H'0000a398 ---------- H'00000000
3318 CONSISTENT 904340 6503843

Network Fluctuations are causing DDB inconsistency on the SCCP card that is serving the network card which have link fluctuations.
The card 1107 is one example of how and when the DDB inconsistency appears on system.
This does not mean that this is the ONLY card that causes ALL the DDB inconsistencies on your system.

1. Taking example of the last DDB inconsistent alarm; this is happening for card 3107:

  ****16-03-22 14:59:01****
  8407.0911 ** SYSTEM Dynamic database is inconsistent
  Card 3107

2. From MFC logs; one can see that card 3107 is serving following cards: 3103, 3104, 3105, 3112, 3115, 3117. So need to see if there is any fluctuations occurring on any of these link cards during the DDB inconsistency time or not.

  1367742 0 1367742 0 0 0
  CARD LOC: 3103 LAST 5 SERVERS: 3101,3113,3107,3101,3113

  3671523 0 3671523 0 0 0
  CARD LOC: 3104 LAST 5 SERVERS: 3107,3101,3113,3107,3101

  1184083 0 1184083 0 0 0
  CARD LOC: 3105 LAST 5 SERVERS: 3107,3101,3113,3107,3101

  3671327 0 3671327 0 0 0
  CARD LOC: 3111 LAST 5 SERVERS: 3113,3107,3101,3113,3107

  1186209 0 1186209 0 0 0
  CARD LOC: 3112 LAST 5 SERVERS: 3107,3101,3113,3107,3101

  364341 0 364341 0 0 0
  CARD LOC: 3115 LAST 5 SERVERS: 3101,3113,3107,3101,3113

  374840 0 374840 0 0 0
  CARD LOC: 3117 LAST 5 SERVERS: 3113,3107,3101,3113,3107

3. From rtrv-log; one can see that the card 3115 is having fluctuations during that time. This card is being server by 3107 so there is always a chance that card 3107 can have a DDB inconsistency alarm due to fluctuations.

  ****16-03-22 14:46:07****
  6375.0200 SLK 3115,B6 RCVRY-LKF: link available
  ****16-03-22 14:46:47****
  6501.0085 IP7CONN scgm043 IP Connection Available
  ****16-03-22 14:48:45****
  6843.0087 IP7CONN scgm043 IP Connection manually removed
  ****16-03-22 14:48:46****
  6844.0236 ** SLK 3115,B6 REPT-LKF: not aligned
  ****16-03-22 14:59:01****
  8407.0911 ** SYSTEM Dynamic database is inconsistent
  Card 3107

4. The way system identifies the DDB inconsistency; there is always a chance that a checksum may result in DDB inconsistency during a fluctuation event. However system should be able to recover at its own most of the times and the next DDB audit event may clear the alarm.
Usually; the DDB Audit Timer has been set as 10 minutes. Changing this value may increase/decrease the frequency of DDB alarms but it is not suggested to change this value.

  rtrv-ss7opts
  Command entered at terminal #21.
;

Command Accepted - Processing
  stpclli 16-03-22 22:07:56 EST EAGLE5 45.0.1-64.70.35

  SS7 OPTIONS
  -----------------------
  LSRESTRICT off
  DISCARDTFCI off
  DISCARDTFCN off
  SLSREPLACE no
  SLANCPORGOPC off
  DDBAUDTIMER 10
  SLANLSN off
  MSGPRI2ITUI dflt
  MSGPRI2ITUN dflt

5. There are also Sev1 seen for ss7_tcbc.c Line 511
This shows that there are network events occurring on link cards very frequently.
When there is a link event that creates a Change Over and/or Change Back, the system Audits and Corrects the Dynamic Data Base (DDB) to ensure ALL cards have the same info after the CO or CB, and the STP is free of DDL Unstable alarms.
In R45 the timer value to check it is small so timer can be hit when the system would have recovered properly.
In R46, the timer is extended past time controlled and time diversion procedures, so the UIM should only be occur in 46 when an actual issue is encountered.

Solution

Need to check the physical network and remove any fluctuations occurring on the network.

References

<NOTE:1916949.1> - Module ss7_tcbc.c Line 511 Class 01b1 Severity 1 and Unsolicited Information Message (UIM) 1461 INFO Link status on card needs re-sync

Attachments

This solution has no attachment