Heartbeat Loss Alarm( UAM 0581)

Asset ID:	1-72-2393787.1
Update Date:	2018-05-02
Keywords:

Solution Type Problem Resolution Sure

Solution 2393787.1 : Heartbeat Loss Alarm( UAM 0581)

Applies to:

Oracle Communications EAGLE (Software) - Version EAGLE 46.5 and later
Information in this document applies to any platform.

Symptoms

There are many ENET cards in STP reporting the UAM 0581. For some of the cards the alarm persistence while for few of the cards it keep fluctuating at every few seconds.

Following was seen in the trouble reports.

1897.0581 * CARD 1301 IPSG Loss of Heartbeat
1910.0581 * CARD 1316 IPSG Loss of Heartbeat
1862.0581 * CARD 2318 IPSG Loss of Heartbeat
3507.0581 * CARD 3205 IPSG Loss of Heartbeat
1831.0581 * CARD 1101 IPSG Loss of Heartbeat

Following was seen in the logs of UAM 581 ( For some of the the cards the alarm fluctuating)

> rtrv-log:dir=bkwd:snum=0581

xxxx xx-yy-1zz hh:mm:ss zzz EAGLE 46.5.0.0.1-70.39.1
  rtrv-log:dir=bkwd:snum=0581
  Command entered at terminal #24.
;

Command Accepted - Processing
xxxx xx-yy-1zz hh:mm:ss zzz EAGLE 46.5.0.0.1-70.39.1
  Card 1115; SYS REL= 46.5.0.0.1-70.39.1; STP CLLI= xxxx; Timezone= zzz

  ****18-04-12 01:42:42****
  7773.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:42:25****
  7336.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:42:12****
  6996.0581 * CARD 3311 IPSG Loss of Heartbeat
  ****18-04-12 01:41:42****
  6253.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:40:44****
  9787.0581 * CARD 3311 IPSG Loss of Heartbeat
  ****18-04-12 01:39:40****
  8172.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:39:15****
  7530.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:38:53****
  6964.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:37:08****
  9315.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:36:58****
  9063.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:36:42****
  8654.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:36:26****
  8237.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:35:40****
  7092.0581 * CARD 1206 IPSG Loss of Heartbeat
  ****18-04-12 01:35:20****
  6563.0581 * CARD 3311 IPSG Loss of Heartbeat
  ****18-04-12 01:34:29****
  5262.0581 * CARD 3311 IPSG Loss of Heartbeat
  UAM Report terminated - max. or num= count reached
  END OF LOG REPORT.

Changes

No changes done in the system configuration( both Eagle and PIC).

Cause

The ENET cards were not able to ping one of the IMF address.

Solution

1. rept-stat-mon:loc=xxxx:type=fcs shows the IMF addreses. These are the internal address used between ENET and PIC IMF.

IMF CONNECTION STATUS TABLE
---------------------------------------------------------------------------
IPADDRESS ALM ID ASSOC NAME PKT CNT SERVICE MODE
---------------------------------------------------------------------------
aa.bb.cc.dd (A) 582 xxxxxxx 9359 Copy All MSUs
xx.yy.zz.qq (B) 582

2. Ping to one of the IMF address from the ENET Card reporting the UAM 0581 was getting failed.

PING aa.bb.cc.dd: 56 data bytes
64 bytes from aa.bb.cc.dd: icmp_seq=0. time=0. ms
64 bytes from aa.bb.cc.dd: icmp_seq=1. time=0. ms
64 bytes from aa.bb.cc.dd: icmp_seq=2. time=0. ms
----aa.bb.cc.dd PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/0/0

PING xx.yy.zz.qq: 56 data bytes

PING: no answer from xx.yy.zz.qq

PING command complete

3. PIC side no abnormality or alarm seen. For the ENET cards for which the UAM 0581 was fluctuating the IMF switch was seen generating the below errors. No Local port error (CRC etc.) seen on ENET cards.

0413:024135.108 TR-V FcSourceUnavailable alarm on bond0.101 with Eagle card: xx.yy.zz.qq. Alarm now raised for these Eagle cards: 17!, xx.yy.zz.qq [394/FcInterface.C:206]
0413:024147.103 TR-V Clearing FastCopy Source Unavailable Alarm on bond0.101 with Eagle Card IP xx.yy.zz.qq [394/FcHeartBeat.C:253]
0413:024147.103 TR-V FcSourceUnavailable alarm cleared on bond0.101 with Eagle card(s): xx.yy.zz.qq [394/FcInterface.C:256]
0413:024151.107 TR-V Raising FastCopy Source/Destination Unavailable Alarm on bond0.101 with Eagle Card IP xx.yy.zz.qq [394/FcHeartBeat.C:247]
0413:024151.107 TR-V FcSourceUnavailable alarm on bond0.101 with Eagle card: xx.yy.zz.qq. Alarm now raised for these Eagle cards: 17!, xx.yy.zz.qq [394/FcInterface.C:206]
0413:024152.105 TR-V Clearing FastCopy Source Unavailable Alarm on bond0.101 with Eagle Card IP xx.yy.zz.qq [394/FcHeartBeat.C:253]
0413:024152.105 TR-V FcSourceUnavailable alarm cleared on bond0.101 with Eagle card(s): xx.yy.zz.qq [394/FcInterface.C:256]
0413:024156.106 TR-V Raising FastCopy Source/Destination Unavailable Alarm on bond0.101 with Eagle Card IP xx.yy.zz.qq [394/FcHeartBeat.C:247]
0413:024156.107 TR-V FcSourceUnavailable alarm on bond0.101 with Eagle card: xx.yy.zz.qq. Alarm now raised for these Eagle cards: 17!,xx.yy.zz.qq [394/FcInterface.C:206]

4. Multiple ways helps to clear the alarm which includes the following.

Hard reset to ENET card
Resync from IMF
Switch port change at PIC IMF side
Change the cable between IMF and ENET Card ports.

There is no impact on monitoring the links by PIC ,as we can see that the MSUs are being copied for the links which are configured on such cards (which were reporting the UAM 0581).

Attachments

This solution has no attachment