OpenConnection Failure with Error Message '-10' in IXP Process Logs

Asset ID:	1-72-1950077.1
Update Date:	2017-11-08
Keywords:

Solution Type Problem Resolution Sure

Solution 1950077.1 : OpenConnection Failure with Error Message '-10' in IXP Process Logs

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 6.6.4 and later
Information in this document applies to any platform.
Each session is getting data from multiple streams.Streams are created after associating pdu dataflow,created on different xMFs, associated with build of the sessions. IF all streams are marked critical and due to issue if session will not get any data from a particular stream then session will stop updating.

Symptoms

Build, Operate or Store process of many sessions were showing error message "OpenConnection failure with error message '-10' on stream name" due to which sessions stopped updating.

Cause

Build log of the sessions were not updating and getting following error message in its log:

1116:103742.658 DTS2 OpenConnection(): Connecting to IP address ip_address_of_IMF failed, port is 2222 and stream name is 'stream_name' [25937/DataTransportService/DtsConsumer/src/DtsConsumer.cpp:127]
1116:103742.658 DTS2 OpenStream(): OpenConnection failure with error message '-10' on stream name 'stream_name' [25937/DataTransportService/DtsConsumer/src/DtsConsumer.cpp:459]
1116:103747.699 DTS2 OpenStream(): Passed too old timestamp: 1416073774, tolerance: 21600, changed to: 1416092867, StreamName: stream_name [25937/DataTransportService/DtsConsumer/src/DtsConsumer.cpp:469]
STREAM are logical connection which is providing data from xMF(IMF or PMF) to build of the sessions.In PIC, STREAMS,connecting xMF to IXP, are after associating PDU DATAFLOW to the BUILD of the session. A session can have multiple of streams created after associating PDU DATAFLOWS from different xMF servers.
Streams can be marked critical or non-critical. If all streams are marked CRITICAL and then build of the session will stop processing data from all streams if it will not be able to process data from either of any stream. The log shared in point 1 is showing that build of the a session is not able to get data from one of the stream named stream_name created on IMF ip_address_of_IMF.

Solution

xMF

There are multiple of root cause for error message "Connecting to IP address ip_address_of_IMF/PMF failed, port is 2222 and stream name is 'stream_name1'" like:

Congestion between IMF on which pdu dataflow is created and IXP on which build of the sessions is created.
IDB discrepancy in xMF due to which pduServer0 or procMgr of xMF is not working properly.
Apply changes after modification in session or PDU DATAFLOW did not propagated to xMF due to which stream did not created on xMF
xMF is showing OUT OF STATE (OOS) due to any hardware failure.

Find below an example for your reference. After getting error message in the build log as showing in point 1. IMF ip_address_of_IMF was showing OOS while doing the health check of IMF subsystem in which IMF ip_address_of_IMF belongs.All spare servers were also showing OOS.

+-----------+-------+-----+-----------+---------+------+---------------------+
| name | state | loc | role | mGroup | assg | HbTime |
+-----------+-------+-----+-----------+---------+------+---------------------+
| IMF-1A | IS | 1A | Slave | MG_1 | 236 | 2014-12-02 08:10:38 |
| IMF-1B | OOS | 1B | Slave | | 201 | 2014-12-02 08:10:38 |
| IMF-1C | OOS | 1C | Slave | | 0 | 2014-11-16 10:41:54 |
| IMF-1D | IS | 1D | ActMaster | MG_7 | 104 | 2014-12-02 08:10:38 |
| IMF-1E | IS | 1E | StbMaster | MG_2 | 163 | 2014-12-02 08:10:38 |
| IMF-1F | OOS | 1F | Slave | | 0 | 2014-11-30 09:55:39 |
| IMF-2A | IS | 2A | Slave | MG_6 | 25 | 2014-12-02 08:10:38 |
| IMF-2B | IS | 2B | Slave | MG_4 | 149 | 2014-12-02 08:10:38 |
| IMF-2C | IS | 2C | Slave | MG_5 | 106 | 2014-12-02 08:10:36 |
+-----------+-------+-----+-----------+---------+------+---------------------+

All command running on IMF-1F server were going in hang state. We were getting following error message in syscheck:

Running modules in class disk...
* smart: FAILURE:: MINOR::5000000000000800 -- Server Disk Self Test Warning
* smart: FAILURE:: Failure type: CurrentPendingSector, Failure message: Device: /dev/twe0 [3ware_disk_00], 2 Currently unreadable (pending) sectors, First occurred: Wed Nov 19 20:43:21 2014 IST.
One or more module in class "disk" FAILED

/var/log/messages of IMF-1F was showing following error message. One hard disk SDA was faulty on this server due to this all process were going in hang state and routeDisp was not showing any stream pushing data to its destination (IXP).

Nov 30 13:12:07 cust-1a last message repeated 2 times
Nov 30 13:13:08 cust-1a last message repeated 2 times
Nov 30 13:13:22 cust-1a smartd[14337]: Device: /dev/twe0 [3ware_disk_00], 2 Currently unreadable (pending) sectors

After replacement of faulty hard disk, streams created on this IMF started pushing data to sessions created on different IXP subsystem and build of the sessions stopped generating log as showing point1.

IXP

IP address can also be an IXP, if it is a Operate or Store process, with the same error message:

Connecting to IP address IXP_IP_add failed, port is 2222 and stream name is 'STREAM_NAME'

and

OpenConnection failure with error message -'10' on stream name 'STREAM_NAME'

This may happen if an IXP server was stopped, but the IDB table "StreamHistory" remains the reference to the server that was stopped. This situation prevents the session to be updated.

To fix the issue, it is needed to edit the IDB table "StreamHistory" on the Master server and to remove all lines related to the server that has been stopped.

Attachments

This solution has no attachment