IXP Datawarehouse Connection Error Alarm

Asset ID:	1-72-1954852.1
Update Date:	2016-08-23
Keywords:

Solution Type Problem Resolution Sure

Solution 1954852.1 : IXP Datawarehouse Connection Error Alarm

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 6.6.4 and later
Information in this document applies to any platform.
In PIC Storage servers are used to store data called xDRs of the session.Each IXP subsystem can have single or multiple of storage servers in a group called DWH. xDR or storage server can be found by following steps:

1. Login to NSP GUI

2. Select CCM-->Mediation-->Sites-->Expand any site-->expand DWH or select storage after expanding ixpxxx, where ixpxxx is subsystem name.

Store process of each session running on any server(PDU or base server) will access storage server to store data for future and we can retrieve these data after executing query in ProTrace.

Symptoms

The symptoms of the Datawarehouse connection error or Pool connection error will be the followings:

Session will start lagging from real time i.e. end time in ProTrace will be x hours behind the real time.
System Alarm or ProAlarm will have alarm like 'Datawarehouse connection error'.
Unable to get data in the ProTrace after executing query.

Cause

'Datawarehouse Connection Error' or 'Pool Connection Error' will be generated when a store process is not able to access the Oracle Database on storage server , they can been seen in ProAlarm and in the log of the store process:

1201:160350.384 IXP3 Send alarm: Datawarehouse connection error User IXP is not connected to database ixpxxx-1a_DWH [27161/DmAlarm.cpp:274]
1201:160400.483 IXP3 Send alarm: Pool connection error.Pool connection on ixpxxxx_Pool [26395/DmAlarm.cpp:274

Solution

Following might be the root cause for this issue:

Oracle Database is down .i.e. oracle services are not running on the server. This will happen when listener service is not up or Database encounter any error (hardware or software).
Following commands can be used to start Oracle services:

service TKLCoracledb restart
or

service TKLCoracledb stop
service TKLCoracledb start
Oracle database is up on the store server but store processes are not able to connect to the Oracle Database and generating following error message in the log:
1201:160350.384 IXP3 Send alarm: Datawarehouse connection error User IXP is not connected to database ixpxxx-1a_DWH [27161/DmAlarm.cpp:274]
1201:160400.483 IXP3 Send alarm: Pool connection error.Pool connection on ixpxxxx_Pool [26395/DmAlarm.cpp:274]

Following might be possible cause for the issue:

SMART ARRAY CONTROLLER is faulty
BATTERY attached to SMART ARRAY CONTROLLER is Faulty.

To analyze above perform following steps:

Case a: On Rack Mount server, login as root user and execute command hpacucli ctrl all show status

[root@ixp0001-1a ~]# hpacucli ctrl all show status
Smart Array P410i in Slot 0 (Embedded)
   Controller Status: OK
   Cache Status: OK
   Battery/Capacitor Status: FAILED

Case b: On Blade Server, login to san controller and check the status of controller. Following messages may generate in the

RAID Controller Status: WARNING: RAID Controller A supercap failed - replace supercap. The write-back cache is now disabled.

Attachments

This solution has no attachment