NSP ProTrace Shows Message Box "Data set may be incomplete - Unable to connect to Storage server/s x.y.z.t"

Asset ID:	1-75-1905754.1
Update Date:	2018-04-06
Keywords:

Solution Type Troubleshooting Sure

Solution 1905754.1 : NSP ProTrace Shows Message Box "Data set may be incomplete - Unable to connect to Storage server/s x.y.z.t"

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 7.1 and later
Information in this document applies to any platform.

Purpose

When doing a query in NSP ProTrace application, a message box with following text is shown:

Data set may be incomplete

- Unable to connect to Storage server/s x.y.z.t

Where x.y.z.t is the IP of IXP xDR server (also known as datawarehouse; later called DWS in this document).

This document describes various steps to troubleshoot this.

Troubleshooting Steps

Main cause is that one of the DWS server is not reachable or Oracle database is down.

First test, database availability:

As oracle, on DWS server, run sqlplus login/password. Output is expected to be the same, but if you see ORA-01034, there is a problem:
SQL*Plus: Release 10.2.0.5.0 - Production on Thu Jul 10 10:23:36 2014

Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory

Enter user-name:

If this show SQL> prompt, it means database is up.

Second test, is to check database reachability through listener:

As oracle, on DWS server, run sqlplus login/password@hostname/SID
SID is usually IXP
Expected output (version might differ):

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Jul 10 10:17:02 2014

Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production
With the Partitioning, Data Mining and Real Application Testing options

SQL> select sysdate from dual;

SYSDATE
---------
10-JUL-14

If you see ORA-12514 like below you have to check listener.log:

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Most common causes of "ORA-01034: ORACLE not available" are:

Hardware failure
On 32 bits OS, Oracle database shared memory issue ORA-04031 unable to allocate xxxx bytes of shared memory
Full partitions
Database files corruption that can be cause by point 1, 2 or 3

Hardware failure

As root, on the DWS server, run syscheck command to check any hardware failure (harddrive, controller, controller batteries, ...).

Any hardware failure must be fixed. After that, database reachability must be retested.
If no errors are reported, this is most likely not the problem. Go to next step.

ORA-04031

On 32 bits OS, check in Oracle DB alert log (alert_IXP.log; path name differ between Oracle DB version) if any recent ORA-04031 errors can be found. If yes, there are two options depending on the oracle version. "select * from v$version;" can be run to confirm this.

To avoid futur occurence, it is highly recommended to convert DWS to 64 bits.

Oracle 10g

Patch 8202632 must be installed.

Oracle 11g

Patch 15969429 must be installed along with OPatch tool 6880880 (p6880880_112000_LINUX.zip).

Full partitions

Full partitions can prevent Oracle DB to run properly has it cannot log new connections tries. This can have multiple causes:

Oracle DB logs growing fast
- Check alert_IXP.log if there are any repeating message. If yes, search this message in My Oracle Support
Oracle DB logs not rotating properly (alert, listener, bdump trc, cdump trc, ...). Check in /var/log/messages logs related to logrotate

Database files corruptions

This can be verified in alert_IXP.log startup sequence.

If datafile corruption is confirmed, solution depends on which tablespace corruption is. If it is on:

DATA_IND, which contains indexes, it can be recovered. Open SR on database product to involve support. https://mediawiki.us.oracle.com/pic_kndb/index.php/ORACLE_rebuild_corrupted_index_tablespace
Any other tablespaces, database will have to be fully rebuilt unless backup is available

Attachments

This solution has no attachment