PIC ProTrace Session(S) Can Be Stopped If `Prod.State` Shows Output Of "Current State: Z"

Asset ID:	1-72-2297798.1
Update Date:	2017-08-17
Keywords:

Solution Type Problem Resolution Sure

Solution 2297798.1 : PIC ProTrace Session(S) Can Be Stopped If `Prod.State` Shows Output Of "Current State: Z"

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 10.1.5 and later
Information in this document applies to any platform.

Symptoms

prod.state output on the PIC IXP or xMF server, when executed as user cfguser, is "Current state: Z" (normal output is state A or B).

$ prod.state
...prod.state (RUNID=00)...
...getting current state...
Current state: Z (product under procmgr)

If `prod.state status is "Z" on a server in an PIC IXP or IMF subsystem,

iFoStat output, when run as user cfguser from one of the other servers in the same subsystem, will show the affected server as Out Of Service ("OOS"). The expected state of the server in iFoStat output is In Service ("IS").
This issue can cause PIC ProTrace session(s) whose processes reside on an affected IXP server to stop updating. This could be confirmed in the ProTrace application of the PIC Web GUI or via the ViewSessionFlow.sh script on the IXP subsystem.

On the affected server, syscheck output points to a problem with NTP, such as "NTP has never been synced in current configuration."

# syscheck
Running modules in class disk...
OK

Running modules in class hardware...
OK

Running modules in class net...
OK

Running modules in class proc...
* ntp: FAILURE:: MAJOR::3000020000000000 -- Server NTP Daemon Never Synchronized
* ntp: FAILURE:: NTP has never been synced in current configuration
One or more module in class "proc" FAILED

Running modules in class system...
OK

Running modules in class upgrade...
OK

LOG LOCATION: /var/TKLC/log/syscheck/fail_log
#

Changes

Cause

The configured NTP time source is not valid.

ntpdate -q ntpserver1 and ntpq -p or ntpq -pn indicate that the server is not sync'd to the NTP time source and may have never been sync'd.

$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
ntpserver1 .INIT. 16 u - 64 0 0.000 0.000 0.000

$ ntpdate -q ntpserver1
server 10.103.56.1, stratum 0, offset 0.000000, delay 0.00000
26 Jul 16:15:13 ntpdate[15479]: no server suitable for synchronization found

On the affected server, when trying to start the IXP or xMF applicaton as user cfguser via `prod.start,` the action fails and, afterward, `prod.state` remains in state "Z." The output of `iqt -p AppEventLog.0` has an event related to prod.start similar to "Untrusted Time Upon Initialization GN_TIMECHG: Time untrusted on initialization. Check NTP configuration."

*C 07/26/2017 16:11:14.101 0 prod.start Untrusted Time Upon Initialization GN_TIMECHG: Time untrusted on initialization. Check NTP configuration.
.. 07/26/2017 16:11:14.917 1 procmgr S/W Status
GN_STATECHG/INF Going to state='Z' from state=n/a [procmgr.cxx:1958]
^^ [15237:procmgr.cxx:1960]

The procmgr trace file on the affected server shows the transition to state Z. As user cfguser, `tr.cat procmgr` output mentions "Going to state='Z'" when the action occurred.

07/26/2017 16:11:14 procmgr#31001{S/W Status}
.. GN_STATECHG/INF Going to state='Z' from state=n/a [procmgr.cxx:1958]
^^ [15237:procmgr.cxx:1960]

Solution

Update the affected server's NTP source to a valid time source, correct the server's time, and start the server's PIC IXP or xMF application's IDB processes.

On the affected server, as root user...

To obtain initial outputs for later reference, execute:
# grep ntpserver1 /etc/hosts
# ntpdate -q ntpserver1
# ntpq -p
# ntpq -pn
# date
Stop the ntpd service:
# service ntpd stop
Use the platcfg utility to update the NTP time source. In the platcfg utility, modify the IP of the existing 'ntpserver1' entry to a valid IP address and then exit the platcfg utility.
Update the server's time to that of the new NTP time source:
# ntpdate ntpserver1
# ntpdate ntpserver1
Start the ntpd service:
# service ntpd start
Obtain final outputs for comparing to initial outputs:
# grep ntpserver1 /etc/hosts
# ntpdate -q ntpserver1
# ntpq -p
# ntpq -pn
# date

On the affected server, as cfguser user, start the IXP or xMF application via `prod.start`:

$ prod.start

Attachments

This solution has no attachment