EMS :: Secondary Netboss Server is Active But No Alarms Received On It

Asset ID:	1-72-2016432.1
Update Date:	2017-02-15
Keywords:

Solution Type Problem Resolution Sure

Solution 2016432.1 : EMS :: Secondary Netboss Server is Active But No Alarms Received On It

Applies to:

Oracle Communications EAGLE (Hardware) - Version EMS 43.x and later
Tekelec

Symptoms

On : EMS 43.x version, EMS (Hardware)

Secondary Netboss Server is Active But No Alarms Received On It

ERROR
-----------------------
Alarms not being updated on secondary EMS server that is being used as the active srver.

STEPS
-----------------------
The issue can be seen in the Fault Management System Netboss view.

BUSINESS IMPACT
-----------------------
Due to this issue, alarm monitoring of Eagle alarms can be impacted.

Cause

The cause of the issue is that on primary server; the EMS application is running.
The primary should not be running as it is not stable. Currently EMS02 should be acting as active server.
Under normal circumstances; the primary server should be active while secondary should be standby.
However; if the EMS application is manually stopped on primary server; then alarms should be reaching secondary server.

Solution

Perform Following Steps:

1. Stop EMS application and Oracle DB on primary server:

nbprimary [netboss]/opt/home/netboss% sudo /etc/init.d/netboss stopall

Password: <Enter netboss user password>
/usr/bin/pkill -15 -x netbossmenu
Waiting 10 seconds to allow processes to die.
Waiting 10 seconds to allow processes to die.
Attempting to stop: /u01/app/netboss/admin/bin/rocportd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
rocportd is not running.
Attempting to stop: /u01/app/netboss/local/bin/nbtf (Modelist: [PRIMARY])
nbtf is not running.
Attempting to stop: /u01/app/netboss/admin/bin/check-failover (Modelist: [NEVER_SHUTDOWN_WITHMODE])
/usr/bin/kill -15 5728
Checking for spawned process: IPRINTD
iprintd is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/send2roc -port 9090 -host ptclstpnbb -stat -info (Modelist: [])
send2roc is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/scanpolld (Modelist: [])
scanpolld is not running.
Attempting to stop: /u01/app/netboss/local/bin/RSNMPcatch (Modelist: [NEVER_SHUTDOWN_WITHMODE])
RSNMPcatch is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/proengine (Modelist: [primary])
/usr/bin/kill -15 1770
Checking for spawned process: PARSE
/usr/bin/kill -15 1840
/usr/bin/kill -15 1830
kill: 1830: no such process
/usr/bin/kill -15 1939
/usr/bin/kill -15 1967
kill: 1967: no such process
/usr/bin/kill -15 1917
kill: 1917: no such process
/usr/bin/kill -15 1904
kill: 1904: no such process
/usr/bin/kill -15 1998
kill: 1998: no such process
/usr/bin/kill -15 1980
kill: 1980: no such process
/usr/bin/kill -15 1983
kill: 1983: no such process
/usr/bin/kill -15 1958
kill: 1958: no such process
/usr/bin/kill -15 1881
kill: 1881: no such process
/usr/bin/kill -15 1921
kill: 1921: no such process
/usr/bin/kill -15 1845
kill: 1845: no such process
/usr/bin/kill -15 1977
kill: 1977: no such process
/usr/bin/kill -15 1999
/usr/bin/kill -15 1995
Attempting to stop: /u01/app/netboss/bin/netbossonmtrapd (Modelist: [PRIMARY])
netbossonmtrapd is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/local/lib/check_scripts/master_script standby (Modelist: [standby])
/usr/bin/kill -15 1377
Checking for spawned process: CHECKNETBOSS
check-netboss is not running.
Attempting to stop: /u01/app/netboss/local/lib/check_scripts/master_script master (Modelist: [primary])
master_script is not running.
Checking for spawned process: CHECKNETBOSS
check-netboss is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/loghistd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
/usr/bin/kill -15 1316
Attempting to stop: /u01/app/netboss/bin/imd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
/usr/bin/kill -15 1264
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/gatewaymgr start (Modelist: [primary])
Executing: /u01/app/netboss/bin/gatewaymgr stop
Checking for spawned process: ALERTGATEWAY
alertgateway is not running.
Attempting to stop: /u01/app/netboss/bin/esid (Modelist: [PRIMARY])
esid is not running.
Checking for spawned process: ESISESSION
esi_session is not running.
Checking for spawned process: ESIXMSG
esi_xmsg is not running.
Attempting to stop: /u01/app/netboss/bin/enmd (Modelist: [])
enmd is not running.
Checking for spawned process: ENMAPP
enmapp is not running.
Checking for spawned process: ENMRX
enmrx is not running.
Checking for spawned process: ENMTX
enmtx is not running.
Checking for spawned process: ENMSOCK
enmsock is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/btrapd (Modelist: [])
btrapd is not running.
Attempting to stop: /u01/app/netboss/bin/beeper /dev/cua/b 9600 -wait (Modelist: [NEVER_SHUTDOWN_WITHMODE])
beeper is not running.
Attempting to stop: /u01/app/netboss/bin/alertman (Modelist: [NEVER_SHUTDOWN_WITHMODE])
/usr/bin/kill -15 1092
Attempting to stop: /u01/app/netboss/bin/roc (Modelist: [NEVER_SHUTDOWN_WITHMODE])
/usr/bin/kill -15 1039
Checking for spawned process: QUEUERPC
/usr/bin/kill -15 1094
/usr/bin/kill -15 1318
/usr/bin/kill -15 1823
Waiting 10 seconds for processes to stop.
Attempting to stop: /u01/app/netboss/admin/bin/rocportd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
rocportd is not running.
Attempting to stop: /u01/app/netboss/local/bin/nbtf (Modelist: [PRIMARY])
nbtf is not running.
Attempting to stop: /u01/app/netboss/admin/bin/check-failover (Modelist: [NEVER_SHUTDOWN_WITHMODE])
check-failover is not running.
Checking for spawned process: IPRINTD
iprintd is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/send2roc -port 9090 -host ptclstpnbb -stat -info (Modelist: [])
send2roc is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/scanpolld (Modelist: [])
scanpolld is not running.
Attempting to stop: /u01/app/netboss/local/bin/RSNMPcatch (Modelist: [NEVER_SHUTDOWN_WITHMODE])
RSNMPcatch is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/proengine (Modelist: [primary])
proengine is not running.
Checking for spawned process: PARSE
parse is not running.
Attempting to stop: /u01/app/netboss/bin/netbossonmtrapd (Modelist: [PRIMARY])
netbossonmtrapd is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/local/lib/check_scripts/master_script standby (Modelist: [standby])
master_script is not running.
Checking for spawned process: CHECKNETBOSS
check-netboss is not running.
Attempting to stop: /u01/app/netboss/local/lib/check_scripts/master_script master (Modelist: [primary])
master_script is not running.
Checking for spawned process: CHECKNETBOSS
check-netboss is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/loghistd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
loghistd is not running.
Attempting to stop: /u01/app/netboss/bin/imd (Modelist: [NEVER_SHUTDOWN_WITHMODE])
imd is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/gatewaymgr start (Modelist: [primary])
Executing: /u01/app/netboss/bin/gatewaymgr stop
Checking for spawned process: ALERTGATEWAY
alertgateway is not running.
Attempting to stop: /u01/app/netboss/bin/esid (Modelist: [PRIMARY])
esid is not running.
Checking for spawned process: ESISESSION
esi_session is not running.
Checking for spawned process: ESIXMSG
esi_xmsg is not running.
Attempting to stop: /u01/app/netboss/bin/enmd (Modelist: [])
enmd is not running.
Checking for spawned process: ENMAPP
enmapp is not running.
Checking for spawned process: ENMRX
enmrx is not running.
Checking for spawned process: ENMTX
enmtx is not running.
Checking for spawned process: ENMSOCK
enmsock is not running.
Attempting to stop: (Modelist: [])
Attempting to stop: /u01/app/netboss/bin/btrapd (Modelist: [])
btrapd is not running.
Attempting to stop: /u01/app/netboss/bin/beeper /dev/cua/b 9600 -wait (Modelist: [NEVER_SHUTDOWN_WITHMODE])
beeper is not running.
Attempting to stop: /u01/app/netboss/bin/alertman (Modelist: [NEVER_SHUTDOWN_WITHMODE])
alertman is not running.
Attempting to stop: /u01/app/netboss/bin/roc (Modelist: [NEVER_SHUTDOWN_WITHMODE])
roc is not running.
Checking for spawned process: QUEUERPC
queuerpc is not running.

2. Stop Oracle DB on primary server:

EMSPrimary[netboss]/opt/home/netboss% sudo /etc/init.d/oracle stop

Stopping Oracle...

SQL*Plus: Release 9.2.0.7.0 - Production on Mon May 25 13:18:54 2015

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

nolog> Connected.
nolog> Database closed.
Database dismounted.
ORACLE instance shut down.
nolog> Disconnected from Oracle9i Release 9.2.0.7.0 - Production
JServer Release 9.2.0.7.0 - Production
Database "NBPRD" shut down.

LSNRCTL for Solaris: Version 9.2.0.7.0 - Production on 25-MAY-2015 13:19:01

Copyright (c) 1991, 2002, Oracle Corporation. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=1521)))
The command completed successfully

2. Restart EMS application on secondary server:

sudo /etc/init.d/netboss stop
sudo /etc/init.d/netboss start

Attachments

This solution has no attachment