Unable To Apply Changes In IMF - Failed to Rediscover Frame and Position

Asset ID:	1-72-1985430.1
Update Date:	2016-08-29
Keywords:

Solution Type Problem Resolution Sure

Solution 1985430.1 : Unable To Apply Changes In IMF - Failed to Rediscover Frame and Position

Applies to:

Oracle Communications Performance Intelligence Center (PIC) Software - Version 6.6.4 and later
Information in this document applies to any platform.
This issue is related to an old Tekelec Bug --> PR 193692 alarmMgr stopped unexpectedly causing the JMX to fail to start.

Symptoms

Unable to Apply changes in ***_IMF

IMF Failure

Failed to rediscover frame and position

Cause

This issue relies on an old Tekelec Bug PR193692

application.log contains :

2010-04-24 16:47:39,799 - ERROR - NSPWL1 - CONFIG - <internal> - SYSTEM - filterApplications: could not discover application on host: 10.50.7.5 - com.steleus.nsp.config.common.CentConfException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.5.1.5; nested exception is: java.net.ConnectException: Connection refused] at com.steleus.nsp.config.common.JmxConnectionManager.<init>(JmxConnectionManager.java:46) at com.steleus.nsp.config.common.JmxServ.filterApplications(JmxServ.java:578) at com.steleus.nsp.config.system.ejb.SystemFacadeBean.rediscoverFrameAndPositonAndVIP(SystemFacadeBean.java:4325) at com.steleus.nsp.config.system.ejb.SystemFacade_2t3xeq_EOImpl.rediscoverFrameAndPositonAndVIP(SystemFacade_2t3xeq_EOImpl.java:3677) at com.steleus.nsp.config.system.ejb.SystemFacade_2t3xeq_EOImpl_WLSkel.invoke(Unknown Source) at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174) at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:335) at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252) at com.steleus.nsp.config.system.ejb.SystemFacade_2t3xeq_EOImpl_1001_WLStub.rediscoverFrameAndPositonAndVIP(Unknown Source) at com.tekelec.cssg.nsp.proadmin.server.model.IASEquipmentServiceModel.rediscoverFramePosition(IASEquipmentServiceModel.java:3113) at com.tekelec.cssg.nsp.proadmin.server.servlets.IASEquipmentServiceImpl.rediscoverFramePosition(IASEquipmentServiceImpl.java:1186) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:528) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:265) Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.50.7.5; nested exception is:

It usually means that JMX or mysql process is not reachable on the mentionned IP.

<p >*Solution
<span >Provide step by step instructions on how to solve the problem.

Check that you can reach VIP on JMX RMI port (1099) from NSP Weblogic 1 and Weblogic 2 server
Check that JMX agent is running by trying to access JMX web page (http://IP_OF_IMF:6969). IP_OF_IMF is VIP

Solution

This issue is due to alarmMgr process stuck:

mftek1-1b:/export/home/cfguser <strong>ps -ef | grep alarmMgr</strong>
root 14388 1 0 Aug25&nbsp;? 00:00:00 /usr/bin/perl /usr/TKLC/plat/bin/alarmMgr --daemon
cfguser <strong>26473</strong> 26472 0 09:59 pts/0 00:00:01 /usr/bin/perl /usr/TKLC/plat/bin/alarmMgr --alarmStatus
cfguser 31120 25414 0 10:25 pts/0 00:00:00 grep alarmMgr
mftek1-1b:/export/home/cfguser <strong>kill 26473</strong>
mftek1-1b:/export/home/cfguser <strong>ps -ef | grep alarmMgr</strong>
root 14388 1 0 Aug25&nbsp;? 00:00:00 /usr/bin/perl /usr/TKLC/plat/bin/alarmMgr --daemon
cfguser 31278 25414 0 10:26 pts/0 00:00:00

You first need to kill the alarmMgr status process with this command as root:

ps -ef | alarmStatus | awk '{print "kill -9 ",$2}' | ksh

As a root user…

Check that it is stopped: alarmMgr --status

Then start the alarmMgr daemon with this command alarmMgr daemon.

Start it again alarmMgr --daemon

Check that it is running: alarmMgr --status

Restart jmx agent and check, after 1 or 2 minutes, that JMX web page is accessible again

Attachments

This solution has no attachment