MP stuck in X state and ha.mystate gives unexpected "failed to exit in 15000 msec timeout" output

Asset ID:	1-72-2399744.1
Update Date:	2018-05-16
Keywords:

Solution Type Problem Resolution Sure

Solution 2399744.1 : MP stuck in X state and ha.mystate gives unexpected "failed to exit in 15000 msec timeout" output

Applies to:

BNS Platform Hardware - Version DSR 7.0.1 and later
Information in this document applies to any platform.

Symptoms

Routes expected to be deployed were showing as configured

Connections associated with MP were were not coming up

ha.mystate and prod.state showed unexpected/unwanted output

[root@MPSERVER filemgmt]# ha.mystate
0321:144041.832 TR-V pid: 39253 failed to exit in 15000 msec timeout [39256/SigCtl.cxx:300]

[root@MPSERVER filemgmt]# prod.state
...prod.state (RUNID=00)...
...getting current state...
Current state: X (product under procmgr)

Changes

Cause

MP not able to communicate with NOAM because a static route to the network the NOAM is on was not configured on the MP and the default route
was not able to route to the NOAM (default route was on XSI network).

This was seen with the output of "ip route" on the bad server as compared to a healthy server. Only one route was missing from the bad server and this was the route to the NOAM

Solution

Add the missing route to the MP then reboot the MP:

sudo /usr/TKLC/plat/bin/netAdm add --route=net --address= <NOAM network ID> --netmask==<netmask> --gateway=<gateway> --device=<device>

The values for the above variables can be determined by looking at the healthy MP.

Reboot MP i.e. "sudo init 6"

After reboot check state of MP with the following commands:

sudo syscheck - should give all "OK"s

ra.stat - should be empty

ping <noam vip> - should give successfully pings, assuming ICMP is allowed in customer network

ha.mystate - should show same output as healthy mate server

prod.state - should show same output as healthy mate server

Attachments

This solution has no attachment