Exalogic Virtual: Guest vServers Shutdown, Start Jobs run indefinitely from EMOC without completing even though the vServer is stopped/started at the OVS level

Asset ID:	1-72-2236562.1
Update Date:	2017-11-01
Keywords:

Solution Type Problem Resolution Sure

Solution 2236562.1 : Exalogic Virtual: Guest vServers Shutdown, Start Jobs run indefinitely from EMOC without completing even though the vServer is stopped/started at the OVS level

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.0.0 and later
Exalogic Elastic Cloud X5-2 Eighth Rack - Version X5 to X5 [Release X5]
Linux x86-64
Oracle Virtual Server x86-64

Symptoms

In Exalogic Virtual racks, issue of Guest vServers shutdown and start Jobs running indefinitely without completing is noticed. Checking the vServer status on the Compute Node on which shutdown or startup operation is triggered we notice that the vServer is already stopped or started (depending on the task whether it is stop or start). But the Job from EMOC is still running at OVM Manager task of shutting down or starting the Virtual Machine.

Checking the OVM Manager job logs we notice that corresponding OVM jobs for shutdown/startup are also still in progress without completing even though the actual vServer already started (or) shutdown on OVS level.

Cause

This issue happens due to ovs-agent communication issues between OVM Manager and OVS Compute Nodes. EMOC tasks like start or stop of vServers will be provisioned to OVM Manager. OVM Manager then communicates to OVS compute nodes for doing stop or start tasks via ovs-agent running on the Compute Nodes.

If there are inconsistencies in ovs-agent then there will be issues in communication between OVM Manager and OVS compute nodes leading to issues like EMOC start or stop jobs for VM's hanging or failing.

Solution

To resolve this issue - restart the ovs-agent on all the Compute Nodes, rediscover all the Compute Nodes from OVM Manager and refresh the repository from OVM Manager. These tasks does not need downtime and will not impact any running VM's. Following are steps to do this.

Restart ovs-agent on Compute Nodes using below commands
service ovs-agent stop

service ovs-agent start
Rediscover all Compute Nodes from OVM Manager using below steps.

a. Login to OVM Manager BUI using admin user
b. Click on the "Servers and VMs" tab
c. Expand the "Server Pools" entry
d. Locate the CN and right click on the CN in the BUI and select "Rediscover Server". Wait for the rediscover Job to complete.
Refresh the repository from OVM Manager using below steps.

a. Login to OVM Manager
b. Go to Repositories tab
c. Expand Repositories on left hand side
d. Select "/". Right click and click on "Refresh"

References

<NOTE:1512139.1> - Oracle Exalogic Elastic Cloud Known Issues - Virtualization Release
<NOTE:1520829.1> - Enterprise Manager Operations Center (EMOC) Known Issues On Exalogic

Attachments

This solution has no attachment