After Planned Server Restart Exalytics OVM Failing With Errors: "bond1.340 on EXALOVS4 is locked, description: Start Virtual Machine"

Asset ID:	1-72-1961148.1
Update Date:	2016-12-01
Keywords:

Solution Type Problem Resolution Sure

Solution 1961148.1 : After Planned Server Restart Exalytics OVM Failing With Errors: "bond1.340 on EXALOVS4 is locked, description: Start Virtual Machine"

Applies to:

Oracle Exalytics Software - Version 1.0.0.3.0 and later
Exalytics In-Memory Machine X3-4 - Version All Versions and later
Linux OS - Version Oracle Linux 5.5 and later
Information in this document applies to any platform.

Symptoms

Exalytics machine with four Oracle Virtual Machines (OVM) was gracefully shut down as part of a planned outage. After restarting the machine, three of the VM's start successfully, but the fourth one fails to start.
It starts to load, but then fails with error like:

bond1.340 on EXALOVS4 is locked, description: Start Virtual Machine

Standard out shows error like:

Server error message: Command: ['xm', 'create',
'/OVS/repositories/0004fb0000030000fe8c6dc76a896e69/virtual Machines/0004fb000006000084d6d95be29917a7/vm.cfg']
failed (1): stderr: Error: Device 268439808 (vdb) could not be connected.
/dev/mapper/SATA_3E128-TS2-550B0_FL008Y1D does not exist.
stdout: Using config file
"/OVS/Repositories/0004fb0000030000fe8c6dc76a896e69/virtual Machines/0004fb000006000084d6d95be29917a7/vm.cfg".

Cause

Hardware failure with the flash drives used by this VM. Hardware Support Team analysis of ILOM snapshot found that there was a defective flash drive. After flash drive replacement, the same error occurred, but showed a different device ID. New flash drives had a new device ID and the OVM manager mapping did not match the device IDs that were detected by the operating system. Only one VM instance relied upon access to these flash drives. The other VMs started because they were not mapped to the failing flash drives.

Solution

After flash drive replacement, restart the server via graceful shutdown and reboot. Confirm that there are no flashing red lights. Then restart the VMs.

NOTE: You may see an error when starting the 4th (broken) VM. This error may indicate that a certain device cannot be found (i.e. flash drive with ID like FL008Y1X). This error occurs because the new flash drive has a different ID than the original installed flash drive. Changes are required to make the VM "see" the new flash drive and device ID.

To remap the flash drive device ID, you need to compare the flash drive IDs as detected by the operating system (OS) and then compare to the device IDs seen in the OVM Manager.

1. Log into VM manager -> do a rescan on the OVM instance to find new devices.
2. Then use commands at OS level to check device IDs and compare to OVM manager IDs.

OS level (as root user)

ls -al /dev/disk/by-id/

VM Manager

Right click on VM in the Manager -> Edit -> select Disks tab -> review Device IDs.

3. Locate/identify old flash drive IDs (via OS command results and those displaying in the VM manager). Four new device IDs are expected.

a. The device IDs at the OS level will not match the Device IDs in the VM manager and will need to be remapped (i.e. so that the OS results and OVM manager match).
b. To remap the new flash drive in OVM manager -> select OVM Disks tab -> Right click the magnifying glass next to the disk ID -> Map new flash drive ID (using device ID seen in OS command results).
c. Do this for all device ID differences noted.
d. Restart all OVMs and confirm all VMs start successfully.

References

<NOTE:1448069.1> - How to run an ILOM Snapshot on a Sun/Oracle X86 System
<NOTE:1527167.1> - Oracle Exalytics In-Memory Machine diagnostic data collection script
<NOTE:1500235.1> - How To Collect an Sosreport on Oracle Linux

Attachments

This solution has no attachment