Exalytics OVM Server Crashed and is Unresponsive: "vector 0x2 is not implemented" and "soft lockup" Errors Displayed on the Console

Asset ID:	1-72-1994681.1
Update Date:	2016-11-02
Keywords:

Solution Type Problem Resolution Sure

Solution 1994681.1 : Exalytics OVM Server Crashed and is Unresponsive: "vector 0x2 is not implemented" and "soft lockup" Errors Displayed on the Console

Applies to:

Exalytics In-Memory Machine X3-4 - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Oracle Exalytics OVM Server version 2.0.1.3.0 (Patchset 5) with two OVM guest instances became unexpectedly non-responsive. Users could not login to applications and it was not possible to login via putty or SSH at the OS level.
Messages displayed on console show:

xen: vector 0x2 is not implemented
Bug: soft lockup - CPU#3 stuck for 22s! [mcelog: 2128]
Modules linked in: xen_pciback tun xen_blkbak xen_netback xen_gntdev xen_evtchm
lock sunrpc mlx4_en(U) bridge stp 11c bonding be2iscsi isci_boot_sysfs _iscsi_....

Similar behavior and soft lockup errors may also occur on Exalytics OVM servers with lower level Exalytics Patchsets (PS2, PS3, etc.).

Cause

Soft lockup messages in var/log/messages shows:

Mar 15 08:24:38 xadexa01 kernel: RIP: e030:[] [] csd_lock_wait+0x12/0x20
Mar 15 08:24:38 xadexa01 kernel: BUG: soft lockup - CPU#1 stuck for 22s! [python:5885]

These errors match the messages in known internal, unpublished <Bug 19638775> BUG: SOFT LOCKUP - RIP: XEN_HYPERCALL_SCHED_OP+0XA/0X20. Problem also reported in internal, Exalytics documentation <Bug 20800586> - EXALYTICS: GUEST OPERATING SYSTEM HANGS WITH SOFT LOCKUP: ERROR ON PS 5 OVM

Solution

To prevent this problem from re-occurring,

1. Upgrade the Exalytics server to PS5, (if not already installed), and then upgrade the OVM server to "Oracle Exalytics Base Image 2.0.1.4.0 for Exalytics Oracle VM x86-64."

The upgrade steps are described in Fusion Middleware Oracle Exalytics In-Memory Machine Installation and Administration Guide for Linux
See Section 2.2.2.2 Applying Oracle Exalytics Release 1 Patchset 5 (1.0.0.5) on an Exalytics Machine Configured for Virtualization

At a high level, you first need to download the "Exalytics OVM server to Oracle Exalytics Base Image 2.0.1.4.0" from Oracle Software Delivery Cloud
Select Business Intelligence Product, Linux x86-64 Platform -> Download "Exalytics OVM server to Oracle Exalytics Base Image 2.0.1.4.0"

2. Follow steps in the installation guide above.

3. Once you have successfully upgraded the server, Oracle Support also recommends that you enable some additional debug features. These features will be useful in the event you experience any future issues with the server. Please review the following documents for the recommended utilities:

OSWatcher Black Box (Includes: [Video]) <Document 301137.1>
Is "kdump" Pre-configured By Default On Oracle Exalytics Server <Document 1639784.1>

References

<NOTE:1639784.1> - Is "kdump" Pre-configured By Default On Oracle Exalytics Server
<BUG:19638775> - BUG: SOFT LOCKUP - RIP: XEN_HYPERCALL_SCHED_OP+0XA/0X20
<BUG:20800586> - EXALYTICS: GUEST OPERATING SYSTEM HANGS WITH SOFT LOCKUP: ERROR ON PS 5 OVM
<NOTE:1579194.1> - Oracle Exalytics Stability and Performance Issues on Compute Nodes Running Linux After Uptime of More Than 208 Days
<NOTE:301137.1> - OSWatcher (Includes: [Video])

Attachments

This solution has no attachment