Exalogic Guest vServer Hangs With "Probing EDD (edd= off to disable)" Message On Console During Boot Process Of Patching

Asset ID:	1-72-2130101.1
Update Date:	2018-05-09
Keywords:

Solution Type Problem Resolution Sure

Solution 2130101.1 : Exalogic Guest vServer Hangs With "Probing EDD (edd= off to disable)" Message On Console During Boot Process Of Patching

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.2.2 and later
Exalogic Elastic Cloud X5-2 Eighth Rack - Version X5 and later
Oracle VM - Version 3.2.9 to 3.2.9 [Release OVM32]
Linux x86-64
Oracle Virtual Server x86-64

Symptoms

In Exalogic virtual racks upgrading vServers from 2.0.6.1.2 to 2.0.6.2.2 (Note that issue is not dependent on the Exalogic version), upgrade got hung and eventually timed out. During rebooting of the vServer from EMOC,vserver boot process is hung again.SSH or ping and also xm console does not respond .

Stopping the vm from OVMM or doing xm destroy and restarting it from OVMM, we can launch console for the vServer in OVMM. vServer console showed that it is hung at point where it says the following message:

Probing EDD (edd= off to disable)

Changes

Upgrading vServers.

Cause

This was caused due to user error. In this case, guest vServer was trying to boot into non-uek Redhat kernel which was somehow present on the vserver.

This was determined by booting into single user mode and verifying the messages log, as well as grub.conf.

Contents of grub.conf for this vServer that was hanging :

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/VolGroup00-LogVol00
# initrd /initrd-[generic-]version.img
#boot=/dev/xvda
default=0 <<<< Default is set to '0', so that it will boot into the kernel that is at '0' index position.
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
serial --unit=0 --speed=19200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-504.16.2.el6.x86_64) <<<< This is REDHAT non UEK kernel at position '0'
root (hd0,0)
kernel /vmlinuz-2.6.32-504.16.2.el6.x86_64 ro root=/dev/mapper/VolGroup00-LogVol00 rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0,9600n8 numa=off enable_fast_entropy crashkernel=auto
initrd /initramfs-2.6.32-504.16.2.el6.x86_64.img

title Oracle Linux Server (2.6.39-400.214.5.1.el6uek.x86_64) <<<<< This is the correct Exalogic supported UEK kernel at index position '1'
root (hd0,0)
kernel /vmlinuz-2.6.39-400.214.5.1.el6uek.x86_64 ro root=/dev/mapper/VolGroup00-LogVol00 rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0,9600n8 numa=off enable_fast_entropy
initrd /initramfs-2.6.39-400.214.5.1.el6uek.x86_64.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/VolGroup00-LogVol00 rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0,9600n8 numa=off
initrd /initramfs-2.6.32-431.el6.x86_64.img

Solution

To resolve this issue follow below steps:

1) Login to single user mode and verify if the kernel being booted is correct UEK kernel and is the expected kernel in Exalogic.

To Boot the guest vServer in single user mode , follow these steps

- Start guest vServer using : xm create -c <vm.cfg location>.
- Interrupt boot by pressing any key, you will see the bootup menu. Hit "e" to edit the boot entry.
- Modify grub entry i.e. append 's' to kernel line at the end and then hit enter key.
- You come back to the bootup menu, hit "b" to boot guest - it will be in single user mode.

2) Verify grub.conf file contents and make sure that the 'default' parameter is pointing to the correct index number which has the UEK Oracle Linux kernel that is supported on Exalogic.

If it is having a different index number pointing to wrong kernel, Update the value of 'default' parameter and save it and reboot normally. Verify that boot is not hanging and continue with step 3 below.

If the entries are correct and vServer is booting into correct kernel but still hanging, do not proceed with next step (3) and collect sosreport in singleuser mode . Contact Support by opening Service Request and attach the sosreport.

3) Once the wrong kernel is identified as in step 2, Delete the non UEK kernel rpm's as follows.

- Take a backup of the guest vServer using ExaBR
- Remove older non-running UEK kernels. Do not remove the current UEK Kernel and associated core Kernel.

Procedure to remove older kernels

  a. Get the list of all kernel entries in grub.conf except the top 2 kernel entries. This can be verified with the following command:

# grep kernel /boot/grub/grub.conf | grep -v "#" | tail -n +3 | awk '{ print $2 }' | sed 's/\/vmlinuz-//g'

  b. Get the rpms associated with each such entry:

# rpm -qa | grep kernel | grep
For example:

rpm -qa | grep kernel | grep 2.6.32-504.16.2.el6.x86_64

  c. Remove each of the rpm as follows:

# rpm -ev --allmatches
For example:

# rpm -ev --allmatches kernel-2.6.32-504.16.2.el6.x86_64.rpm

4) If the Patching of the vServer is not yet completed (verify by running imageinfo command on the vserver and check that the kernel), proceed with the exapatch command to patch that vServer.

References

<NOTE:1571367.1> - Exalogic Infrastructure PSU Upgrade - Known Issues

Attachments

This solution has no attachment