Oracle VM
 
Linux OS
 
Private Cloud Appliance X5-2 Hardware
 

PLA-Support>Infrastructure>Operating Systems and Virtualization>Virtualization>Oracle VM
 

In this Document

Description

Occurrence

Symptoms

Workaround

Proactively cleaning /boot before the upgrade:

Repairing your system if /boot ran out of disk space during the upgrade:

Patches

History

References

Applies to:

Description

This document explains how to remove older kernels from your Oracle VM Server ( dom0/hypervisor ) in order to ensure that there is sufficient space available in /boot before starting the upgrade from Oracle VM 3.x to Oracle VM 3.3.x.
Furthermore it explains how to recover if /boot ran out of space during an upgrade.
The Oracle VM 3.3.x Installer will fail if there is insufficient space in /boot during the upgrade

Occurrence

This issue can happen during upgrading Oracle VM Server 3.x from lower versions.

Symptoms

 The installer/upgrader finishes, but re-booting fails with a panic afterwards

Workaround

Proactively cleaning /boot before the upgrade:

1. Before upgrading from Oracle VM 3.x to 3.3.x please check that there is enough space available in /boot
   
   
   # df -h /boot
   
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda1              99M   75M   24M  58% /boot
    
   For the upgrade you should have at least 35 MB free otherwise installing/upgrading to Oracle VM might fail.
   
   
2. At first it has to be checked what kernel version is running
   
   
   # uname -r
   2.6.39-300.29.1.el5uek
    
   with the kernel version we can determine what in /boot should be kept.
   
   
   # ls -la /boot/*2.6.39-300.29.1.el5uek*
   -rw-r--r-- 1 root root  114857 Feb 14  2013 /boot/config-2.6.39-300.29.1.el5uek
   -rw------- 1 root root 3225904 Feb 28  2013 /boot/initrd-2.6.39-300.29.1.el5uek.img
   -rw-r--r-- 1 root root  187348 Feb 14  2013 /boot/symvers-2.6.39-300.29.1.el5uek.gz
   -rw-r--r-- 1 root root 2628534 Feb 14  2013 /boot/System.map-2.6.39-300.29.1.el5uek
   -rwxr-xr-x 1 root root 4120816 Feb 14  2013 /boot/vmlinuz-2.6.39-300.29.1.el5uek
   
    
   
3. At this stage you can backup these files to ensure you have a backup in case they are accidently deleted during the following process.
   
   
   # mkdir /tmp/backup
   # cp /boot/*2.6.39-300.29.1.el5uek* /tmp/backup
   # cp -a /boot/grub /tmp/backup
   # cp -a /boot/xen* /tmp/backup
    
   
   Verify that all files have been copied
   
   

   # ls -la /tmp/backup/
   total 27484
   drwxr-xr-x 3 root root     4096 Sep 18 22:45 .
   drwxrwxrwt 4 root root     4096 Sep 18 22:30 ..
   -rw-r--r-- 1 root root   114857 Sep 18 22:30 config-2.6.39-300.29.1.el5uek
   drwxr-xr-x 2 root root     4096 Mar 31  2013 grub
   -rw------- 1 root root  3225904 Sep 18 22:30 initrd-2.6.39-300.29.1.el5uek.img
   -rw-r--r-- 1 root root   187348 Sep 18 22:30 symvers-2.6.39-300.29.1.el5uek.gz
   -rw-r--r-- 1 root root  2628534 Sep 18 22:30 System.map-2.6.39-300.29.1.el5uek
   -rwxr-xr-x 1 root root  4120816 Sep 18 22:30 vmlinuz-2.6.39-300.29.1.el5uek
   -rw-r--r-- 1 root root   720993 Sep 18 22:45 xen-4.1.3OVM.gz
   -rw-r--r-- 1 root root   720993 Sep 18 22:45 xen-4.1.gz
   -rw-r--r-- 1 root root   720993 Sep 18 22:45 xen-4.gz
   -rw-r--r-- 1 root root   720993 Sep 18 22:45 xen.gz
   -rw-r--r-- 1 root root 11327008 Sep 18 22:45 xen-syms-4.1.3OVM
   

    

    
   
   
4. Furthermore it might be good to have the rpm versions of the running kernel available
   
       kernel-uek-2.6.39-300.29.1.el5uek.x86_64.rpm
       kernel-uek-firmware-2.6.39-300.29.1.el5uek.noarch.rpm
   
   
   on the system, so if you accidently remove the running kernel you can re-install with the rpm's.
   These files can be downloaded from https://linux.oracle.com
   
   
   
   NOTE: If you accidently removed the running kernel, do not reboot. Re-install the rpm's before a reboot !
   
    
   
5. Deleting the older kernels, currently kernel 2.6.39-300.29.1.el5uek is running as seen in the above "uname -r" output, so we have to exclude this kernel in the following output ( grep -v 2.6.39-300.29.1.el5uek )
   
   
   # rpm -qa | grep kernel-uek | grep -v 2.6.39-300.29.1.el5uek
   kernel-uek-firmware-2.6.39-400.19.1.el5uek
   kernel-uek-2.6.39-200.1.14.el5uek
   kernel-uek-firmware-2.6.39-300.22.2.el5uek
   kernel-uek-2.6.39-400.19.1.el5uek
   kernel-uek-firmware-2.6.39-200.1.14.el5uek
   kernel-uek-2.6.39-300.22.2.el5uek
     
   The above listed kernels can be safely removed.
   
   
   
   NOTE: Your system may have different kernels than listed above, the goal of this document is to only keep the currently running and newer kernels installed on the system.
   There maybe cases were you have to keep more than one or two kernels on your system, so before starting to remove all but the running and newer kernels ensure that they are really not
   needed anymore. The recommendation is not to keep more than 2 kernels if you have problems with free space in /boot
    
   
   To remove these kernels do the following:
   
   
   # rpm -e kernel-uek-firmware-2.6.39-200.1.14.el5uek kernel-uek-2.6.39-200.1.14.el5uek
   
   
   NOTE: kernel-uek-firmware and kernel-uek should be the same version
    
   
   afterwards you can see in /boot that the initrd, config and vmlinuz files for kernel 2.6.39-200.1.14.el5uek have been removed
   
   
   # ls -la /boot
   total 48803
   drwxr-xr-x  4 root root     3072 Sep 18 22:54 .
   drwxr-xr-x 30 root root     4096 Aug  1 01:26 ..
   -rw-r--r--  1 root root   114857 Jan  4  2013 config-2.6.39-300.22.2.el5uek
   -rw-r--r--  1 root root   114857 Feb 14  2013 config-2.6.39-300.29.1.el5uek
   -rw-r--r--  1 root root   115022 Mar 22  2013 config-2.6.39-400.19.1.el5uek
   drwxr-xr-x  2 root root     1024 Sep 18 22:54 grub
   -rw-------  1 root root  3225904 Jan 16  2013 initrd-2.6.39-300.22.2.el5uek.img
   -rw-------  1 root root  3225904 Feb 28  2013 initrd-2.6.39-300.29.1.el5uek.img
   -rw-------  1 root root  3227004 Mar 31  2013 initrd-2.6.39-400.19.1.el5uek.img
   drwx------  2 root root    12288 Oct 16  2012 lost+found
   -rw-r--r--  1 root root   187331 Jan  4  2013 symvers-2.6.39-300.22.2.el5uek.gz
   -rw-r--r--  1 root root   187348 Feb 14  2013 symvers-2.6.39-300.29.1.el5uek.gz
   -rw-r--r--  1 root root   188934 Mar 22  2013 symvers-2.6.39-400.19.1.el5uek.gz
   -rw-r--r--  1 root root  2628249 Jan  4  2013 System.map-2.6.39-300.22.2.el5uek
   -rw-r--r--  1 root root  2628534 Feb 14  2013 System.map-2.6.39-300.29.1.el5uek
   -rw-r--r--  1 root root  2634072 Mar 22  2013 System.map-2.6.39-400.19.1.el5uek
   -rwxr-xr-x  1 root root  4121424 Jan  4  2013 vmlinuz-2.6.39-300.22.2.el5uek
   -rwxr-xr-x  1 root root  4120816 Feb 14  2013 vmlinuz-2.6.39-300.29.1.el5uek
   -rwxr-xr-x  1 root root  4129392 Mar 22  2013 vmlinuz-2.6.39-400.19.1.el5uek
   -rw-r--r--  1 root root   720993 Feb  4  2013 xen-4.1.3OVM.gz
   lrwxrwxrwx  1 root root       15 Feb 28  2013 xen-4.1.gz -> xen-4.1.3OVM.gz
   lrwxrwxrwx  1 root root       15 Feb 28  2013 xen-4.gz -> xen-4.1.3OVM.gz
   lrwxrwxrwx  1 root root       15 Feb 28  2013 xen.gz -> xen-4.1.3OVM.gz
   -rw-r--r--  1 root root 11327008 Feb  4  2013 xen-syms-4.1.3OVM
     
   
   and more space is available,
   
   
   # df -h /boot
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda1              99M   65M   34M  58% /boot
   
   
   Removing another old kernel,
   
   
   # rpm -e kernel-uek-firmware-2.6.39-300.22.2.el5uek kernel-uek-2.6.39-300.22.2.el5uek
     
   
   afterwards there should be sufficient space in /boot for the Oracle VM 3.3.x upgrade
   
   
   # df -h /boot
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda1              99M   55M   44M  58% /boot
     
   
   
    

Repairing your system if /boot ran out of disk space during the upgrade:

In case you already tried to upgrade Oracle VM to 3.3.1 and you are unable boot with the installed kernel and your /boot directory is full

While creating the initramfs dracut could not complete the file because the filesytem was full, therefore the boot fails as the initramfs
has been corrupted. The boot fails with a kernel panic.

To solve this, please do the following

1. When the server is booting, try to get into the grub menu and select a different kernel to boot
   
   
2. Since the old kernels based on Oracle Linux 5 have been removed by the installer you will have to manually remove files in /boot
   
   # df -h /boot
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda1              99M   99M   0M  100% /boot
    
   
   First we will check the kernel version of the running kernel,
   
   # uname -r
   2.6.39-300.29.1.el5uek
     
   
   we would like to keep the current version and the newly installed Oracle Linux 6 based Oracle VM kernel.
   
   A typical /boot directly will look like
   
   
   # ls  -la
   total 55402
   dr-xr-xr-x  5 root root     1024 Jul  2 20:06 .
   dr-xr-xr-x 25 root root     4096 Sep 14 01:03 ..
   -rw-r--r--  1 root root   114857 Oct 08  2013 config-2.6.39-300.32.6.el5uek
   -rw-r--r--  1 root root   114857 Oct 11  2013 config-2.6.39-300.29.1.el5uek
   -rw-r--r--  1 root root   130413 May  8 04:51 config-3.8.13-26.4.2.el6uek.x86_64
   drwxr-xr-x  3 root root     1024 Jul  2 19:56 efi
   drwxr-xr-x  2 root root     1024 Jul  2 20:03 grub
   -rw-------  1 root root 21863189 Jul  2 20:03 initramfs-3.8.13-26.4.2.el6uek.x86_64.img
   -rw-------  1 root root  3753923 Jul  3 06:48 initrd-2.6.39-300.32.6.el5uek.img
   -rw-------  1 root root  3553923 Jul  1 06:48 initrd-2.6.39-300.29.1.el5uek.img
   drwx------  2 root root    12288 Jul  1 06:45 lost+found
   -rw-r--r--  1 root root   187348 Oct 11  2013 symvers-2.6.39-300.32.6.el5uek.gz
   -rw-r--r--  1 root root   197348 Oct 08  2013 symvers-2.6.39-300.29.1.el5uek.gz
   -rw-r--r--  1 root root   218201 May  8 04:59 symvers-3.8.13-26.4.2.el6uek.x86_64.gz
   -rw-r--r--  1 root root  2628573 Oct 11  2013 System.map-2.6.39-300.32.6.el5uek
   -rw-r--r--  1 root root  2728573 Oct 08  2013 System.map-2.6.39-300.29.1.el5uek
   -rw-r--r--  1 root root  2751890 May  8 04:51 System.map-3.8.13-26.4.2.el6uek.x86_64
   -rwxr-xr-x  1 root root  4121136 Oct 11  2013 vmlinuz-2.6.39-300.32.6.el5uek
   -rwxr-xr-x  1 root root  4421136 Oct 08  2013 vmlinuz-2.6.39-300.29.1.el5uek
   -rwxr-xr-x  1 root root  4589808 May  8 04:51 vmlinuz-3.8.13-26.4.2.el6uek.x86_64
   -rw-r--r--  1 root root      172 May  8 04:51 .vmlinuz-3.8.13-26.4.2.el6uek.x86_64.hmac
   -rw-r--r--  1 root root   836179 May 14 21:48 xen-4.3.1OVM.gz
   lrwxrwxrwx  1 root root       15 Jul  2 19:56 xen-4.3.gz -> xen-4.3.1OVM.gz
   lrwxrwxrwx  1 root root       15 Jul  2 19:56 xen-4.gz -> xen-4.3.1OVM.gz
   lrwxrwxrwx  1 root root       15 Jul  2 19:56 xen.gz -> xen-4.3.1OVM.gz
   -rw-r--r--  1 root root     9663 May 14 21:46 xenpolicy.24
   -rw-r--r--  1 root root 15466388 May 14 21:48 xen-syms-4.3.1OVM
     
   
   after upgrading. Here we have three kernels installed
   
   
   2.6.39-300.29.1.el5uek ( the one we have just booted, we have to keep it )
   
   3.8.13-26.4.2.el6uek ( Oracle VM 3.3.x kernel, we have to keep it )
   
   2.6.39-300.32.6.el5uek ( not in use, not needed for Oracle VM 3.3.x )
   
   
   so we can delete files related to kernel 2.6.39-300.32.6.el5uek to free up space.
   
   These files are in our case:
   
   
   # ls -la *2.6.39-300.32.6.el5uek*
   -rw-r--r-- 1 root root  114857 Oct 11  2013 config-2.6.39-300.32.6.el5uek
   -rw------- 1 root root 3553923 Jul  1 06:48 initrd-2.6.39-300.32.6.el5uek.img
   -rw-r--r-- 1 root root  187348 Oct 11  2013 symvers-2.6.39-300.32.6.el5uek.gz
   -rw-r--r-- 1 root root 2628573 Oct 11  2013 System.map-2.6.39-300.32.6.el5uek
   -rwxr-xr-x 1 root root 4121136 Oct 11  2013 vmlinuz-2.6.39-300.32.6.el5uek
    
   
   
   
3. To delete them, do
   
   # rm config-2.6.39-300.32.6.el5uek initrd-2.6.39-300.32.6.el5uek.img symvers-2.6.39-300.32.6.el5uek.gz System.map-2.6.39-300.32.6.el5uek vmlinuz-2.6.39-300.32.6.el5uek
   
    
4. Then we need to delete the corrupted initramfs for the Oracle Linux 6 based Oracle VM kernel.
   
   # rm initramfs-3.8.13-26.4.2.el6uek.x86_64.img
     
   
   
5. Now we should have enough space for properly re-building the initramfs for 3.8.13-26.4.2.el6uek. You can use the following command:
   
   # dracut -f /boot/initramfs-3.8.13-26.4.2.el6uek.x86_64.img 3.8.13-26.4.2.el6uek.x86_64
    
   The first parameter -f would overwrite any existing initramfs with the name /boot/initramfs-3.8.13-26.4.2.el6uek.x86_64.img ( the second parameter ) , the third paramter is 3.8.13-26.4.2.el6uek.x86_64 which is the kernel version for which we want to build a new initramfs. See "man dracut" for more details.
   
   
   
6. After this you can reboot and boot with the 3.8.13-26.4.2.el6uek.x86_64 kernel.
   
   

Please see Installation and Upgrade Guide for Release 3.3, Chapter 5 Upgrading Oracle VM for more information about upgrading and installing Oracle VM.

Patches

 At the time of writting no patched iso has been released

History

 25-SEP-2014 - Alert published

References

This solution has no attachment