How to Enable HugePages for a Para-Virtualized Machine ( PVM ) on Oracle VM 3.2

Asset ID:	1-71-1529373.1
Update Date:	2018-02-27
Keywords:

Solution Type Technical Instruction Sure

Solution 1529373.1 : How to Enable HugePages for a Para-Virtualized Machine ( PVM ) on Oracle VM 3.2

Applies to:

Oracle VM - Version 3.2.4 and later
Private Cloud Appliance X5-2 Hardware - Version All Versions to All Versions [Release All Releases]
Linux x86-64

Goal

The goal of this document is to instruct on how to properly configure and enable the HugePages Kernel feature in a Oracle VM 3.2 environment for a PVM (Paravirtualized Virtual Machine). In order to accomplish this, the VM Server (Dom0) requires to be configured properly as well the Virtual Machine (guest or DomU). You can use Hugepages in a HVM (Hardware Virtualized Machine) as usual, with or without having this feature ('superpages') configured.

Requirements:

Oracle VM - Version 3.2.4 and later (earlier versions might work but are not supported)

Guest OS requirements:

Oracle Linux 5.8, 5.9 with Oracle Unbreakable Enterprise Kernel 2.6.39-100.0.21.el5uek or later
Oracle Linux 6.3, 6.4 with Oracle Unbreakable Enterprise Kernel 2.6.39-100.0.21.el6uek or later

Note: For Oracle VM 3.4.x, PVM and Hugepages see Document 2216622.1

Note: Hugepages feature is disabled for PVM guest running UEK4 4.1.12-85.el6uek and later.

Kernel changelog shows: * Mon Jan 23 2017 Chuck Anderson <chuck.anderson@oracle.com> [4.1.12-85.el6uek]
- x86/mm/xen: Suppress hugetlbfs in PV guests (Jan Beulich) [Orabug: 25227135] {CVE-2016-3961}

The recommendations is to change the domain type for virtual machines from Paravirtualized (PVM) to Hardware virtualized, with paravirtualized drivers (PVHVM) for those cases.

Solution

The first goal to is to enable superpages in the Dom0, this step requires to modify the Kernel boot parameters of the Xen from the Grub configuration:

1) Edit your /etc/grub.conf configuration file and find the proper entry for the running Kernel, e.g:

-----/etc/grub.conf-----
default=0
timeout=5
title Oracle VM Server-ovs (xen-4.1.3 2.6.39-300.12.0.el5uek)
root (hd0,0)
kernel /xen.gz dom0_mem=657M
module /vmlinuz-2.6.39-300.12.0.el5uek ro root=UUID=a1be5923-5da0-4fea-bc4b-fa60e9baf8e3
module /initrd-2.6.39-300.12.0.el5uek.img
------------------------

You need to focus in the line that defines the Xen kernel file and memory assignment and append the 'allowsuperpage' parameter. For short just replace:

kernel /xen.gz dom0_mem=657M

kernel /xen.gz dom0_mem=657M allowsuperpage

Once you save the file and reboot your Oracle VM Server, the Xen Kernel will allow their guest Kernels to request and assign HugePages.

Important Note: You must implement this change on ALL Oracle VM Servers that may run HugePage-enabled VMs. Otherwise, migrating a VM from a server with it enabled to one without will cause the VM to crash.

2) As described earlier, if the first step was performed properly, you need to instruct your Linux guest (DomU) to allow the usage of HugePages.

2.1) For that you need to locate the guest configuration file under the filesystem of Dom0:

/OVS/Repositories/0004fb0000030000afd63c01883f7dc2/VirtualMachines/0004fb00000600008734dbcee54609ea/vm.cfg
{------repository UUID---------} {------Virtual Machine UUID-----}
| |
+-----------------------------------------------+
UUIDs can be obtained from the VM Manager

Open and edit the vm.cfg file and append the following line at the end:

superpages=1

Example:

vif = ['mac=00:21:f6:00:00:00,bridge=0a9dc000']
OVM_simple_name = 'OVM3Manager-PROD'
vnclisten = '127.0.0.1'
serial = 'pty'
disk = ['file:/OVS/Repositories/0004fb0000030000afd63c01883f7dc2/VirtualDisks/0004fb0000120000d3bf417d5f2c5354.img,xvda,w']
vncunused = 1
uuid = '0004fb00-0006-0000-8734-dbcee54609ea'
on_reboot = 'restart'
boot = 'dc'
cpu_weight = 27500
memory = 1884
cpu_cap = 0
maxvcpus = 1
OVM_high_availability = False
vnc = 1
OVM_description = ''
on_poweroff = 'destroy'
on_crash = 'restart'
guest_os_type = 'linux'
name = '0004fb00000600008734dbcee54609ea'
builder = 'pvm'
vcpus = 1
keymap = 'en-us'
OVM_os_type = 'Oracle Linux 5'
OVM_cpu_compat_group = ''
OVM_domain_type = 'xen_pvm'
superpages=1

Keep in mind that every time you edit this VM in the Oracle VM Manager, the superpages entry will be removed from the vm.cfg. We are currently working on supporting this parameter in the Oracle VM Manager.

Save the configuration file and start the guest from the Oracle VM Manager.

2.2) Alternatively, you can also use Oracle VM Utilities to modify DomU configuration. It has same effect as in previous 2.1 step, without manually editing vm.cfg on dom0.

a. Download the Oracle VM Utilities and install it on Oracle VM Manager: http://docs.oracle.com/cd/E35328_01/E35333/html/vmutl-install.html

b. Go to the Oracle VM Utilities working directory and set PVM guest to enable HugePages, for example:

[root@ovmm ~]# cd /u01/app/oracle/ovm-manager-3/ovm_utils
[root@ovmm ovm_utils]# ./ovm_vmcontrol -u admin -p <admin password> -h localhost -v <OVM_simple_name> -c setsuperpage
Oracle VM VM Control utility 1.0.1.
Connected.
Command : setsuperpage
Enabling superpages.
Superpages enabled for '<OVM_simple_name>'.

c. Changes would be reflected within vm.cfg file:

[root@ovs ~]# cat /OVS/Repositories/<Repo UUID>/VirtualMachines/<VM guest UUID>/vm.cfg | grep superpages
superpages = 1

d. To unset HugePages for a PVM Guest:

[root@ovmm ~]# cd /u01/app/oracle/ovm-manager-3/ovm_utils
[root@ovmm ovm_utils]# ./ovm_vmcontrol -u admin -p <admin password> -h localhost -v <OVM_simple_name> -c unsetsuperpage
Oracle VM VM Control utility 1.0.1.
Connected.
Command : unsetsuperpage
Disabling superpages.
Superpages disabled for '<OVM_simple_name>'.

3) The last step is to log into the running guest and instruct the system to reserve/allocate

some huge pages:

# echo 1024 > /proc/sys/vm/nr_hugepages
# cat /proc/sys/vm/nr_hugepages
1024

For more details about setting HugePages for Oracle Linux, refer to the KM note:

HugePages on Oracle Linux 64-bit ( Doc ID 361468.1 )

Known Issues

unpublished Bug 16917037 - Guest VM starts with an error if the size is not a multiple of 2MB. When a guest is started with the superpage flag, its memory is populated entirely with 2MB superpages. Therefore the memory size must be a multiple of 2MB.
unpublished Bug 16988306 - Kernel panic under certain workloads on 32-bit PVHVM Oracle Linux 5.8 with hugepage setting enabled.

Hugepage Testing Report

http://ca-testlink.us.oracle.com/testlink/lib/results/printDocument.php?apikey=951159c53ce11c82498677fda777c2dd&type=test_report&level=testproject&tproject_id=58587&tplan_id=59859&header=y&summary=y&toc=y&body=y&passfail=y&cfields=y&metrics=y&author=y&requirement=y&keyword=y&notes=y&headerNumbering=y&format=0

References

<BUG:16917037> - CAN'T START A PVM GUEST + SUPERPAGES WHICH THE MEM SIZE IS 2011M
<NOTE:361468.1> - HugePages on Oracle Linux 64-bit
<BUG:16988306> - KERNEL PANIC ON EL5U8 X86 GUEST WITH HUGEPAGE SETTING WHEN RUN ST-MEM2 OLT TEST
<NOTE:757719.1> - Oracle VM: Comparison/Conversion between Guest Virtualisation Modes: HVM, PVM and HVM with PV Drivers
<BUG:17436272> - LIVE MIGRATION PVM WITH HUGE PAGES ENABLED TO A NON-ENABLED ORACLE VM SERVER

Attachments

This solution has no attachment