ODA Oracle Database Appliance Huge Page / Large page sizing

Asset ID:	1-72-2108596.1
Update Date:	2018-01-09
Keywords:

Solution Type Problem Resolution Sure

Solution 2108596.1 : ODA Oracle Database Appliance Huge Page / Large page sizing

Applies to:

Oracle Database Appliance X4-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Wanting to size the Large Pages / Huge Pages on the ODA.
Encountering errors starting a database on the ODA after adding databases or increasing database memory parameters.
Increasing the physical memory to an existing ODA but unsure what parameter changes should be made at the OS level and for the database(s).

Changes

Adding new databases
Increasing the size of new databases

Cause

First time usage: trying to configure Huge Pages for the first time
Changes in number of databases and associated memory requiring changes in OS parameter settings including Huge Pages
Increase in physical memory resources ( e.g. after a memory upgrade)
Increase in physical memory consumption
Memory tuning

Solution

Refer to

ODA HA: Use OAKCLI RECONFIGURE OSPARAMS to Calculate, Update and Set Kernel / Memory Values Including Hugepages (Doc ID 2220254.1)

Automatic Memory Management (AMM) and Huge Pages are mutually exclusive.
- While the ASM database default is AMM, all user databases should not use AMM
ODA Templates do not use AMM
Each database instance should be set with the parameter

use_large_pages= ONLY
Run the OAKCLI RECONFIGURE OSPARAMS command after new database creation for recommendations

Optionally (manual approximations)

If the ODA has 256GB of memory per node the recommended hugepages is 64000.
If the ODA has 512GB of memory per node the double the vm.nr_hugepages setting to 128000.
If the ODA has 768GB of memory per node the set the vm.nr_hugepages to 192000.

If adding memory from 256GB to 512GB double the values in memlock soft/hard setting for grid and oracle users in /etc/security/limits.conf

For Kernel parameter settings you can use the following note: While note was created for Exadata the calculations are also correct for the ODA.

Exadata Kernel Tuning: Compute Node /etc/sysctl.conf parameters: Settings which and are candidates to adjust when adding or removing physical memory (RAM)

To add:

96gb <-- V1
128gb
768gb

To add hard values for:

memlock soft/hard setting for grid and oracle users in /etc/security/limits.conf

[root@odaV1a~]# grep Huge /proc/meminfo

HugePages_Total: 27300
HugePages_Free: 17359
HugePages_Rsvd: 681
HugePages_Surp: 0
Hugepagesize: 2048 kB

[root@odaV1a~]# grep Mem /proc/meminfo
MemTotal: 98934296 kB
MemFree: 6859688 kB

view /etc/security/limits.conf

...

# Oracle-Validated setting for memlock 72000000
grid soft nofile 131072
grid hard nofile 131072
grid soft nproc 131072
grid hard nproc 131072
grid soft core unlimited
grid hard core unlimited
grid soft memlock 72000000
grid hard memlock 72000000
oracle soft memlock 72000000
oracle hard memlock 72000000

# Oracle-Validated setting for nofile soft limit is 131072
oracle soft nofile 131072

# Oracle-Validated setting for nofile hard limit is 131072
oracle hard nofile 131072

# Oracle-Validated setting for nproc soft limit is 131072
oracle soft nproc 131072

# Oracle-Validated setting for nproc hard limit is 131072
oracle hard nproc 131072

96 GB per server node. Six 8 GB low-voltage DDR3 RDIMM (single or dual rank) at 1333-MHz per server node.

Note - Quad-rank DIMMs are not supported.

X3-2 X4-2

Memory = 256 GB per server node. Sixteen 16 GB low-voltage DDR3 RDIMM at 1600 MHz per server node.

X5-2

256 GB per server node. Eight 32 GB low-voltage DDR4 LRDIMM at 2133 MHz per server node.

Note - Optional memory expansion to 512 GB or 768 GB. Both server nodes must have the same amount of memory.

References

<NOTE:1598563.1> - Exadata Kernel Tuning: Compute Node /etc/sysctl.conf parameters: Settings which and are candidates to adjust when adding or removing physical memory (RAM).
<NOTE:2220254.1> - ODA HA: Use OAKCLI RECONFIGURE OSPARAMS to Calculate, Update and Set Kernel / Memory Values Including Hugepages
<NOTE:1392497.1> - USE_LARGE_PAGES To Enable HugePages
<NOTE:401749.1> - Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration
<NOTE:361323.1> - HugePages on Linux: What It Is... and What It Is Not...

Attachments

This solution has no attachment