ODA HA: Use OAKCLI RECONFIGURE OSPARAMS to Calculate, Update and Set Kernel / Memory Values Including Hugepages

Asset ID:	1-71-2220254.1
Update Date:	2018-01-09
Keywords:

Solution Type Technical Instruction Sure

Solution 2220254.1 : ODA HA: Use OAKCLI RECONFIGURE OSPARAMS to Calculate, Update and Set Kernel / Memory Values Including Hugepages

Applies to:

Oracle Database Appliance X6-2 HA Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X4-2 - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.
ODA, ODA HA, ODA X6-2 HA, ODAVP, Hugepages, OS kernel parameters,

Goal

One challenge with the ODA is calculating the correct OS / Kernel values for memory:
Specific use cases include:

New memory added to an existing ODA System (DIMMs)
Calculating HugePages based on current and changing usage
Generic memory optimization based on current and changing usage

This note describes the steps to set Kernel parameters

Automatically
Manually

This note is regarding the ODA HA and not X6-2 S X6-2 M or X6-2L (ODA Lite single servers) which use ODA*CLI commands.

Solution

Automatic Method

The Oracle Database Appliance High Availability ( ODA HA ) Oracle Appliance Kit command line ( OAKCLI) has introduced a new method to calculate and update Kernel parameters using the command

oakcli reconfigure osparams

This same command can calculate and update the HugePages parameter for the ODA which may vary with usage over time
Currently if using the manual method these altered values need to be reset after each bounce including patching:

The OAKCLI RECONFIGURE OSPARAMS command is available on the current ODA releases

# oakcli reconfigure

Example

[root@odax5-2 ~]# oakcli reconfigure osparams

Configured value for memlock is: 183000000
Suggested value for memlock is: 184000000

Do you want to use suggested value of memlock?: yes/no
yes

Configured value for vm.nr_hugepages is: 65604
Suggested value for vm.nr_hugepages parameter is: 64598

Do you want to update to suggested value?: yes/no
yes

IMPORTANT TIPS

Automatic Memory Management (AMM) and Huge Pages are mutually exclusive.

While the ASM database default is AMM, all user databases should not use AMM
e.g. ODA Templates do not use AMM
Each database instance should be set with the parameter use_large_pages
+ requires a bounce of the instance

use_large_pages= ONLY
Run the OAKCLI RECONFIGURE OSPARAMS command after new database creation for recommendations

Manual Method

Manual Method ( Optional if using older ODA versions)

1. Increase kernel.sem on BOTH nodes (See Table 1)

Example:

From

250

32000

100

142

To:

1510

214420

1510

200

e.g. To alter from current values to target values for kernel.sem

# vi /etc/sysctl.conf
kernel.sem=1510 214420 1510 200
# sysctl -p /etc/sysctl.conf

2. Increase the hugepages from 67200 to 103865 on both nodes as below:

e.g. To alter from current values to target values for vm.nr_hugepages

# vi /etc/sysctl.conf
vm.nr_hugepages=103865
# sysctl -p /etc/sysctl.conf

3. Increase memlock limit from 192000000 to 256000000 on both nodes as below:

e.g. To alter from current values to target values for memlock

# vi /etc/security/limits.conf
oracle soft memlock 256000000
oracle hard memlock 256000000

4. For shmmax and shmall the current setting fully covers all the memory combinations.

total_mem = 'cat /proc/meminfo | grep MemTotal: | awk '{print $2}'

For Huge pages = (total_mem / 4096 ) * 1.05 ( We add extra 5% to handle any other processes like oc4j )
For memlock = 3/4 * total_mem

Here are some estimates below.

Table 1


Environment Variable	384GB	512GB	768G
oracle memlock	29400000	391000000	588000000
hugepages	102681	137088	205900

Here are some tested settings for the ODA if using 50+ databases
===============================
The following was the maximum creation on X5-2 fully populated (768G), this has been tested .

1. Increase kernel.sem from ‘250 32000 100 142’ to ‘1510 214420 1510 200’ on both nodes.

# vi /etc/sysctl.conf
kernel.sem=1510 214420 1510 200

# sysctl -p /etc/sysctl.conf

2. Increase the hugepages from 67200 to 103865 on both nodes as below:

# vi /etc/sysctl.conf
vm.nr_hugepages=103865
# sysctl -p /etc/sysctl.conf

3.Increase memlock limit from 192000000 to 256000000 on both nodes as below:

# vi /etc/security/limits.conf
oracle soft memlock 256000000
oracle hard memlock 256000000

Each database has following configurations:

alter system set sga_target=4g scope=spfile sid='*';
alter system set pga_aggregate_target=2g scope=both sid='*';
alter system set processes=1500 scope=spfile sid='*';
alter system set cpu_count=8 scope=both sid='*';

Since you use hupepages "kernel.shmmax ,kernel.shmall " changes are not required.
You will need to set the above parameters with every reboot.
Development is working on a permanent fix.

References

<NOTE:2218451.1> - ODA: After a DIMM Upgrade Several Key OS Kernel Values Were Reset Back to the Original Values After ODA patching / Upgrade - Bug 25192165
<BUG:25192165> - ODA PATCHING RESETS KERNEL PARAMETERS TO DEFAULT AFTER DIMM UPGRADE
<BUG:25190881> - ENH:ODA OAKCLI RECONFIGURE OSPARAMS CHECKS + UPDATES KERNEL AFTER A DIMM UPGRADE

Attachments

This solution has no attachment