HugePages on Oracle Linux 64-bit (Doc ID 361468.1)

HugePages on Oracle Linux 64-bit (Doc ID 361468.1)

APPLIES TO:

Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Cloud Service - Version N/A and later
Linux x86-64
Oracle Linux
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)

PURPOSE

This document aims to provide.  本文档旨在提供

• Basic configuration of HugePages on 64-bit Linux  HugePages在64位Linux上的基本配置
• Fundamental reasons to use HugePages on Linux  在Linux上使用HugePages的根本原因
• References to known problems  参考已知问题
• References to technical background on HugePages  对HugePages的技术背景的引用

SCOPE

Information in this document is useful for Linux system administrators and Oracle database administrators working with system administrators.
本文档中的信息对于Linux系统管理员和与系统管理员一起工作的Oracle数据库管理员很有用。
This document covers information about Linux HugePages for 64-bit architectures. For more generic and uses on 32-bit and for references please see Document 361323.1
本文档涵盖有关用于64位体系结构的Linux HugePages的信息。有关32位的更多通用信息和用法，以及参考信息，请参阅Document 361323.1
The configuration steps provided here is primarily for Oracle Linux. Still the same concepts and configurations should apply to other Linux distributions.  此处提供的配置步骤主要针对Oracle Linux。相同的概念和配置仍应适用于其他Linux发行版

DETAILS

Introduction  介绍

HugePages is a feature of the Linux kernel  which allows larger pages to manage memory as the alternative to the small 4KB pagesize. For a detailed introduction, see Document 361323.1

HugePages是Linux内核的一项功能，它允许较大的页面来管理内存，以替代较小的4KB页面大小。有关详细介绍，请参见Document 361323.1

Why Do You Need HugePages?  为什么需要HugePages？

HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your combined database SGAs is large (like more than 8GB, can even be important for smaller), you will need HugePages configured. Note that the size of the SGA matters. Advantages of HugePages are:  如果您有较大的RAM和SGA，则HugePages对于在Linux上提高Oracle数据库性能至关重要。如果多个数据库SGA的和值很大（例如超过8GB，甚至对于较小的SGA也很重要），则需要配置HugePages。请注意，SGA的大小很重要。HugePages的优点是：

• Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.  较大的页面大小和较少的页面数：默认页面大小为4K，而HugeTLB大小为2048K。这意味着系统将需要处理少512倍的页面。
• Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual address range than a regular sized page, a probability of getting a TLB hit per TLB entry with HugePages are higher than with regular pages. This reduces the number of times page tables are walked to obtain physical address from a virtual address.  减少页面表遍历：由于HugePage比常规大小的页面覆盖更大的连续虚拟地址范围，因此使用HugePages的每个TLB条目获得TLB命中的可能性高于常规页面。这减少了遍历页表以从虚拟地址获取物理地址的次数。
• Less Overhead for Memory Operations: On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.  减少内存操作的开销：在虚拟内存系统（任何现代OS）上，每个内存操作实际上是两个抽象内存操作。使用HugePages，由于要处理的页面数量较少，因此可以避免页面表访问中可能出现的瓶颈。
• Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux kernel will use less memory to create pagetables to maintain virtual to physical mappings for SGA address range, in comparison to regular size pages. This makes more memory to be available for process-private computations or PGA usage.  更少的内存使用：从Oracle数据库的角度来看，与常规大小的页面相比，Linux内核通过HugePages将使用更少的内存来创建页表来维护SGA地址范围的虚拟到物理映射。这使更多的内存可用于进程专用计算或PGA使用。
• No Swapping: We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.  不交换：我们必须避免在所有Document 1295478.1上的Linux OS上进行交换。HugePages是不可交换的（而常规页面是可交换的）。因此，没有页面替换机制的开销。HugePages通常被认为是固定的。
• No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1  没有'kswapd'操作：如果要分页的区域很大（例如，用于50GB内存的1300万页表项），则kswapd将变得非常繁忙，并且将使用大量CPU资源。使用HugePages时，kswapd不会参与管理它们。另见Document 361670.1

There is a general misconception where the HugePages is a feature specific to 32-bit Linux.  HugePages is a generic feature available to all word-sizes and architectures. Just that there are some specifics with 32-bit platforms. Please see Document 361323.1 for further references.  有一个普遍的误解，其中HugePages是特定于32位Linux的功能。HugePages是可用于所有字长和体系结构的通用功能。只是32位平台有一些细节。请参阅Document 361323.1，以获取更多参考。

The HugePages configuration described in this document does not cause the O/S components to use HugePages. HugePages will be used by applications which explicitly make use of HugePages in their code (like majority of Oracle RDBMS SGA given proper configuration). Therefore, you will still see swap usage on the system as the regular O/S components, or non-HugePages-aware applications use swappable pages. 本文档中描述的HugePages配置不会导致O/S组件使用HugePages。HugePages将由在其代码中显式使用HugePages的应用程序使用（就像大多数Oracle RDBMS SGA给出了正确的配置一样）。因此，您仍将在系统上看到交换使用情况，因为常规的O/S组件或非HugePages感知的应用程序都使用可交换页面。

How to Configure  如何配置

The configuration steps below will guide you to do a persistent system configuration where you would need to do a complete reboot of the system. Please plan your operations accordingly:  以下配置步骤将指导您进行持久性系统配置，在该配置中，您需要完全重新引导系统。请相应地计划您的操作：

Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than installed RAM. e.g. If you have 64GB RAM installed, you may set:  第1步：有MEMLOCK在/etc/security/limits.conf文件中的用户限制设置。将该值（以KB为单位）设置为略小于已安装的RAM。例如，如果您安装了64GB RAM，则可以设置：

*   soft   memlock    60397977
*   hard   memlock    60397977

There is no harm in setting this value larger than your SGA requirements.
将此值设置为大于SGA要求没有任何害处。
The parameters will be set by default on:  这些参数将默认设置为：

• Oracle Linux with oracle-validated package (See Document 437743.1) installed.
• Oracle Exadata DB compute nodes

These settings can also be done in a *.conf file under /etc/security/limits.d/ directory. Also if there is a *.conf file in /etc/security/limits.d/ that has settings for memlock, setting it up in /etc/security/limits.conf may not work. Make sure you check the files /etc/security/limits.d/*.conf files.

这些设置也可以在/etc/security/limits.d/目录下的*.conf文件中进行。另外，如果/etc/security/limits.d/中有一个*.conf文件，该文件具有用于memlock的设置，则在/etc/security/limits.conf中进行设置可能会不起作用。确保检查文件/etc/security/limits.d/*.conf文件。

Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit

步骤2：重新登录到Oracle产品所有者帐户(e.g. 'oracle') 并检查memlock

$ ulimit -l
60397977

Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for further information in case you encounter the error below:  

步骤3：如果您具有Oracle Database 11g或更高版本，则创建的默认数据库将使用与HugePages不兼容的自动内存管理（AMM）功能。在继续操作之前，请禁用AMM。要禁用此功能，请将初始化参数MEMORY_TARGET和MEMORY_MAX_TARGET设置为0(zero)。如果遇到以下错误，请参阅Document 749851.1以获取更多信息：

ORA-00845: MEMORY_TARGET not supported on this system

Note: Starting in 11.2.0.3, AMM is no longer configured by default if DBCA detects that the machine has more than 4GB of RAM. See Document 1453227.1 for details.  

注意：从11.2.0.3开始，如果DBCA检测到计算机的RAM大于4GB，则默认情况下将不再配置AMM。有关详细信息，请参见Document 1453227.1

Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production. Use the script hugepages_settings.sh in Document 401749.1 to calculate the recommended value for the vm.nr_hugepageskernel parameter. e.g.:  

步骤4：确保所有数据库实例（包括ASM实例）均已启动，并且可以在生产环境中运行。使用脚本  hugepages_settings.sh在Document 401749.1来计算的推荐值vm.nr_hugepages内核参数。例如：

$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
$

Note: You can also calculate a proper value for the parameter yourself but that is not advised if you do not have extensive experience with HugePages and concepts.  注意：您也可以自己计算参数的适当值，但是如果您对HugePages和概念没有丰富的经验，则不建议这样做。

Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:

步骤5：编辑文件/etc/sysctl.conf并在此处设置vm.nr_hugepages参数：

...
vm.nr_hugepages = 1496
...

This will make the parameter to be set properly with each reboot. 这将使每次重新启动时正确设置参数。

Step 6: Stop all the database instances and reboot the server
步骤6：停止所有数据库实例并重新启动服务器
(Although settings could have been done dynamically they would not be effective to the extent we require before a reboot. The best practice is to do a persistent system configuration and perform a reboot to complete the configuration as presented through the steps above) （尽管设置可以动态完成，但在重新启动之前，设置达到我们要求的程度无效。最佳做法是进行持久性系统配置并执行重新启动以完成操作通过上述步骤显示的配置）

What If the Database / SGA Configurations Change?  如果Database / SGA配置发生更改怎么办？

The performed configuration is basically based on the RAM installed and combined size of SGA of database instances you are running. Based on that when:  所执行的配置基本上是基于安装的RAM和正在运行的数据库实例的SGA的总大小。基于以下情况

• Amount of RAM installed for the Linux OS changed  为Linux OS安装的RAM数量已更改
• New database instance(s) introduced  引入了新的数据库实例
• SGA size / configuration changed for one or more database instances  一个或多个数据库实例的SGA大小/配置已更改

you should revise your HugePages configuration to make it suitable to the new memory framework. If not you may experience one or more problems below on the system:  您应该修改HugePages配置以使其适合新的内存框架。如果不是这样，您可能会在系统上遇到以下一个或多个问题

• Poor database performance  数据库性能不佳
• System running out of memory or excessive swapping  系统内存不足或交换过多
• Database instances cannot be started  数据库实例无法启动
• Crucial system services failing  关键系统服务失败

Check and Validate the Configuration  检查并验证配置

After the system is rebooted, make sure that your database instances (including the ASM instances) are started. Automatic startup via OS configuration or CRS, or manual startup (whichever method you use) should have been performed. Check the HugePages state from /proc/meminfo. e.g.:

重新引导系统后，请确保已启动数据库实例（包括ASM实例）。应该已经执行了通过OS配置或CRS的自动启动，或者手动启动（无论使用哪种方法）。从 /proc/meminfo 检查HugePages状态。例如：

# grep HugePages /proc/meminfo
HugePages_Total:    1496
HugePages_Free:      485
HugePages_Rsvd:      446
HugePages_Surp:        0

The values in the output will vary. To make sure that the configuration is valid, the HugePages_Free value should be smaller than HugePages_Total.  输出中的值将有所不同。为确保配置有效，HugePages_Free值应小于HugePages_Total。

Also there should be some HugePages_Rsvd if PRE_PAGE_SGA is 'false' for all the Oracle database instances.. HugePages_Rsvd counts free pages that are reserved for use (requested for an SGA, but not touched/mapped yet). PRE_PAGE_SGA determines if the all SGA pages are read-in when the instance starts up. If parameter is set to 'true' then the OS page table entries are prebuilt for each page of the SGA, leading to HugePages reservation of those pages. For Oracle database versions before 12.1 the default value for PRE_PAGE_SGA is 'false'. So the HugePages_Rsvd would be higher than 0. But since 12c the PRE_PAGE_SGA defaults to 'true' which would cause the HugePages_Rsvd to be 0.

如果参数PRE_PAGE_SGA在所有Oracle数据库实例的值为'false'，则还应该有一些HugePages_Rsvd .. HugePages_Rsvd会统计保留使用free的网页（要求为SGA，但没有touched/mapped）。PRE_PAGE_SGA确定实例启动时是否已读入所有SGA页。如果将参数设置为'true'，则将为SGA的每个页面预构建OS页面表条目，从而导致保留这些页面的HugePages。对于12.1之前的Oracle数据库版本，PRE_PAGE_SGA的默认值为'false'。因此，HugePages_Rsvd将大于0。但是由于12c，PRE_PAGE_SGA默认为'true'，这将导致HugePages_Rsvd为0。

The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your total combined SGA as instances allocate pages dynamically and proactively as needed.  当实例根据需要动态和主动分配页面时，Hugepages_Free和HugePages_Rsvd的总和可能小于您的合并SGA总数。

Troubleshooting  故障排除

Some of the common problems and how to troubleshoot them are listed in the following table: 

下表列出了一些常见问题及其解决方法

Symptom	Possible Cause	Troubleshooting Action
System is running out of memory or swapping 系统内存不足或正在交换	Not enough HugePages to cover the SGA(s) and therefore the area reserved for HugePages are wasted where SGAs are allocated through regular pages. 没有足够的HugePages来覆盖SGA，因此，为HugePages保留的区域被浪费在通过常规页面分配SGA的地方。	Review your HugePages configuration to make sure that all SGA(s) are covered. 查看您的HugePages配置，以确保覆盖所有SGA。
Databases fail to start 数据库无法启动	memlock limits are not set properly memlock 限制设置不正确	Make sure the settings in limits.conf apply to database owner account. 确保limits.conf中的设置适用于数据库所有者帐户。
One of the database fail to start while another is up 其中一个数据库无法启动，而另一个数据库正在启动	The SGA of the specific database could not find available HugePages and remaining RAM is not enough. 特定数据库的SGA找不到可用的HugePage，并且剩余的RAM不足。	Make sure that the RAM and HugePages are enough to cover all your database SGAs 确保RAM和HugePages足以覆盖所有数据库SGA
Cluster Ready Services (CRS) fail to start CRS无法启动	HugePages configured too large (maybe larger than installed RAM) HugePages配置太大（可能大于已安装的RAM）	Make sure the total SGA is less than the installed RAM and re-calculate HugePages. 确保总SGA小于已安装的RAM，然后重新计算HugePages。
HugePages_Total = HugePages_Free	HugePages are not used at all. No database instances are up or using AMM. 完全不使用HugePages。没有数据库实例启动或使用AMM。	Disable AMM and make sure that the database instances are up. See Doc ID 1373255.1 禁用AMM并确保数据库实例已启动
Database started successfully and the performance is slow 数据库启动成功，性能降低	The SGA of the specific database could not find available HugePages and therefore the SGA is handled by regular pages, which leads to slow performance 特定数据库的SGA找不到可用的HugePage，因此SGA由常规页面处理，这导致性能降低	Make sure that the HugePages are many enough to cover all your database SGAs 确保HugePages足以覆盖您所有的数据库SGA

Known Problems and Limitations  已知问题和局限性

Below are some of the known and related problems and limitations related to the HugePages feature:

以下是与HugePages功能相关的一些已知问题和相关问题和限制：

• Document 749851.1 HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux
• Document 829850.1 Hugepages Are Not Used by Database Buffer Cache
• Document 803238.1 Oracle Not Utilizing Hugepages
• Document 728063.1 Setup HugePages in an Guest Does Not Work with Oracle VM 2.1 or 2.1.1
• Document 860350.1 /proc/meminfo Does Not Provide HugePages Information on Oracle Enterprise Linux (OEL5)
• Document 1557478.1 ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and UEK2 Kernels

Further Reading  进一步阅读

To be able to do advanced / manual configurations with HugePages you need to understand the implementation and theory behind the concept. You may read the following for further information:

为了能够使用HugePages进行高级/手动配置，您需要了解概念的实现和理论。您可以阅读以下内容以获取更多信息：

• Document 361323.1 HugePages on Linux: What It Is... and What It Is Not...
• Document 401749.1 Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration
• Document 397568.1 Linux IA64 example of allocating 48GB SGA using hugepages

REFERENCES

NOTE:361323.1 - HugePages on Linux: What It Is... and What It Is Not...

NOTE:1373255.1 - 11.2.0.1/11.2.0.2 to 11.2.0.3 Grid Infrastructure and Database Upgrade on Exadata Database Machine
NOTE:401749.1 - Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration

NOTE:361670.1 - Slow Performance with High CPU Usage on 64-bit Linux with Large SGA