在Linux IA64上使用hugepages分配48GB SGA的示例 (Doc ID 397568.1)

Linux IA64 example of allocating 48GB SGA using hugepages (Doc ID 397568.1)

APPLIES TO:

Linux OS - Version 2.6.9 to 2.6.9
Oracle Cloud Infrastructure - Version N/A and later
Linux Itanium
***Checked for relevance on 07-Arp-2010***
***Checked for relevance on 23-Oct-2011***
***Checked for relevance on 27-Mar-2013***
***Checked for relevance on 03-Sep-2014***
***Checked for relevance on 24-Jul-2017***
Linux Kernel - Version: 2.6.9 to 2.6.9

GOAL

Startup a 48GB SGA on IA64 using hugepages. 使用大量页面在IA64上启动48GB SGA

SOLUTION

Setup

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 52428800 ! 50GB

# sysctl -w kernel.shmmax=53687091200
kernel.shmmax = 53687091200

# sysctl -w vm.nr_hugepages=192

Now, check and see if you got them, if not, add to /etc/sysctl.conf & reboot so memory is contiguous:  现在，检查是否有它们，如果没有，则将其添加到/etc/sysctl.conf并重新启动

[root@<HOST> oracle]# cat /proc/meminfo
MemTotal: 66673600 kB
MemFree: 12929696 kB
Buffers: 88176 kB
Cached: 2244144 kB
SwapCached: 0 kB
Active: 888160 kB
Inactive: 1573472 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 66673600 kB
LowFree: 12929696 kB
SwapTotal: 4128736 kB
SwapFree: 4128736 kB
Dirty: 80 kB
Writeback: 0 kB
Mapped: 165424 kB
Slab: 114352 kB
Committed_AS: 381888 kB
PageTables: 4464 kB
VmallocTotal: 137430321152 kB
VmallocUsed: 5264 kB
VmallocChunk: 137430314992 kB
HugePages_Total: 192 
HugePages_Free: 192
Hugepagesize: 262144 kB

Remember, this is 64bit, so you  don't need to set the init.ora parameter use_indirect_data_buffers=true, this parameter is for 32-bit only  (VLM Window 32-bit stuff).  请记住，这是64位，因此您不需要设置init.ora参数use_indirect_data_buffers = true，该参数仅适用于32位（VLM Window 32位）。

Startup oracle

SQL> startup nomount
ORACLE instance started.

Total System Global Area 5.1540E+10 bytes
Fixed Size 2090400 bytes
Variable Size 3690994272 bytes
Database Buffers 4.7832E+10 bytes
Redo Buffers 14680064 bytes

 How it looks running:  

# ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 4980742 oracle 640 2097152 14
0x00000000 5013511 oracle 600 51539607552 14
0x4d06d874 5046280 oracle 640 114688 14

$ cat /proc/meminfo
MemTotal: 66673600 kB
MemFree: 12845984 kB
Buffers: 88192 kB
Cached: 2248256 kB
SwapCached: 0 kB
Active: 1001888 kB
Inactive: 1573456 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 66673600 kB
LowFree: 12845984 kB
SwapTotal: 4128736 kB
SwapFree: 4128736 kB
Dirty: 704 kB
Writeback: 0 kB
Mapped: 297632 kB
Slab: 114400 kB
Committed_AS: 3046240 kB
PageTables: 7664 kB
VmallocTotal: 137430321152 kB
VmallocUsed: 5264 kB
VmallocChunk: 137430314992 kB
HugePages_Total: 192
HugePages_Free: 0 
Hugepagesize: 262144 kB

Now look at /proc/<PID OF PMON>/maps to check if the shared memory segments are backed by hugetlb or not.  现在查看/proc/<PID OF PMON>/maps，以检查共享内存段是否由ugeltlb支持。
Non-HUGETLB allocations show allocation starting at address 40000000000:  Non-HUGETLB显示从地址40000000000开始的分配：

# grep 4980742 /proc/24048/maps # grep by shmid
40000000000-40000200000 rw-s 00000000 00:07 4980742 /SYSV00000000 (deleted)
# grep 5046280 /proc/24048/maps
40040000000-4004001c000 rw-s 00000000 00:07 5046280 /SYSV4d06d874 (deleted)

 HUGETLB allocation show allocation of shared memory segments starting at address 8000000000000000:  HUGETLB显示从地址8000000000000000开始的共享内存段的分配

# grep 5013511 /proc/24048/maps

8000000000000000-8000000c00000000 rw-s 00000000 00:0b 5013511 /71 (deleted)

Common problems   常见问题

This next section discusses some common problem seen when using hugepages but where the proper O/S prep work has not been done. 下一部分将讨论使用hugepages但未完成适当的操作系统准备工作时遇到的一些常见问题。

Problem 1:
----------
Too small shmmax results in not all shm segment allocated as hugetlb: shmmax太小导致不是所有的shm段都分配为hugetlb

# cat /proc/meminfo
.....
HugePages_Total: 192
HugePages_Free: 31 <
# ipcs -a
.....
------ Shared Memory Segments --------key shmid owner perms bytes nattch status
0x00000000 4685830 oracle 640 2097152 14
0x00000000 4718599 oracle 600 268435456 14
0x00000000 4751368 oracle 600 8589934592 14
0x00000000 4784137 oracle 600 8589934592 14
0x00000000 4816906 oracle 600 8589934592 14
0x00000000 4849675 oracle 600 8589934592 14
0x00000000 4882444 oracle 600 8589934592 14
0x00000000 4915213 oracle 640 8573157376 14 
0x4d06d874 4947982 oracle 640 114688 14

Problem 2:
----------
If the max locked mem ulimit is smaller than requested SGA size, result is no HUGETLB allocated. Setup problem, not-a-bug.
如果最大锁定内存ulimit小于请求的SGA大小，则结果是未分配HUGETLB。安装问题，不是错误。
Using STRACE to analyze the startup problem:  使用STRACE分析启动问题
--------------------------------------------
Gather an strace of the SGA creation problem startup using:  使用以下命令收集SGA创建问题启动的记录

$ strace -f -o strace.log sqlplus '/ as sysdba'
SQL> startup nomount
SQL> exit

Problem 3:
----------
If you install a ia64 server using RAC and NFS storage, when you enable hugepages (vm.nr_hugepages) the server will hang during database open.  如果使用RAC和NFS存储安装ia64服务器，则在启用大页面（vm.nr_hugepages）时，该服务器将在数据库打开期间挂起。
Eliminating any of the 3 (RAC/NFS/Hugepages) will work around the problem.
消除这3个(RAC/NFS/Hugepages)中的任何一个都可以解决此问题。
This is normally seen during a DB restart, or also possibly as a boot hang, because CRS starts up the database during system boot.
通常，这是在数据库重新启动期间看到的，或者也可能是引导挂起，因为CRS在系统引导期间启动了数据库

Solution:  解决方案

Update NFS to properly avoid marking huge pages dirty during O_DIRECT reads. This accounting is not required on huge pages and can lead to deadlocks when NFS tries to mark an area in the middle of the huge page as dirty.
更新NFS以正确避免在O_DIRECT读取期间将大页标记为脏页。NFS试图将大页面中间的区域标记为脏区域时，不需要在大页面上进行此统计，并且可能导致死锁。
This check deals with a case where a dirty page would be operated on in a manner that breaks with hugetlb, though for ramfs it does fine.
此检查处理的情况是，脏页将以巨大的tlb中断的方式进行操作，尽管对于ramfs来说确实很好
Fixed in Oracle kernel version: 2.6.9-55.0.0.5.2.EL
Submitted to Red Hat for inclusion in RHEL 4.6

REFERENCES

NOTE:361323.1 - HugePages on Linux: What It Is... and What It Is Not...