centos7 hugepages的设置

当前系统centos7

[root@qht116 ~]# uname -a
Linux qht116 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
 

oracle 18c安装好后，数据库启动时alert log报以下提醒，是没有配置hugepage的原因

系统默认的pagsize只有4k，对于大sga来说，pagesize对数据库和系统的交换性能起到很大的影响，centos7已自动开启2M的pagesize，不过需要手动指定pagesize的数量。

**********************************************************************
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
 Per process system memlock (soft) limit = 64K
 Expected per process system memlock (soft) limit to lock
 SHARED GLOBAL AREA (SGA) into memory: 100G
 Available system pagesizes:
  4K, 2048K
 Supported system pagesize(s):
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
        4K       Configured               8        26214408        NONE
     2048K                0           51201               0        NONE
RECOMMENDATION:
 1. For optimal performance, configure system with expected number
 of pages for every supported system pagesize prior to the next
 instance restart operation.
 2. Increase per process memlock (soft) limit to at least 100GB
 to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
**********************************************************************

对于设置多少pagesize，Oracle在MOS上发布出一个Shell脚本hugepages_settings.sh，可以动态计算出推荐的HugePage数量。

#!/bin/bash
#
# hugepages_setting.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
NUM_PG=1
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
do
   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
   if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
   fi
done
# Finish with results
case $KERN in
   '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
          echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
   '2.6' | '3.8' | '3.10' | '4.1' | '4.14' ) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    *) echo "Unrecognized kernel version $KERN. Exiting." ;;
esac
# End

我计算一下当前系统的情况：

建议的数量51203个,每2M一个，正好是sga的大小(sag=100G)

[root@qht116 ~]# chmod u+x hugepages_setting.sh 
[root@qht116 ~]# ./hugepages_setting.sh 
Recommended setting: vm.nr_hugepages = 51203

根据建议的数据修改/etc/sysctl.con

[root@qht116 ~]# echo "vm.nr_hugepages = 51203" >> /etc/sysctl.conf 
[root@qht116 ~]# cat /etc/sysctl.conf 
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

fs.file-max = 6815744  
kernel.sem = 250 32000 100 128  
kernel.shmmni = 4096  
kernel.shmall = 1073741824  
kernel.shmmax = 6597069766656
kernel.panic_on_oops = 1  
net.core.rmem_default = 262144  
net.core.rmem_max = 4194304  
net.core.wmem_default = 262144  
net.core.wmem_max = 1048576  
#net.ipv4.conf.eth3.rp_filter = 2       
#net.ipv4.conf.eth2.rp_filter = 2
#net.ipv4.conf.eth0.rp_filter = 1  
fs.aio-max-nr = 1048576  
net.ipv4.ip_local_port_range = 9000 65500 

vm.nr_hugepages = 51203

[root@qht116 ~]# sysctl -p

接着还要修改/etc/security/limits.conf,开启用户对内存的限制

51203*2048=104,863,744

将下面两行加入/etc/security/limits.conf

*   soft   memlock    104863744
*   hard   memlock    104863744

重新Log In后检查内存限制生效了

[root@qht116 ~]#  ulimit -l
104863744

[root@qht116 ~]# cat /etc/security/limits.conf 

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 65536
oracle hard nofile 65536
*   soft   memlock    104863744
*   hard   memlock    104863744

# End of file

重启了系统后，HugePages_Total这个值才对。
[oracle@qht116 ~]$ grep HugePages /proc/meminfo
AnonHugePages:     26624 kB
HugePages_Total:   51203
HugePages_Free:    51203
HugePages_Rsvd:        0
HugePages_Surp:        0

重启数据库后就正常了

**********************************************************************
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
 Per process system memlock (soft) limit = 100G
 Expected per process system memlock (soft) limit to lock
 SHARED GLOBAL AREA (SGA) into memory: 100G
 Available system pagesizes:
  4K, 2048K
 Supported system pagesize(s):
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
        4K       Configured               8        20571010        NONE
     2048K           51203        51201           51201        NONE
RECOMMENDATION:
 1. For optimal performance, configure system with expected number
 of pages for every supported system pagesize prior to the next
 instance restart operation.
**********************************************************************

参考：

ORACLE-BASE - Configuring HugePages for Oracle on Linux (x86-64)

为Linux版本Oracle 11gR2配置HugePage - 张冲andy - 博客园