大页内存的资料

• debian wiki Hugepages
  每个发行版开启大页内存的方式，可能都有点差异，需以文档为准。

• 大页内存(Huge Pages)
  本文的作者依据在项目中运用大页内存技术解决实际问题的经验，对适用场景，做了一些总结：

  • 程序耗费的内存巨大。
  • 程序访问内存时，随机访问，局部性很差。

• What is the purpose of “khugepaged” Linux Kernel Daemon?

  In HugePage (which must be hardware supported), there are two different HugePage sizes on the x86_64 architecture: 2 MiB and 1 GiB.

  • If the CPU supports 2 MiB pages, it has the “PSE” cpuinfo flag.

  • If the CPU supports 1 GiB pages, it has the “PDPE1GB” cpuinfo flag.

  使用如下命令，检查CPU是否支持大页。

  grep pse /proc/cpuinfo | uniq
  grep pdpe1gb /proc/cpuinfo | uniq
  

• Linux 大页内存 Huge Pages 虚拟内存

• 大页内存(Huge Pages)

  通过增大操作系统页的大小来减小页表，从而避免快表缺失。
  适用范围，程序耗费内存很小或者程序的访存局部性很好，大页内存很难获得性能提升。

• Linux 调优篇：虚拟化调优（hugepage 大页内存）* 叁

• Linux大页内存(hugepage)使用

• 调整Linux实例大页内存的方法

• 内存管理特性分析（十二）:大页(huge pages)技术分析

• Linux 操作系统原理 — 内存 — 大页内存

• Reliably allocating huge pages in Linux

libhugetlbfs

• libhugetlbfs
  官方文档中应该提供了从源码构建为二进制的方法，需要研究一下。

• libhugetlbfs(7) - Linux man page
  适用场景的说明：

  • use large amounts of address space
  • suffer a performance hit due to TLB misses

  在相同的测试场景下，检测大页内存特性对性能是否有提升的方法：

  Wall-clock time or oprofile can be used to determine if there is a performance benefit from using libhugetlbfs or not.

  在应用程序启动时，启用大页内存特性，执行如下命令：

  export [environment options] [LD_PRELOAD=libhugetlbfs.so] target_application
  

  上述命令中的环境变量，详细说明可参考HOWTO：

  • HUGETLB_MORECORE
  • HUGETLB_SHM
  • HUGETLB_ELFMAP
  • HUGETLB_FORCE_ELFMAP
  • HUGETLB_RESTRICT_EXE
  • HUGETLB_MORECORE_SHRINK
  • HUGETLB_NO_PREFAULT
  • HUGETLB_NO_RESERVE
  • HUGETLB_MORECORE_HEAPBASE
  • HUGETLB_PATH
  • HUGETLB_SHARE

• using libhugetlbfs to allocate pages

• Introduction to libhugetlbfs
  安装软件包，执行如下命令：

  sudo apt-get install libhugetlbfs-dev libhugetlbfs-bin -y
  sudo ln -s /usr/bin/ld.hugetlbfs /usr/share/libhugetlbfs/ld
  

  上述命令使用发行版提供的软件包。
  启用大页内存，执行如下命令：

  echo 1000 > /proc/sys/vm/nr_hugepages
  cat /proc/meminfo | grep HugePages_
  

  编译和链接的选项，如下：

  -B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl,--no-as-needed
  

  完整的编译和链接的命令样例，如下：

  gcc -B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl,--no-as-needed -o memory memory.c
  

  样例代码memory.c，内容如下：

  #include <stdio.h>
  #include <stdlib.h>
  #include <signal.h>
  #include <unistd.h>
  
  #define NUM_PAGES   100
  
  void *memory;  // pointer to allocated memory
  
  // signal handler for Control-C
  void sigint_handler(int sig)
  {
      printf("\nControl-C (SIGINT) signal caught!\n");
  
      // Free the memory
      free(memory);
  
      printf("Exiting...\n");
      exit(0);
  }
  
  int main()
  {
      // Determine the system's page size using sysconf
      long page_size = sysconf(_SC_PAGESIZE);
  
      if (page_size == -1)
      {
          perror("sysconf error");
          return 1;
      }
  
      printf("System page size is %ld\n", page_size);
  
      // Register a signal handler for SIGINT
      signal(SIGINT, sigint_handler);
  
      // Calculate the total size to allocate for NUM_PAGES
      long total_size = NUM_PAGES * page_size;
  
      // Allocate the memory using malloc
      memory = malloc(total_size);
  
      if (memory == NULL)
      {
          perror("malloc error");
          return 1;
      }
  
      printf("Allocated %ld bytes (%d pages) of memory.\n", total_size, NUM_PAGES);
  
      // Print the process ID
      printf("Process ID: %d\n", getpid());
  
      // Wait for Control-C
      while (1)
          sleep(1);
  
      return 0;
  }
  

  启动应用程序时，指定启用大页内存，执行如下命令：

  HUGETLB_ELFMAP=RW ./memory
  

  检查大页内存的使用情况，执行如下命令：

  cat /proc/<pid>/smaps | less
  

  启动程序时，输出更多的调试信息，执行如下命令：

  HUGETLB_ELFMAP=RW HUGETLB_DEBUG=3  ./memory
  

• Red Hat 4.138. libhugetlbfs

Transparent Hugepage

THP即Transparent Huge Pages。

• Transparent Hugepage Support

  THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task’s address space. Unless THP is completely disabled, there is khugepaged daemon that scans memory and collapses sequences of basic pages into PMD-sized huge pages.

• Transparent Hugepage Support

• How to use, monitor, and disable transparent hugepages in Red Hat Enterprise Linux 6 ,7, 8, 9, and 10 Beta?

• Transparent Huge Pages Refresher

• Red Hat 7.4. Configuring Transparent Huge Pages

  With THP, the kernel automatically assigns huge pages to processes, so huge pages do not need to be reserved manually.

• Oracle Transparent HugePages

  The Transparent HugePages (THP) feature is enabled by default in Oracle Linux. With THP, the kernel automatically assigns huge pages to processes. With THP, you can assign only 2 MB pages on x86_64 platforms.

• Oracle Disabling Transparent HugePages

  Oracle recommends that you disable Transparent HugePages before you start installation.
  Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.
  Transparent HugePages can cause memory allocation delays during runtime.

  检查Transparent HugePages是否启用，执行如下命令：

  cat /sys/kernel/mm/transparent_hugepage/enabled
  

  如果找不到目录/sys/kernel/mm/transparent_hugepage，说明内核不支持Transparent HugePages特性。

  参照Oracle的文档，配置Transparent HugePages。

• Transparent Huge Pages on Linux

  Transparent Huge Pages is only relevant to servers with Intel processors.
  SAP strongly recommends that you disable Transparent Huge Pages on all your SAP HANA servers.

  确认当前是否有进程使用了大页内存，可执行如下命令：

  cat /proc/meminfo/ | grep AnonHugePages
  

• The Linux Process Journey — “khugepaged”

  The kernel thread “khugepaged” is created using the “kthread_run()” function. It is responsible for the “Transparent Hugepage Support” (aka THP).

  相关代码，如下：

  int start_stop_khugepaged(void)
  {
    int err = 0;
  
    mutex_lock(&khugepaged_mutex);
    if (hugepage_pmd_enabled()) {
      if (!khugepaged_thread)
        khugepaged_thread = kthread_run(khugepaged, NULL,
                "khugepaged");
      if (IS_ERR(khugepaged_thread)) {
        pr_err("khugepaged: kthread_run(khugepaged) failed\n");
        err = PTR_ERR(khugepaged_thread);
        khugepaged_thread = NULL;
        goto fail;
      }
  
      if (!list_empty(&khugepaged_scan.mm_head))
        wake_up_interruptible(&khugepaged_wait);
    } else if (khugepaged_thread) {
      kthread_stop(khugepaged_thread);
      khugepaged_thread = NULL;
    }
    set_recommended_min_free_kbytes();
  fail:
    mutex_unlock(&khugepaged_mutex);
    return err;
  }
  

• I’ve had bad luck with transparent hugepages on my Linux machines
  关闭THP特性，执行如下命令：

  echo never >/sys/kernel/mm/transparent_hugepage/enabled
  

  限制THP的defragmentation的效果，执行如下命令：

  echo never >/sys/kernel/mm/transparent_hugepage/defrag
  

• Using Huge Pages in Linux Applications Part 2: Transparent HugePage

  HugeTLB is much simpler.

  • use HugeTLB if performance and full control over memory management is your top priority.
  • otherwise, use THP.

  上述观点，比较有意思。

HugeTLB

• HugeTLB Pages

  The libhugetlbfs library provides a wide range of userspace tools to help with huge page usability, environment setup, and control.

• Red Hat 7.3. Configuring HugeTLB Huge Pages

• HugeTLB - Large Page Support in the Linux Kernel

• Oracle HugeTLB Pages

  HugeTLB pages are also called static huge pages.

  参照Oracle的文档，配置HugeTLB特性。

• File-Based Configuration Parameters for HugeTLB Pages

  nr_hugepages defines the number of pages in a pool, including the following:

  • File /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages for the number of pages in the pool of 2 MB pages.
  • File /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages for the number of pages in the pool of 1 GB pages.

  相关的参数：

  • nr_hugepages
  • nr_overcommit_hugepages
  • free_hugepages
  • surplus_hugepages

  相关的参数目录：

  • /proc/sys/vm
  • /sys/kernel/mm/hugepages
  • /sys/devices/system/node

参考资料

• The /proc/meminfo File in Linux
  MemTotal，系统的总内存。
  MemFree，当前未使用的内存。
  MemAvailable，当前可用的内存。

  Buffers，临时存储空间，一般不超出20MB。