NUMA相关

NUMA相关的几个概念

• Socket是一个物理上的概念，指的是主板上的cpu插槽。
• Node是一个逻辑上的概念，对应于socket。
• Core就是一个物理cpu 核,一个独立的硬件执行单元。
• Thread就是超线程的概念，是一个逻辑cpu，共享core上的执行单元。

NUMA使用node来管理cpu和内存。

# ll /sys/devices/system/node/
total 0
-r--r--r--. 1 root root 4096 May 14 21:19 has_cpu
-r--r--r--. 1 root root 4096 May 14 21:19 has_memory
-r--r--r--. 1 root root 4096 May 14 21:19 has_normal_memory
drwxr-xr-x. 4 root root    0 May  7 23:13 node0
drwxr-xr-x. 4 root root    0 May  7 23:13 node1
-r--r--r--. 1 root root 4096 May 14 21:19 online
-r--r--r--. 1 root root 4096 May 14 21:19 possible
drwxr-xr-x. 2 root root    0 May  7 23:13 power
-rw-r--r--. 1 root root 4096 May 14 21:19 uevent

Socket的信息可以通过/proc/cpuinfo查看，里面的physical id标示的就是socket号。

# cat /proc/cpuinfo | grep "physical id" | sort -u
physical id	: 0
physical id	: 1

查看每个Node的cpu

# ll /sys/devices/system/node/node0/cpu* 
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu0 -> ../../cpu/cpu0
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu1 -> ../../cpu/cpu1
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu10 -> ../../cpu/cpu10
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu11 -> ../../cpu/cpu11
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu12 -> ../../cpu/cpu12
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu13 -> ../../cpu/cpu13
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu14 -> ../../cpu/cpu14
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu15 -> ../../cpu/cpu15
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu16 -> ../../cpu/cpu16
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu17 -> ../../cpu/cpu17
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu18 -> ../../cpu/cpu18
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu19 -> ../../cpu/cpu19
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu2 -> ../../cpu/cpu2
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu20 -> ../../cpu/cpu20
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu21 -> ../../cpu/cpu21
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu22 -> ../../cpu/cpu22
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu23 -> ../../cpu/cpu23
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu24 -> ../../cpu/cpu24
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu25 -> ../../cpu/cpu25
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu3 -> ../../cpu/cpu3
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu4 -> ../../cpu/cpu4
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu5 -> ../../cpu/cpu5
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu52 -> ../../cpu/cpu52
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu53 -> ../../cpu/cpu53
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu54 -> ../../cpu/cpu54
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu55 -> ../../cpu/cpu55
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu56 -> ../../cpu/cpu56
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu57 -> ../../cpu/cpu57
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu58 -> ../../cpu/cpu58
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu59 -> ../../cpu/cpu59
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu6 -> ../../cpu/cpu6
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu60 -> ../../cpu/cpu60
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu61 -> ../../cpu/cpu61
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu62 -> ../../cpu/cpu62
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu63 -> ../../cpu/cpu63
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu64 -> ../../cpu/cpu64
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu65 -> ../../cpu/cpu65
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu66 -> ../../cpu/cpu66
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu67 -> ../../cpu/cpu67
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu68 -> ../../cpu/cpu68
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu69 -> ../../cpu/cpu69
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu7 -> ../../cpu/cpu7
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu70 -> ../../cpu/cpu70
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu71 -> ../../cpu/cpu71
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu72 -> ../../cpu/cpu72
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu73 -> ../../cpu/cpu73
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu74 -> ../../cpu/cpu74
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu75 -> ../../cpu/cpu75
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu76 -> ../../cpu/cpu76
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu77 -> ../../cpu/cpu77
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu8 -> ../../cpu/cpu8
lrwxrwxrwx. 1 root root    0 May  8 02:58 /sys/devices/system/node/node0/cpu9 -> ../../cpu/cpu9

查看cpu 相关参数

# lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
CPU(s):              104
Thread(s) per core:  2
Core(s) per socket:  26
Socket(s):           2

# lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
CPU(s):                88
Thread(s) per core:    2
Core(s) per socket:    22
Socket(s):             2

CPU(s) 为超线程数
Socket(s) 为物理cpu数
Core(s) per socket: 每颗物理cpu 22核
Thread(s) per core: 每个核有2个超线程

# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                88
On-line CPU(s) list:   0-87
Thread(s) per core:    2
Core(s) per socket:    22
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
Stepping:              1
CPU MHz:               1202.609
BogoMIPS:              4396.03
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              56320K
NUMA node0 CPU(s):     0-21,44-65
NUMA node1 CPU(s):     22-43,66-87

查看numa 相关信息

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
node 0 size: 130977 MB
node 0 free: 2772 MB
node 1 cpus: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
node 1 size: 131072 MB
node 1 free: 2477 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

Node0的本地内存大小为 130977 MB，Node1的本地内存大小为 131072 MB。
Node0到本地内存的distance为10，到node1的内存distance距离为21；
Node1到本地内存的distance为10，到node0的内存distance距离为21。

numactl

# numactl --hardware 
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
node 0 size: 130977 MB
node 0 free: 123557 MB
node 1 cpus: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
node 1 size: 131072 MB
node 1 free: 124540 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 128154 MB
node 0 free: 100217 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 129014 MB
node 1 free: 109274 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

$ numastat
                           node0           node1
numa_hit              2635013083      2451596149
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit             33789           34103
local_node            2634845682      2451337466
other_node                167401          258683

In more detail:

• numa_hit

A process wanted to allocate memory from this node, and succeeded.

• numa_miss

A process wanted to allocate memory from another node, but ended up with memory from this node.

• numa_foreign

A process wanted to allocate on this node, but ended up with memory from another node.

• local_node

A process ran on this node’s CPU, and got memory from this node.

• other_node

A process ran on a different node’s CPU and got memory from this node.

• interleave_hit

Interleaving wanted to allocate from this node and succeeded.

For easier reading you can use the numastat utility from the numactl package (http://oss.sgi.com/projects/libnuma/). Note that it only works well right now on machines with a small number of CPUs.