X86/X64的内存地址编址与寻址原理剖析

一、物理地址和逻辑地址

物理地址：加载到内存地址寄存器中的地址，内存单元的真正地址。在前端总线上传输的内存地址都是物理内存地址，编号从0开始一直到可用物理内存的最高端。这些数字被北桥(Nortbridge chip)映射到实际的内存条上。物理地址是明确的、最终用在总线上的编号，不必转换，不必分页，也没有特权级检查(no translation, no paging, no privilege checks)。

逻辑地址：CPU所生成的地址。逻辑地址是内部和编程使用的、并不唯一。例如，你在进行C语言指针编程中，可以读取指针变量本身值(&操作)，实际上这个值就是逻辑地址，它是相对于你当前进程数据段的地址（偏移地址），不和绝对物理地址相干。

如果你有内存测试和内存地址Decode这方面的需求，可以私信我，我们一起进步！

二、分页内存管理方案(Paging)

2.1 逻辑地址到物理地址的映射机制

分页的最大作用就在于：使得进程的物理地址空间可以是非连续的。

物理内存被划分为一小块一小块，每块被称为帧(Frame)。分配内存时，帧是分配时的最小单位，最少也要给一帧。在逻辑内存中，与帧对应的概念就是页(Page)。

逻辑地址的表示方式是：前部分是页码后部分是页偏移。

例如，已知逻辑空间地址为2^m个字节（也就是说逻辑地址的长度是m位），已知页大小是2^n字节。那么一共可以有2^(m-n)个页。因此页码部分会占m-n位，之后的n位，用来存储页偏移。

举个例子， 页大小为4B，而逻辑内存为32B（8页），逻辑地址0的页号为0，查询页表(page table)可知：页号0对应帧5，帧的大小等于页的大小4B，因此逻辑地址0映射为物理地址5*4+0=20。逻辑地址3（4*0+3，页号0对应帧5）映射为物理地址5*4+3=23。逻辑地址13(4*3+1，页号为3，偏移为1，因此帧号为2)，映射到物理地址9。

2.2 页表的实现

页表是逻辑地址转化到物理地址的关键所在。如何存储页表成为核心问题。

每个操作系统都有自己的方法来保存页表。绝大多数都会为每个进程分配一个页表。现在由于页表都比较大，所以放在内存中(以往是放在一组专用寄存器里)，其指针存在进程控制块(PCB)里，当进程被调度程序选中投入运行时，系统将其页表指针从进程控制块中取出并送入用户寄存器中。随后可以根据此首地址访问页表。

页表的存储方式是TBL(Translation look-aside buffer, 翻译后备缓冲器)+内存。TBL实际上是一组硬件缓冲所关联的快速内存。TBL中存储页表中的一小部分条目，条目以键值对方式存储。

 

2.3 页表的数据结构

2.3.1 层次化分页

对于具有32位逻辑地址空间的计算机系统，如果系统的页大小为4KB(2^12B)，那么页表可以拥有2^(32-12)个，也就是一百多万个条目，假设每个条目占有4B，那每个进程都需要4MB的物理地址空间来存放页表本身。而且，页表本身需要分配在连续内存中。

为此，Hierarchical Paging(层次化分页)被提出，实际上就是将页号分为两部分，第一部分作为索引，第二部分作为页号的偏移。

以一个4kb页大小的32位系统为例。一个逻辑地址被分为20位的页码和12位的页偏移。因为要对页表进行再分页，所以该页号可分为10位的页码和10位的页偏移。这样一个逻辑地址就表示如下形式：

地址转换过程如下：

地址由外向内转换，因此此方法也被称为forward-mapped page table(向前映射表)。

 

2.3.2 哈希页表

处理超过32位地址空间的常用方法是使用hashed page table(哈希页表)，并以虚拟页码作为哈希值。哈希页表的每一条目都包括一个链表的元素，这些元素哈希成同一位置。每个元素有三个域：虚拟页码，所映射的帧号，指向链表中下一个元素的指针。

个人看来，哈希页表的地址转换方式，实际上是Chaining(链接)方式，也就是一种哈希函数的溢出处理方式(另一种溢出处理方式叫做Open Addressing，开放寻址)，具体过程如下：

逻辑地址需要大于32bit的地址空间来表示，但是操作系统仍只有32bit来表示地址。此时人们便想到虚拟页地址，虚拟地址可以在32bit表示范围之内，然后利用哈希函数完成逻辑地址到虚拟地址的映射，由于虚拟地址更少，哈希函数会出现溢出，这里使用Chaining来解决溢出。

逻辑地址中的页号(下图中的p)经过哈希函数的计算，算出虚拟地址中的页号，根据虚拟页号可以在哈希表中以O(1)方式寻址，用p与链表中的每一个元素的第一个域相比较。如果匹配，那么相应的帧号就用来形成物理地址。如果不匹配，就对链表中的下一个节点进行比较，以寻找一个匹配的页号。

 

三、分段内存管理方案(Segmentation)

逻辑地址空间由一组段组成。每个段都有名字和长度。地址指定了段名称和段内偏移。因此用户通过两个量来指定地址：段名称和偏移。段是编号的，通过段号而非段名称来引用。因此逻辑地址由有序对构成：

 <segment-number,offset>(<段号s, 段内偏移d>)

段偏移d因该在0和段界限之间，如果合法，那么就与基地址相加而得到所需字节在物理内存中的地址。因此段表是一组基地址和界限寄存器对。

例如下图，有5个段，编号0~4，例如段2为400B开始于位置4300，对段2第53字节的引用映射成位置4300+53=4353。而段0字节1222的引用则会触发地址错误，因为该段的仅为1000B长(界限为1000)。

 

四、综合分页(Paging)和分段(Segmentation)内存管理方案

现有的Intel兼容计算机(x86)上，采用的内存管理方案是分段和分页合并的管理方案。

在这个方案中，逻辑地址，如前一节中所说，是由一个段标识符加上一个指定段内相对地址的偏移量，表示为 [段标识符：段内偏移量]。它的逻辑地址转换的过程如下图所示：

当CPU要执行一条引用了内存地址的指令时，转换过程就开始了。第一步是把逻辑地址转换成线性地址，由分段内存管理模块实现。

 

4.1 逻辑地址转换到线性地址

在IBM OS/2 32位版本的操作系统，和Intel 386的环境下。操作系统采用的内存分配方式就是分段和分页合并的方式。

逻辑地址的实际上是一对<选择符，偏移>。

选择符的内容如下：

从左开始，13比特位是索引号(或者称为段号)，通过这个索引，可以定位到段描述符(segment descriptor)，而段描述符是可以真正记载了有关一个段的位置和大小信息， 以及访问控制的状态信息。段描述符一般由8个字节组成。由于8字节较大，而Intel为了保持向后兼容，将段寄存器仍然规定为16 比特(尽管每个段寄存器事实上有64比特，但对于程序员来说，段寄存器就是16-bit的)，那么很明显，我们无法通过16-bit长度的段寄存器来直接引用64-bit的段描述符。因此在逻辑地址中，只用13bit记录其索引。而真正的段描述符，被放于数组之中。

这个内存中的数组就叫做GDT(Global Descriptor Table，全局描述表)，Intel的设计者门提供了一个寄存器GDTR用来存放GDT的入口地址。程序员将GDT设定在内存中某个位置之后，可以通过LGDT指令将GDT的入口地址装入此寄存器GDTR，从此以后，CPU就根据此寄存器中的内容作为GDT的入口来访问GDT了。

除了GDT之外，还有LDT(Local Descriptor Table，本地描述表)，但与GDT不同的是，LDT在系统中可以存在多个，每个进程可以拥有自己的LDT。LDT的内存地址在LDTR寄存器中。

在之前图中的TI位，就是用来表示此索引所指向的段描述符是存于全局描述表中，还是本地描述表中。=0，表示用GDT，=1表示用LDT。

RPL位，占2bit，是保护信息位。

计算出段描述符后，加上偏移量，便是线性地址。转换过程如下：

 

4.2 线性地址到物理地址

此过程由分页内存管理模块实现，在Intel 386的环境下，线性地址转换为物理地址的过程，和层次分页机制中，逻辑地址转换为物理地址的方法类似。如下图。

如上图所示，线性地址是32位，

对于4KB小页的情况：高十位是页目录项索引，中间十位是页表项索引，最后12位是页内偏移；

对于4MB大页的情况：高十位是页目录项索引，低22位都是页内偏移。

页目录物理基地址存放在CR3中，共有1024项，因此用线性地址高10位作索引，找到相应的页目录项。在小页模式中，该项保存的是页表的高20位地址，因为页表只有4KB，所以低12位不需要。通过线性地址中间十位作页表项索引和页表基址进行计算得到页表项，该项中保存的物理页面的基址，基址加上线性地址低12位页内偏移，就得到了物理地址。在大页模式中，就省去了查页表这一步骤。

 

4.3 逻辑地址到线性地址，再到物理地址

Intel 80386的地址转换全过程如下图：

 

五、编址与寻址

5.1 编址

我们以一个256MB的内存条为例进行说明：

1. 按字编址：    对于这个256M内存来说，它的寻址范围是64M，而每个内存地址可以存储32bit数据。
2. 按半字编址：对于这个256M内存来说，它的寻址范围是128M，而每个内存地址可以存储16bit数据。
3. 按字节编址：对于这个256M内存来说，它的寻址范围是256M，而每个内存地址可以存储8bit数据。

 

现在的计算机主要都是采用按字节编址的方式。所以我们可以把内存简单的看成一个线性数组，数组每个元素的大小为8bit，我们称为一个存储单元，对这些存储单元我们从0x00000000开始沿着Rank的访问编址，每8个Cell加1，这样数组里的每个元素都由地址，超过一个Rank的，从上一个Rank的最后一个Cell的地址加1作为基地址。这一点很重要， 这也是为什么对于32位计算机来说，能使用的最多容量的内存为4GB。

 

5.2 寻址

当CPU通过地址总线传过来一个物理地址时，内存条是如何寻找到一个Cell的呢。我们首先要看内存条是如何根CPU连接的。

 

DDR3 128*64bit*16的地址线情况如上图所示。

当物理地址从MMU单元出来时，CPU会把需要访问的地址当做一个数据给内存控制器，如果不是集成在CPU内部，那么通过DMI总线（CPU链接北桥的总线）传送给北桥，当然这个是会加上一些标识以方便北桥识别，这个数据是需要访问的内存地址， 当北桥（内存控制器）收到这个地址后，通过集成在控制器中的"逻辑映射"， 内存控制器就知道这个地址对应了了那个rank（p-bank），那个L-bank，L-bank中的那个row地址，L-bank中的那个Column地址，及确定了那个CS#信号有效（选择rank），那组bank#信号有效（选择L-bank），  然后北桥在遵从SPD设定的内存的时序下按照固定的时序发送CS#信号 bank#信号 row address 、column address 特殊状态下还会有DQM信号、BL信号等等。也就是说从内存控制器出来，然后是选择DIMM,根据片选到Rank,再到Bank，然后在Chip的地址解码器里输出行地址和列地址，定位到Cell。其实这也就是说物理地址要包含行地址、列地址，bank地址、rank信息等。根据上面的表，物理地址可能的构成如下：

数据要写入内存的一个cell，或者从内存中的一个cell读取数据，首先要完成对这个cell的寻址。寻址的过程，首先是将需要操作的cell的对应行地址信号和列地址信号输入行/列地址缓冲器，然后先通过行解码器（Row Decoder）选择特定的行地址线路，以激活特定的行地址。每一条行地址线路会与多条列地址线路和cell相连接，为了侦测列地址线路上微弱的激活信号，还需要一个额外的感应放大器（Sense Amplifier）放大这个信号。当行激活之后，列地址缓冲器中的列地址信号通过列解码器（Column Decoder）确定列地址，并被对应的感应放大器通过连接IO线路，这样cell就被激活，并可供读写操作，寻址完成。从行地址激活，到找到列地址这段时间，就是tRCD。

 

六、DRAM物理地址如何映射到内部的row、column和bank

How physical addresses map to rows and banks in DRAM

参考：http://lackingrhoticity.blogspot.com/2015/05/how-physical-addresses-map-to-rows-and-banks.html

Now I'll discuss how these CPUs' memory controllers map physical addresses to locations in DRAM -- specifically, to row, bank and column numbers in DRAM modules. Let's call this the DRAM address mapping. I'll use one test machine as a case study.

Motivation: the rowhammer bug

I am interested in the DRAM address mapping because it is relevant to the "rowhammer" bug.

Rowhammer is a problem with some DRAM modules whereby certain pessimal memory access patterns can cause memory corruption. In these DRAMs, repeatedly activating a row of memory (termed "row hammering") can produce electrical disturbances that produce bit flips in vulnerable cells in adjacent rows of memory.

These repeated row activations can be caused by repeatedly accessing a pair of DRAM locations that are in different rows of the same bank of DRAM. Knowing the DRAM address mapping is useful because it tells us which pairs of addresses satisfy this "same bank, different row" (SBDR) property.

Guessing and checking an address mapping

For my case study, I have a test machine containing DRAM that is vulnerable to the rowhammer problem. Running rowhammer_test on this machine demonstrates bit flips.

I'd like to know what the DRAM address mapping is for this machine, but apparently it isn't publicly documented: This machine has a Sandy Bridge CPU, but Intel don't document the address mapping used by these CPUs' memory controllers.

rowhammer_test does not actually need to identify SBDR address pairs.rowhammer_test just repeatedly tries hammering randomly chosen address pairs. Typically 1/8 or 1/16 of these pairs will be SBDR pairs, because our machine has 8 banks per DIMM (and 16 banks in total). So, while we don't need to know the DRAM address mapping to cause bit flips on this machine, knowing it would help us be more targeted in our testing.

Though the address mapping isn't documented, I found that I can make an educated guess at what the mapping is, based on the DRAM's geometry, and then verify the guess based on the physical addresses that rowhammer_test reports. rowhammer_test can report the physical addresses where bit flips occur ("victims") and the pairs of physical addresses that produce those bit flips ("aggressors"). Since these pairs must be SBDR pairs, we can check a hypothesised address mapping against this empirical data.

Memory geometry

The first step in hypothesising an address mapping for a machine is to check how many DIMMs the machine has and how these DIMMs are organised internally.

I can query information about the DIMMs using the decode-dimms tool on Linux. (In Ubuntu, decode-dimms is in the i2c-tools package.) This tool decodes the DIMMs' SPD (Serial Presence Detect) metadata.

My test machine has 2 * 4GB SO-DIMMs, giving 8GB of memory in total.

decode-dimms reports the following information for both of the DIMMs:

Size                                            4096 MB
Banks x Rows x Columns x Bits                   8 x 15 x 10 x 64
Ranks                                           2

This means that, for each DIMM:

• Each of the DIMM's banks contains 2^15 rows (32768 rows).
• Each row contains 2^10 * 64 bits = 2^16 bits = 2^13 bytes = 8 kbytes.

Each DIMM has 2 ranks and 8 banks. Cross checking the capacity of the DIMM gives us the reported size, as expected:

8 kbytes per row * 32768 rows * 2 ranks * 8 banks = 4096 MB = 4 GB

The DRAM address mapping

On my test machine, it appears that the bits of physical addresses are used as follows:

• Bits 0-5: These are the lower 6 bits of the byte index within a row (i.e. the 6-bit index into a 64-byte cache line).
• Bit 6: This is a 1-bit channel number, which selects between the 2 DIMMs.
• Bits 7-13: These are the upper 7 bits of the index within a row (i.e. the upper bits of the column number).
• Bits 14-16: These are XOR'd with the bottom 3 bits of the row number to give the 3-bit bank number.
• Bit 17: This is a 1-bit rank number, which selects between the 2 ranks of a DIMM (which are typically the two sides of the DIMM's circuit board).
• Bits 18-32: These are the 15-bit row number.
• Bits 33+: These may be set because physical memory starts at physical addresses greater than 0.

Why is the mapping like that?

This mapping fits the results from rowhammer_test (see below), but we can also explain that the address bits are mapped this way to give good performance for typical memory access patterns, such as sequential accesses and strided accesses:

• Channel parallelism: Placing the channel number at bit 6 means that cache lines will alternate between the two channels (i.e. the two DIMMs), which can be accessed in parallel. This means that if we're accessing addresses sequentially, the load will be spread across the two channels.

  As an aside, Ivy Bridge (the successor to Sandy Bridge) apparently makes the mapping of the channel number more complex. An Intel presentation mentions "Channel hashing" and says that this "Allows channel selection to be made based on multiple address bits. Historically, it had been "A[6]". Allows more even distribution of memory accesses across channels."

• Bank thrashing: Generally, column, bank and row numbers are arranged to minimise "bank thrashing" (frequently changing a bank's currently activated row).

  Some background: DRAM modules are organised into banks, which in turn are organised into rows. Each bank has a "currently activated row" whose contents are copied into a row buffer which acts as a cache that can be accessed quickly. Accessing a different row takes longer because that row must be activated first. So, the DRAM address mapping places SBDR pairs as far apart as possible in physical address space.

  Row hammering is a special case of bank thrashing where two particular rows are repeatedly activated (perhaps deliberately).

• Bank parallelism: Banks can be accessed in parallel (though to a lesser degree than channels), so the bank number changes before the row number as the address is increased.

• XOR scheme: XORing the row number's lower bits into the bank number is a trick to avoid bank thrashing when accessing arrays with large strides. For example, in the mapping above, the XORing causes addresses X and X+256k to be placed in different banks instead of being an SBDR pair.

  Bank/row XORing schemes are described in various places, such as:

  • David Tawei Wang's PhD thesis, "Modern DRAM memory systems: Performance analysis and scheduling algorithms", 2005. See section 5.3.5, "Bank Address Aliasing (stride collision)". This thesis has some good background info on DRAM in general.
  • The paper "Reducing DRAM Latencies with an Integrated Memory Hierarchy Design", 2001. See Figure 3.

Checking against rowhammer_test's output

I ran rowhammer_test_ext (the extended version of rowhammer_test) on my test machine for 6 hours, and it found repeatable bit flips at 22 locations. (See the raw data and analysis code.)

The row hammering test generates a set of (A1, A2, V) triples, where:

• V is the victim address where we see the bit flip.
• A1 and A2 are the aggressor addresses that we hammer.
• We sort A1 and A2 so that A1 is closer to V than A2 is. We tentatively assume that the closer address, A1, is the one that actually causes the bit flips (though this wouldn't necessarily be true if the DRAM address mapping were more complicated).

There are three properties we expect to hold for all of these results:

• Row: A1 and V's row numbers should differ by 1 -- i.e. they should be in adjacent rows. (A2 can have any row number.)

  This property makes it easy to work out where the bottom bits of the row number are in the physical address.

  We find this property holds for all but 2 of the results. In those 2 results, the row numbers differ by 3 rather than 1.

• Bank: V, A1 and A2 should have the same bank number. Indeed, we find this property holds for all 22 results. This only holds when applying the row/bank XORing scheme.

• Channel: V, A1 and A2 should have the same channel number. This holds for all the results. It happens that all of our results have channel=0, because rowhammer_test only selects 4k-aligned addresses and so only tests one channel. (Maybe this could be considered a bug.)

Possible further testing

There are two further experiments we could run to check whether the DRAM address mapping evaluates the SBDR property correctly, which I haven't tried yet:

• Timing tests: Accessing SBDR address pairs repeatedly should be slower than accessing non-SBDR pairs repeatedly, because the former cause row activations and the latter don't.

• Exhaustive rowhammer testing: Once we've found an aggressor address, A1, that causes a repeatable bit flip, we can test this against many values of address A2. Hammering (A1, A2) can produce bit flips only if this is an SBDR pair.

Furthermore, taking out one DIMM from our test machine should remove the channel bit from the DRAM address mapping and change the aggressor/victim addresses accordingly. We could check whether this is the case.

 

参考：

https://zhidao.baidu.com/question/583245605.html

https://www.cnblogs.com/felixfang/p/3420462.html

http://blog.csdn.net/windowseight/article/details/8279863

http://bbs.chinaunix.net/thread-2083672-1-1.html

http://blog.csdn.net/erazy0/article/details/6457626#comments

https://www.cnblogs.com/yilang/p/11103061.html

https://blog.csdn.net/fenggewan/article/details/92026057

https://www.cnblogs.com/QQParadise/articles/2430204.html