Linux Topics (4.3) - opROM Bar attribute confuse of PCIe device

懒懒兔

已于 2024-04-23 18:48:40 修改

阅读量833

点赞数 18

分类专栏： Linux Topics 文章标签： linux 服务器

于 2024-03-08 11:06:34 首次发布

本文链接：https://blog.csdn.net/cjayzdyk/article/details/136553902

版权

Linux Topics 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

3. BAR Initialization in Linux

3.1 Initialization Flow

3.2 pci_read_bases(）

1. Confuse

原因是查看linux kernel boot log的时候，发现一个SSD 设备的opROM bar 属性被标志为pref,

pci 0001:01:00.0: [144d:a80a] type 00 class 0x010802
pci 0001:01:00.0: reg 0x10: [mem 0x740004110000-0x740004113fff 64bit]
pci 0001:01:00.0: reg 0x30: [mem 0x740004100000-0x74000410ffff pref]

但是我记得该设备的BAR不是prefetchable，到了OS 下， lspci 看了下，也没有标志prefetchable

#lspci -vvs 0001:01:00.0

0001:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
Region 0: Memory at 740004010000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at 740004000000 [disabled] [size=64K]

Dump 一下配置空间，也证明了 opROM BAR prefetch bit 没有置位：

00: 4d 14 0a a8 06 04 10 00 00 02 08 01 10 00 00 00
10: 04 00 01 04 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 4d 14 12 a8
30: 00 00 10 04 40 00 00 00 00 00 00 00 ff 01 00 00

但是查看了 /sys/devices 下面的设备资源信息，却是跟boot log 一致，标识opROM BAR为prefetchable。

#define IORESOURCE_PREFETCH 0x00002000 /* No side effects */
cat /sys/devices/pci0001\:00/0001\:00\:01.0/0001\:01\:00.0/resource
0x0000740004010000 0x0000740004013fff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000740004000000 0x000074000400ffff 0x0000000000046200 //ROM BAR, Pref Bit= 0x2000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000

那opROM BAR 如何初始化的呢？为何要这样标识呢？首先要查查代码

2. PCIe Config Space

2.1 BAR Introduction

Config space 的BARs 如果下图

Type0 PCIe device的 config space, 共有 6 个BAR(0x10 – 0x24)，一个ROM BAR(0x30).
Type1 PCIe bridge 的 config space, 共有 2个BAR(0x10 – 0x14)，一个ROM BAR(0x38).

ROM Bar 主要用来在BIOS 初始化的时候，先在Pcie enumeration的时候分配合适的地址，

用来关联该device的 opROM driver(UEFI 时候，包的就是UEFI driver）

对于BIOS 来说，

BIOS 把某个PCIE device的 opROM driver 直接包在代码里面，可以override 设备自带的较久的opROM driver, 这种设备是设计的时候已经设计在主板上了。
对于外置add-in card, 一般都是BIOS 初始化的是时候，获取card 本身自带的opROM driver 来初始化。

2.2 BAR Attributes

重点看一下 bit 3, 用来标识是否 Prefetchable.

3. BAR Initialization in Linux

3.1 Initialization Flow

acpi_pci_root_add()

pci_acpi_scan_root()

root_ops->release_info = pci_acpi_generic_release_info;

root_ops->prepare_resources = pci_acpi_root_prepare_resources;

acpi_pci_root_create()

|-->ops->prepare_resources(info);

| |-->pci_acpi_root_prepare_resources; //Get resource from ACPI(_CRS..)

|--> pci_acpi_root_add_resources(info);

|-->pci_create_root_bus()

| |-->pci_register_host_bridge()

|-->pci_scan_child_bus(bus);

| pci_scan_child_bus_extend(bus, 0);

| pci_scan_slot(bus, devfn);

| pci_scan_single_device()

| |-->pci_scan_device(bus, devfn); //扫描slot，并创建pci dev

| | |-->pci_setup_device()

| | |-->pci_read_bases()

3.2 pci_read_bases(）

static void pci_read_bases(struct pci_dev *dev, unsigned int howmany, int rom)

{

unsigned int pos, reg;

if (dev->non_compliant_bars) return;

/* Per PCIe r4.0, sec 9.3.4.1.11, the VF BARs are all RO Zero */

if (dev->is_virtfn) return;

for (pos = 0; pos < howmany; pos++) {

struct resource *res = &dev->resource[pos];

reg = PCI_BASE_ADDRESS_0 + (pos << 2);

pos += __pci_read_base(dev, pci_bar_unknown, res, reg);

}

if (rom) {

struct resource *res = &dev->resource[PCI_ROM_RESOURCE];

dev->rom_base_reg = rom;

res->flags = IORESOURCE_MEM | IORESOURCE_PREFETCH | //why opROM BAR masked as prefetch?

IORESOURCE_READONLY | IORESOURCE_SIZEALIGN;

__pci_read_base(dev, pci_bar_mem32, res, rom);

}

代码里面已经很明显了，如果是opROM bar, 那么就要标志为 IORESOURCE_PREFETCH,

虽然 prefetchable memory(MMIO)有它的好处，

Prefetchable memory is memory which the CPU can request in advance as an optimization, before actual code operates on it. This happens because the CPU guesses it may be needed in the future or because the code contains explicit prefetch instructions. Then it prefetches those memory locations while processing some other instructions or doing some other work.

但是一般来说，都是根据BAR 自己的属性定义来配置才对。

查下了 Linux kernel git repo, 在 Linus 开始 2.6 版本的时候，就已经是这样了，虽然之后由于side-effect, 取消了该设定，但是后来又revert了。

至于最初肯定是出于性能目的这样设置，但是暂时找不到最初的记录，真是好奇害死猫，留个TBD吧。

commit bb44609361fe87c5e136c2b8dfde59bcbdbabf61

Author: Gary Hade <garyhade@us.ibm.com>

Date: Tue Dec 11 17:09:13 2007 -0800

PCI: Restore PCI expansion ROM P2P prefetch window creation

Restore PCI expansion ROM P2P prefetch window creation.

This patch reverts previous "Avoid creating P2P prefetch

window for expansion ROMs" change due to regressions that

were spotted on some systems.

Signed-off-by: Gary Hade <garyhade@us.ibm.com>