PCIe的＜配置/IO/MEM空间＞的区别

IO空间跟内存空间区别

x86架构采用独立编址将内存操作与外设IO操作分开了才有了内存空间和IO空间的区分。
x86平台cpu内部对内存和外设寄存器访问的指令也是不同的。arm等其他平台都采用统一编址，不区分内存和外设的访问
IO空间：访问外部设备寄存器的地址区域
内存空间：访问内存的地址空间，32位平台为4G

为什么需要内存空间

常见的设备都只提供寄存器供cpu访问，对于低速外设这样的模式是足够的。但是对于需要大量、高速数据交互的外设就需要引入外设内存空间了。
在网卡、显卡这样的pci高速外设中不仅有寄存器还有了一块内存

IO空间跟配置空间区别

IO空间就和i2c设备的寄存器空间一样，用来获取外设状态、配置外设。
配置空间是一段特殊的IO空间，它的作用是为外设内存空间、IO空间分配物理地址基地址，即配置BAR（Base Address Registers）

设备树示例

pcie0: pcie@0xd4288000{
				compatible = "asr,falcon-pcie";
				device_type = "pci";
				#address-cells = <3>;
				#size-cells = <2>;
				bus-range = <0x00 0xff>;
				linux,pci-domain = <0>;
				reg = <0xd4210000 0x800>,	/* Falcon PCIe PHY registers */
					  <0xd4288000 0x1000>;	/* Falcon PCIe config space */
				reg-names = "pciephy", "pciectrl";
				phys = <&pcieport0 0>;
				phy-names = "pcie-phy";
				ranges =  <0x81000000 0 0 0xE0010000 0 0x00010000			/* downstream I/O */
						0x82000000 0 0xE0020000 0xE0020000 0 0x01000000>;	/* memory */
				lpm-qos = <PM_QOS_CPUIDLE_BLOCK_AXI>;
				num-lanes = <1>;
				interrupts = <18>;
				#interrupt-cells = <1>;
				interrupt-parent = <&intc>;
				interrupt-map-mask = <0 0 0 0>;
				interrupt-map = <0 0 0 0 &intc 18>;
				clocks = <&soc_clocks ASR1803_CLK_PCIE0>;
				status = "disabled";
			};

如上图设备树中，pcie两组地址转换关系，每一组的几个数字分别定义了属性（32bit），pci地址空间（64bit），cpu地址（32bit/64bit），长度（64bit）
属性的含义：npt000ss bbbbbbbb dddddfff rrrrrrrr （https://elinux.org/Main_Page）

n: relocatable region flag (doesn't play a role here)
p: prefetchable (cacheable) region flag
t: aliased address flag (doesn't play a role here)
ss: space code
00: configuration space
01: I/O space
10: 32 bit memory space
11: 64 bit memory space
bbbbbbbb: The PCI bus number. PCI may be structured hierarchically. So we may have PCI/PCI bridges which will define sub busses.
ddddd: The device number, typically associated with IDSEL signal connections.
fff: The function number. Used for multifunction PCI devices.
rrrrrrrr: Register number; used for configuration cycles.	

IO/MEM空间在系统中的呈现

IO空间
/ # cat /proc/ioports
00000000-0000ffff : pcie@0xd4288000
MEM空间
/ # cat /proc/iomem
e0020000-e101ffff : pcie@0xd4288000
  e0020000-e0023fff : 0000:00:00.0
  e0100000-e01fffff : PCI Bus 0000:01
    e0100000-e013ffff : 0000:01:00.0

MEM空间在开机日志中的呈现

首先是root bus中io和mem的基地址，然后是给bridge(type 01)分配的mem地址，然后写到bridge的bar0; 然后是给device(type 00)分配的mem地址，然后写到device的bar0; 就可以开始数据交互了

[    3.117858] pcie-falcon d4210000.pcie: host bridge /soc/axi@d4200000/pcie@0xd4288000 ranges:
[    3.126373] pcie-falcon d4210000.pcie:    IO 0xe0010000..0xe001ffff -> 0x00000000
[    3.133880] pcie-falcon d4210000.pcie:   MEM 0xe0020000..0xe101ffff -> 0xe0020000
[    3.442779] ASR Falcon PCIe Host 1x link negotiated (gen 2), maxpayload 128  maxreqsize 128
[    3.451324] pcie-falcon d4210000.pcie: PCI host bridge to bus 0000:00
[    3.457794] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.463256] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.469451] pci_bus 0000:00: root bus resource [mem 0xe0020000-0xe101ffff]
[    3.476348] pci 0000:00:00.0: [1e5d:3001] type 01 class 0x078000
[    3.482360] pci 0000:00:00.0: reg 0x10: [mem 0xf0000000-0xf0003fff 64bit pref]
[    3.489654] pci 0000:00:00.0: supports D1 D2
[    3.493927] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    3.502044] PCI: bus0: Fast back to back transfers disabled
[    3.507751] pci 0000:01:00.0: [59e7:0001] type 00 class 0x028000
[    3.513793] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    3.520141] pci 0000:01:00.0: supports D1 D2
[    3.524414] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[    3.531616] PCI: bus1: Fast back to back transfers disabled
[    3.537231] pci 0000:00:00.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
[    3.544006] pci 0000:00:00.0: BAR 0: assigned [mem 0xe0020000-0xe0023fff 64bit pref]
[    3.551757] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0100000-0xe013ffff]
[    3.558563] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.563507] pci 0000:00:00.0:   bridge window [mem 0xe0100000-0xe01fffff]

[   18.296417] PCI driver: driver init start.
[   18.296539] pci probe begin!
[   18.296569] cion_pci 0000:01:00.0: enabling device (0140 -> 0142)
[   18.309051] bar0: 0xe0100000, len = 0x40000
[   18.309051] bar1: 0x0, len = 0x0
[   18.309051] bar2: 0x0, len = 0x0
[   18.309082] bar3: 0x0, len = 0x0
[   18.309082] bar4: 0x0, len = 0x0
[   18.309082] bar5: 0x0, len = 0x0
[   18.309082] BAR 0: set to [mem 0xe0100000, len = 0x40000]
[   18.309112] BAR 0: ioremap addr = 0xc8d40000
[   18.309112] pci_resource_start[0xe0100000],pci_resource_en[0xe013ffff]
[   18.309173] CION: pci_dev->irq      : 34.
[   18.309204] PCI driver: probe succ!

配置空间访问

根据bus,devfn等信息，访问到对应设备的配置空间

int cfg_read(void __iomem *addr, int where, int size, u32 *val)
{
	*val = readl(addr);
	if (size == 1)
		*val = (*val >> (8 * (where & 3))) & 0xff;
	else if (size == 2)
		*val = (*val >> (8 * (where & 3))) & 0xffff;
	else if (size != 4)
		return PCIBIOS_BAD_REGISTER_NUMBER;

	return PCIBIOS_SUCCESSFUL;
}

static int falcon_pcie_hw_rd_cfg(struct falcon_pcie_port *port, u32 bus, u32 devfn,
			      int where, int size, u32 *val)
{
	u32 value;
	int ret;

	mutex_lock(&port->lock);
	if (PCI_FUNC(devfn) == 0) {
		value = readl(port->base + PCIE_CFGNUM);
		if (bus == 1) {
			writel((value|(0x1<<8)), port->base + PCIE_CFGNUM);	
			ret = cfg_read(port->base + FALCON_PCIE_CONFIG_OFFSET
					+ (where & ~0x3), where, size, val);
		} else if (bus == 0) {
			writel((value&(~(0x1<<8))), port->base + PCIE_CFGNUM);
			ret = cfg_read(port->base + FALCON_PCIE_CONFIG_OFFSET
						+ (where & ~0x3), where, size, val);
		}
	}

	mutex_unlock(&port->lock);
	return PCIBIOS_SUCCESSFUL;
}