Check pci information on Linux sysfs

As we know, we can get PCI bus , device and driver info from Linux sysfs.

PCI bus architecture could be presented as below:

CPU -> host bridge(RC)->root PCI bus->slot <-insert-> bridge(PCI to PCIe) -> child PCI bus->slot <-insert-> nvme device

Here "slot" is a kind of PCI device. we know PCI bus can have 32 devices, each devices has 8 functions. This "slot" is the device, the bridge and nvme device is the function of the device.

On my platform, from CPU to bridge are in X86 PCIe Root Port, and device is my NMVe card. The physical slot on the board is the slot which device insert in.

First is PCI bus:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci
devices  drivers_autoprobe  rescan              slots
drivers  drivers_probe      resource_alignment  uevent

we can see slots devices and drivers under PCI bus.

PCI Bus

1.Devices

In devices, there are all devices under PCI buses, maybe not only one PCI root bus:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/devices/
0000:00:00.0  0000:00:1f.0  0000:16:0e.0  0000:17:00.1  0000:64:0c.3
0000:00:04.0  0000:00:1f.2  0000:16:0e.1  0000:64:00.0  0000:64:0c.4
0000:00:04.1  0000:00:1f.3  0000:16:0e.2  0000:64:05.0  0000:64:0c.5
0000:00:04.2  0000:00:1f.4  0000:16:0e.3  0000:64:05.2  0000:64:0c.6
0000:00:04.3  0000:00:1f.6  0000:16:0e.4  0000:64:05.4  0000:64:0c.7
0000:00:04.4  0000:01:00.0  0000:16:0e.5  0000:64:08.0  0000:64:0d.0
0000:00:04.5  0000:02:00.0  0000:16:0e.6  0000:64:09.0  0000:64:0d.1
0000:00:04.6  0000:03:00.0  0000:16:0e.7  0000:64:0a.0  0000:64:0d.2
0000:00:04.7  0000:16:00.0  0000:16:0f.0  0000:64:0a.1  0000:64:0d.3
0000:00:05.0  0000:16:05.0  0000:16:0f.1  0000:64:0a.2  0000:65:00.0
0000:00:05.2  0000:16:05.2  0000:16:1d.0  0000:64:0a.3  0000:b2:03.0
0000:00:05.4  0000:16:05.4  0000:16:1d.1  0000:64:0a.4  0000:b2:05.0
0000:00:08.0  0000:16:08.0  0000:16:1d.2  0000:64:0a.5  0000:b2:05.2
0000:00:08.1  0000:16:08.1  0000:16:1d.3  0000:64:0a.6  0000:b2:05.4
0000:00:08.2  0000:16:08.2  0000:16:1e.0  0000:64:0a.7  0000:b2:12.0
0000:00:14.0  0000:16:08.3  0000:16:1e.1  0000:64:0b.0  0000:b2:12.1
0000:00:14.2  0000:16:08.4  0000:16:1e.2  0000:64:0b.1  0000:b2:12.2
0000:00:16.0  0000:16:08.5  0000:16:1e.3  0000:64:0b.2  0000:b2:15.0
0000:00:17.0  0000:16:08.6  0000:16:1e.4  0000:64:0b.3  0000:b2:16.0
0000:00:1c.0  0000:16:08.7  0000:16:1e.5  0000:64:0c.0  0000:b2:16.4
0000:00:1c.2  0000:16:09.0  0000:16:1e.6  0000:64:0c.1  0000:b2:17.0
0000:00:1c.5  0000:16:09.1  0000:17:00.0  0000:64:0c.2  0000:b3:00.0

the rule of the device name is domain:bus:device.function. Actually they are all soft link to /sys/devices:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/devices/ -l
total 0
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:00.0 -> ../../../devices/pci0000:00/0000:00:00.0
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.0 -> ../../../devices/pci0000:00/0000:00:04.0
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.1 -> ../../../devices/pci0000:00/0000:00:04.1
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.2 -> ../../../devices/pci0000:00/0000:00:04.2
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.3 -> ../../../devices/pci0000:00/0000:00:04.3
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.4 -> ../../../devices/pci0000:00/0000:00:04.4
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.5 -> ../../../devices/pci0000:00/0000:00:04.5
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.6 -> ../../../devices/pci0000:00/0000:00:04.6
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:04.7 -> ../../../devices/pci0000:00/0000:00:04.7
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:05.0 -> ../../../devices/pci0000:00/0000:00:05.0
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:05.2 -> ../../../devices/pci0000:00/0000:00:05.2
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:05.4 -> ../../../devices/pci0000:00/0000:00:05.4
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:08.0 -> ../../../devices/pci0000:00/0000:00:08.0
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:08.1 -> ../../../devices/pci0000:00/0000:00:08.1
lrwxrwxrwx 1 root root 0 2月  15 14:04 0000:00:08.2 -> ../../../devices/pci0000:00/0000:00:08.2

So we can find them in /sys/devices directory:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/devices/
breakpoint   msr         software      uncore_cha_5               uncore_iio_free_running_1  uncore_imc_3  uncore_irp_5    virtual
cpu          pci0000:00  system        uncore_iio_0               uncore_iio_free_running_2  uncore_imc_4  uncore_m2m_0
cstate_core  pci0000:16  tracepoint    uncore_iio_1               uncore_iio_free_running_3  uncore_imc_5  uncore_m2m_1
cstate_pkg   pci0000:64  uncore_cha_0  uncore_iio_2               uncore_iio_free_running_4  uncore_irp_0  uncore_m3upi_0
intel_pt     pci0000:b2  uncore_cha_1  uncore_iio_3               uncore_iio_free_running_5  uncore_irp_1  uncore_m3upi_1
isa          platform    uncore_cha_2  uncore_iio_4               uncore_imc_0               uncore_irp_2  uncore_pcu
kprobe       pnp0        uncore_cha_3  uncore_iio_5               uncore_imc_1               uncore_irp_3  uncore_ubox
LNXSYSTM:00  power       uncore_cha_4  uncore_iio_free_running_0  uncore_imc_2               uncore_irp_4  uprobe

Now here we can see pci0000:00, pci0000:16 , pci0000:64 and pci0000:b2. they are pci host bridge. And they present one PCI root bus. We can see the PCI root bus at:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/devices/pci0000\:00/pci_bus/
0000:00

And we can see all the devices belongs to this PCI root bus:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/devices/pci0000\:00/
0000:00:00.0  0000:00:04.3  0000:00:04.7  0000:00:08.0  0000:00:14.2  0000:00:1c.2  0000:00:1f.3   INT344B:00  uevent
0000:00:04.0  0000:00:04.4  0000:00:05.0  0000:00:08.1  0000:00:16.0  0000:00:1c.5  0000:00:1f.4   pci_bus
0000:00:04.1  0000:00:04.5  0000:00:05.2  0000:00:08.2  0000:00:17.0  0000:00:1f.0  0000:00:1f.6   PNP0003:00
0000:00:04.2  0000:00:04.6  0000:00:05.4  0000:00:14.0  0000:00:1c.0  0000:00:1f.2  firmware_node  power

See, we can find the PCI devices here. PCI host bridge and PCI bus are different structure in the kernel, but they present the same PCI root bus.

2.drivers

When we see the device, we don't know what kind of device it is. Then we can go to the drivers directory to find out it.

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/drivers
 agpgart-intel   ata_generic          ehci-pci       iwlwifi   ohci-pci     serial          virtio-pci
 agpgart-via     ata_piix             imsttfb        mei_me    parport_pc   skx_uncore      xen-platform-pci
 ahci           'Cavium PTP Driver'   ioatdma        nouveau   pata_sis     snd_hda_intel   xhci_hcd
 asiliantfb      e1000e               iosf_mbi_pci   nvme      pcieport     uhci_hcd

We can see here lists all the supported drivers on PCI buses. let's take a look of nvme.

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/drivers/nvme/
0000:65:00.0  0000:b3:00.0  bind  module  new_id  remove_id  uevent  unbind

Here are two NVMe devices. One is on PCI root bus 0000:65, another is on PCI root bus 0000:b3.

Let's take a look of pcieport:

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/drivers/pcieport/
0000:00:1c.0  0000:00:1c.2  0000:00:1c.5  0000:16:00.0  0000:64:00.0  0000:b2:03.0  bind  new_id  remove_id  uevent  unbind

Here are 6 pcieports.

3. slots

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci/slots/
0  2  6  6-1  8191  8191-1  8191-2  8191-3  8191-4  8191-5  8191-6  8191-7  8191-9

here slot present pci_slot{} . in ACPI DSDT tree, it's on PCI bus.

[    0.234039] alexpci:check_slot,Checking slot on path: \_SB_.PC02.BR2A
[    0.234050] alexpci:register_slot,create ACPI slots: bus is 64, slot is 6
[    0.234051] alexpci:check_slot,Checking slot on path: \_SB_.PC02.BR2B
[    0.234061] alexpci:register_slot,create ACPI slots: bus is 64, slot is 8191-2
[    0.234062] alexpci:check_slot,Checking slot on path: \_SB_.PC02.BR2C
[    0.234072] alexpci:register_slot,create ACPI slots: bus is 64, slot is 8191-4
[    0.234072] alexpci:check_slot,Checking slot on path: \_SB_.PC02.BR2D
[    0.234082] alexpci:register_slot,create ACPI slots: bus is 64, slot is 8191-5

another blog will introduce ACPI DSDT tree and PCI bus/slots.

Here slot 6 is the slot on root pci_bus{} 0000:64, bridge 0000:64:00.0 inserts in the slot. And slot 6-1 is the slot on child pci_bus{} 0000:65, nvme device 0000:65:00.0 inserts in this slot. 

we can check slot's address to know which bus it belongs to:

cdsvlab@cdsvlab-MS-7A95:~$ cat /sys/bus/pci/slots/6-1/address 
0000:65:00
cdsvlab@cdsvlab-MS-7A95:~$ cat /sys/bus/pci/slots/6/address 
0000:64:00

Here "slot" is a kind of PCI device. we know PCI bus can have 32 devices, each devices has 8 functions. This "slot" is the device, the bridge 0000:64:00.0 and nvme device 0000:65:00.0 is the function of the device.

Next is PCIe bus.

PCIe Bus

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci_express/
devices  drivers  drivers_autoprobe  drivers_probe  uevent

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci_express/devices/
0000:00:1c.0:pcie001  0000:00:1c.5:pcie001  0000:64:00.0:pcie004
0000:00:1c.2:pcie001  0000:64:00.0:pcie001  0000:b2:03.0:pcie001

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci_express/drivers
aer  dpc  pciehp  pcie_pme

cdsvlab@cdsvlab-MS-7A95:~$ ls /sys/bus/pci_express/drivers/pciehp/
0000:64:00.0:pcie004  bind  uevent  unbind

0000:64:00.0:pcie001 is pme feature of bridge 0000:64:00.0

0000:64:00.0:pcie004 is hotplug feature of bridge 0000:64:00.0

Now we see all the PCI and PCIe nodes in sysfs.