NXP(Freescale) QorIQ T2080 PCIe使用

作者

QQ群：852283276
微信：arm80x86
微信公众号：青儿创客基地
B站：主页 https://space.bilibili.com/208826118

参考

PCI device (64bit memory problem)
PCIe link doesn’t come up with XIO2001 PCI bridge on iMX6Q custom board
p2020 PCIe linux kernel module reads 0xffffffff on BARs
LS1021A PCIe Link speed is Gen1 (2.5GB) not Gen2 (5GB)
PCI-E Switch not normal working on LS1012A
LS2085A: Devices behind PCIe Switch not working
T2080 : Question for PCIE errors
[转载]老男孩读PCIe之六：配置和地址空间

RCW

配置链路速度2.5/5/8Gbps，决定PCIe Gen1/2/3，使用QCVS生成即可，uboot和内核默认配置就可以使用。

linux

开机打印，

Found FSL PCI host bridge at 0x0000000ffe270000. Firmware bus number: 0->15
PCI host bridge /pcie@ffe270000  ranges:
 MEM 0x0000000c40000000..0x0000000c4fffffff -> 0x00000000e0000000 
  IO 0x0000000ff8030000..0x0000000ff803ffff -> 0x0000000000000000
/pcie@ffe270000: PCICSRBAR @ 0xff000007
setup_pci_atmu: end of DRAM 100000000
/pcie@ffe270000: Setup 64-bit PCI DMA window
/pcie@ffe270000: WARNING: Outbound window cfg leaves gaps in memory map. Adjusting the memory map could reduce unnecessary bounce buffering.
/pcie@ffe270000: DMA window size is 0xe0000000
EDAC PCI0: Giving out device to module MPC85xx_edac controller mpc85xx_pci_err: DEV ffe270000.pcie (INTERRUPT)
MPC85xx_edac acquired irq 23 for PCI Err
MPC85xx_edac PCI err registered
platform ffe270000.pcie:pcie@0: Invalid size 0xfffff9 for dma-range
software IO TLB [mem 0xf2e00000-0xf6e00000] (64MB) mapped at [c0000000f2e00000-c0000000f6dfffff]
PCI: Probing PCI hardware
fsl-pci ffe270000.pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
pci_bus 0000:00: root bus resource [mem 0xc40000000-0xc4fffffff] (bus address [0xe0000000-0xefffffff])
pci_bus 0000:00: root bus resource [bus 00-0f]
pci 0000:00:00.0: [Firmware Bug]: reg 0x10: invalid BAR (can't size)
pci 0000:00:00.0: PCI bridge to [bus 01-ff]
pci 0000:01:00.0: PCI bridge to [bus 02-ff]
pci 0000:02:01.0: PCI bridge to [bus 03-ff]
pci 0000:02:02.0: PCI bridge to [bus 04-ff]
pci 0000:02:03.0: PCI bridge to [bus 05-ff]
pci 0000:02:04.0: PCI bridge to [bus 06-ff]
pci 0000:02:05.0: PCI bridge to [bus 07-ff]
pci 0000:02:08.0: PCI bridge to [bus 08-ff]
pci 0000:02:09.0: PCI bridge to [bus 09-ff]
pci 0000:02:0a.0: PCI bridge to [bus 0a-ff]
pci 0000:02:0b.0: PCI bridge to [bus 0b-ff]
pci 0000:02:0c.0: PCI bridge to [bus 0c-ff]
pci 0000:02:0d.0: PCI bridge to [bus 0d-ff]
pci 0000:02:0e.0: PCI bridge to [bus 0e-ff]
pci 0000:02:0f.0: PCI bridge to [bus 0f-ff]
PCI: Cannot allocate resource region 6 of device 0000:00:00.0, will remap
pci 0000:00:00.0: BAR 6: no space for [mem size 0x01000000 pref]
pci 0000:00:00.0: BAR 6: failed to assign [mem size 0x01000000 pref]
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc41100000-0xc411fffff]
pci 0000:02:04.0: PCI bridge to [bus 06]
pci 0000:02:04.0:   bridge window [mem 0xc41200000-0xc43ffffff]
pci 0000:02:05.0: PCI bridge to [bus 07]
pci 0000:02:05.0:   bridge window [mem 0xc44000000-0xc440fffff]
pci 0000:02:08.0: PCI bridge to [bus 08]
pci 0000:02:08.0:   bridge window [mem 0xc44100000-0xc441fffff]
pci 0000:02:09.0: PCI bridge to [bus 09]
pci 0000:02:09.0:   bridge window [mem 0xc44200000-0xc442fffff]
pci 0000:02:0a.0: PCI bridge to [bus 0a]
pci 0000:02:0a.0:   bridge window [mem 0xc44300000-0xc443fffff]
pci 0000:02:0b.0: PCI bridge to [bus 0b]
pci 0000:02:0c.0: PCI bridge to [bus 0c]
pci 0000:02:0c.0:   bridge window [mem 0xc44400000-0xc444fffff]
pci 0000:02:0d.0: PCI bridge to [bus 0d]
pci 0000:02:0d.0:   bridge window [mem 0xc44500000-0xc445fffff]
pci 0000:02:0e.0: PCI bridge to [bus 0e]
pci 0000:02:0e.0:   bridge window [mem 0xc44600000-0xc446fffff]
pci 0000:02:0f.0: PCI bridge to [bus 0f]
pci 0000:02:0f.0:   bridge window [mem 0xc44700000-0xc447fffff]
pci 0000:01:00.0: PCI bridge to [bus 02-0f]
pci 0000:01:00.0:   bridge window [mem 0xc41100000-0xc447fffff]
pci 0000:00:00.0: PCI bridge to [bus 01-0f]
pci 0000:00:00.0:   bridge window [io  0x10000-0x1ffff]
pci 0000:00:00.0:   bridge window [mem 0xc40000000-0xc4fffffff]

扫描设备，

root@t2080rdb:~# lspci -vv
00:00.0 PCI bridge: Freescale Semiconductor Inc T2080 without security (rev 11) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at 0000
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-0000ffff
        Memory behind bridge: e0000000-efffffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
                Address: 00000000ff044148  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [148 v1] #19
        Kernel driver in use: pcieport
root@t2080rdb:~# lspci -tv
-[0000:00]---00.0-[01-0f]----00.0-[02-0f]--+-01.0-[03]--
                                           +-02.0-[04]--
                                           +-03.0-[05]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-04.0-[06]----00.0  Device 0731:8000
                                           +-05.0-[07]----00.0  Xilinx Corporation Device 7028
                                           +-08.0-[08]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-09.0-[09]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-0a.0-[0a]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-0b.0-[0b]--
                                           +-0c.0-[0c]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-0d.0-[0d]----00.0  Samsung Electronics Co Ltd Device a808
                                           +-0e.0-[0e]----00.0  Samsung Electronics Co Ltd Device a808
                                           \-0f.0-[0f]----00.0  Samsung Electronics Co Ltd Device a808
root@t2080rdb:~# cat /proc/iomem
00000000-ffffffff : System RAM
c20000000-c2fffffff : srio win
c30000000-c3fffffff : srio win
c40000000-c4fffffff : /pcie@ffe270000
  c40000000-c4fffffff : PCI Bus 0000:01
    c41000000-c4103ffff : 0000:01:00.0
    c41100000-c447fffff : PCI Bus 0000:02
      c41100000-c411fffff : PCI Bus 0000:05
        c41100000-c41103fff : 0000:05:00.0
          c41100000-c41103fff : nvme
      c41200000-c43ffffff : PCI Bus 0000:06
        c42000000-c43ffffff : 0000:06:00.0
          c42000000-c43ffffff : pcie_ep
      c44000000-c440fffff : PCI Bus 0000:07
        c44000000-c4400ffff : 0000:07:00.0
          c44000000-c4400ffff : xdma
      c44100000-c441fffff : PCI Bus 0000:08
        c44100000-c44103fff : 0000:08:00.0
          c44100000-c44103fff : nvme
      c44200000-c442fffff : PCI Bus 0000:09
        c44200000-c44203fff : 0000:09:00.0
          c44200000-c44203fff : nvme
      c44300000-c443fffff : PCI Bus 0000:0a
        c44300000-c44303fff : 0000:0a:00.0
          c44300000-c44303fff : nvme
      c44400000-c444fffff : PCI Bus 0000:0c
        c44400000-c44403fff : 0000:0c:00.0
          c44400000-c44403fff : nvme
      c44500000-c445fffff : PCI Bus 0000:0d
        c44500000-c44503fff : 0000:0d:00.0
          c44500000-c44503fff : nvme
      c44600000-c446fffff : PCI Bus 0000:0e
        c44600000-c44603fff : 0000:0e:00.0
          c44600000-c44603fff : nvme
      c44700000-c447fffff : PCI Bus 0000:0f
        c44700000-c44703fff : 0000:0f:00.0
          c44700000-c44703fff : nvme
fd0000000-fd7ffffff : fd0000000.nor
fe0000000-fe7ffffff : fe0000000.nor
fe8000000-fefffffff : fe0000000.nor
ffe008000-ffe008fff : mpc85xx_mc_err
ffe0c0000-ffe0d0fff : srio regs
ffe100100-ffe10017f : dma
ffe100180-ffe1001ff : dma
ffe100200-ffe10027f : dma
ffe100280-ffe1002ff : dma
ffe100400-ffe10047f : dma
ffe100480-ffe1004ff : dma
ffe100500-ffe10057f : dma
ffe100580-ffe1005ff : dma
ffe101100-ffe10117f : dma
ffe101180-ffe1011ff : dma
ffe101200-ffe10127f : dma
ffe101280-ffe1012ff : dma
ffe101400-ffe10147f : dma
ffe101480-ffe1014ff : dma
ffe101500-ffe10157f : dma
ffe101580-ffe1015ff : dma
ffe110000-ffe110fff : /soc@ffe000000/spi@110000
ffe114000-ffe114fff : mmc0
ffe11c500-ffe11c507 : serial
ffe11c600-ffe11c607 : serial
ffe11d500-ffe11d507 : serial
ffe11d600-ffe11d607 : serial
ffe1e0000-ffe1e07ff : rman-inbound-block0
ffe1e0b00-ffe1e0fff : rman-uio
ffe1e1000-ffe1e17ff : rman-inbound-block1
ffe1e2000-ffe1e27ff : rman-inbound-block2
ffe1e3000-ffe1e37ff : rman-inbound-block3
ffe270e00-ffe27ffff : mpc85xx_pci_err
ffe400000-ffe4fffff : fman
  ffe400000-ffe45ffff : fman-muram
  ffe482000-ffe482fff : fman-port-hc
  ffe483000-ffe483fff : fman-port-hc
  ffe484000-ffe484fff : fman-port-hc
  ffe485000-ffe485fff : fman-port-hc
  ffe486000-ffe486fff : fman-port-hc
  ffe487000-ffe487fff : fman-port-hc
  ffe488000-ffe488fff : fman-port-hc
  ffe489000-ffe489fff : fman-port-hc
  ffe48a000-ffe48afff : fman-port-hc
  ffe48b000-ffe48bfff : fman-port-hc
  ffe48c000-ffe48cfff : fman-port-hc
  ffe48d000-ffe48dfff : fman-port-hc
  ffe490000-ffe490fff : fman-port-hc
  ffe491000-ffe491fff : fman-port-hc
  ffe4a8000-ffe4a8fff : fman-port-hc
  ffe4a9000-ffe4a9fff : fman-port-hc
  ffe4aa000-ffe4aafff : fman-port-hc
  ffe4ab000-ffe4abfff : fman-port-hc
  ffe4ac000-ffe4acfff : fman-port-hc
  ffe4ad000-ffe4adfff : fman-port-hc
  ffe4b0000-ffe4b0fff : fman-port-hc
  ffe4b1000-ffe4b1fff : fman-port-hc
  ffe4dc000-ffe4dcfff : fman-vsp
  ffe4e0000-ffe4e0fff : mac
  ffe4e2000-ffe4e2fff : mac
  ffe4e4000-ffe4e4fff : mac
  ffe4e6000-ffe4e6fff : mac
  ffe4f0000-ffe4f0fff : mac
  ffe4f2000-ffe4f2fff : mac
  ffe4fe000-ffe4fefff : fman-rtc
root@t2080rdb:~# lspci -x -s 08:00.0
08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a804
00: 4d 14 04 a8 06 04 10 00 00 02 08 01 08 00 00 00
10: 04 00 10 e4 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 4d 14 01 a8
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
root@t2080rdb:~# lspci -x -s 06:00.0 
06:00.0 Memory controller: Device 0731:8000
00: 31 07 00 80 06 04 10 00 00 00 80 05 08 00 00 00
10: 04 00 00 e2 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 31 07 01 00
30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 00 00

关闭D3状态切换，

root@t2080rdb:~# echo 1 > /sys/module/vfio_pci/parameters/disable_idle_d3

解绑驱动，

root@t2080rdb:~# echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

绑定vfio，

root@t2080rdb:~# echo 0001:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind

ATMU

PCIe Controller寄存器地址，

• Outbound ATMU负责将本地地址空间翻译到外设地址空间
• Inbound ATMU负责把外设地址空间翻译到本地地址空间

对于Bus Master设备，所有的传输都是Inbound？

Outbound

• PF功能：a = 0，1
• ATMU：b = 0，1，2，3，4
• ATMU：c = -----1，2，3，4

寄存器	位置	作用
PFa_PEXOTARb	C00h + (a × 2000h) + (b × 20h)	翻译地址[43:12]
PFa_PEXOTEARb	C04h + (a × 2000h) + (b × 20h)	翻译地址[63:44]
PFa_PEXOWARb	C10h + (a × 2000h) + (b × 20h)	属性
PFa_PEXOWBARc	C28h + (a × 2000h) + (c × 20h)	窗基地址[27:00]

看下PF0寄存器，PF1没用，是全0，

root@t2080rdb:~# memtool md -l 0xffe270c00+0x100
ffe270c00: 00000000 00000000 00000000 00000000                ................
ffe270c10: 80044027 00000000 00000000 00000000                ..@'............
ffe270c20: 000e0000 00000000 00c40000 00000000                ................
ffe270c30: 8004401b 00000000 00000000 00000000                ..@.............
ffe270c40: 00000000 00000000 00ff8030 00000000                ...........0....
ffe270c50: 8008800f 00000000 00000000 00000000                ................
ffe270c60: 00000000 00000000 00000000 00000000                ................
ffe270c70: 00000000 00000000 00000000 00000000                ................
ffe270c80: 00000000 00000000 00000000 00000000                ................
ffe270c90: 00000000 00000000 00000000 00000000                ................
ffe270ca0: 00000000 00000000 00000000 00000000                ................
ffe270cb0: 00000000 00000000 00000000 00000000                ................
ffe270cc0: 00000000 00000000 00000000 00000000                ................
ffe270cd0: 00000000 00000000 00000000 00000000                ................
ffe270ce0: 00000000 00000000 00000000 00000000                ................
ffe270cf0: 20f44017 00000000 00000000 00000000                 .@.............

b	PFa_PEXOTARb	PFa_PEXOTEARb	OTAb	PFa_PEXOWARb	PFa_PEXOWBAR1
0	0x00000000	0x00000000	0x00000000	0x80044027(Memory read/Memory write/1T win)	0x00000000
1	0x000e0000	0x00000000	0xe0000000	0x8004401b(Memory read/Memory write/256M win)	0x00c40000
2	0x00000000	0x00000000	0x00000000	0x8008800f(IO read/IO Write/64K win)	0x00ff8030

Inbound

• PF功能：a = 0，1
• ATMU：b = 0，1，2，3
• ATMU：c = -----1，2，3
• ATMU：d = ----------2，3

寄存器	位置	作用
PFa_PEXITARb	DE0h + (a × 2000h) - (b × 20h)	翻译地址[27:00]
PFa_PEXIWBARc	DE8h + (a × 2000h) - (c × 20h)	窗基地址[43:12]
- PFa_PEXIWBAR0	In RC mode, PEXIWBAR0 is located in the PCI Express type 1 configuration header space	窗基地址[63:00]
- PFa_PEXIWBAR1	PEXIWBAR1 only supports 32-bit PCI Express address space	窗基地址[31:12]
PFa_PEXIWBEARd	DECh + (a × 2000h) - (d × 20h)	窗基地址[63:44]
PFa_PEXIWARb	DF0h + (a × 2000h) - (b × 20h)	属性

看下PF0寄存器，PF1没用，是全0，

root@t2080rdb:~# memtool md -l 0xffe270d80+0x100   
ffe270d80: 00000000 00000000 00000000 00000000                ................
ffe270d90: 00000000 00000000 00000000 00000000                ................
ffe270da0: 00000000 00000000 10000000 00000000                ................
ffe270db0: a0f5501f 00000000 00000000 00000000                ..P.............
ffe270dc0: 00000000 00000000 00000000 00000000                ................
ffe270dd0: a0f5501f 00000000 00000000 00000000                ..P.............
ffe270de0: 00ffe000 00000000 00000000 00000000                ................
ffe270df0: 80e44017 00000000 00000000 00000000                ..@.............
ffe270e00: 00000000 00000000 00fdfff6 00000000                ................
ffe270e10: 00020000 00000000 00000000 00000000                ................
ffe270e20: 80000001 00000000 00000800 00000000                ................
ffe270e30: 00000000 00000000 00000000 00000000                ................
ffe270e40: 00000000 00000000 00000000 00000000                ................
ffe270e50: 00000000 00000000 00000000 00000000                ................
ffe270e60: 00000000 00000000 00000000 00000000                ................
ffe270e70: 00000000 00000000 00000000 00000000                ................

关于PEXIWBAR0，

这个0xff000007的0x7有问题，应该是100b，

root@t2080rdb:~# lspci -x -s 00:00.0
00:00.0 PCI bridge: Freescale Semiconductor Inc T2080 without security (rev 11)
00: 57 19 31 08 07 05 10 00 11 00 20 0b 08 00 01 00
10: 07 00 00 ff 00 00 00 00 00 01 0f 00 00 f0 00 20
20: 00 e0 f0 ef f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 e0 ff 01 00 00

b	PFa_PEXITARb	PFa_PEXIWBARb	PFa_PEXIWBEARb	IWBARb	PFa_PEXIWARb
0	0x00ffe000	-	-	0xff000007	0x80e44017(CCSRBAR/16M win)
1	0x00000000	0x00000000	0x00000000	0x00000000	0xa0f5501f(Read, snoop local processor/Write, snoop local processor/4G win)
2	0x00000000	0x10000000	0x00000000	0x100_00000000	0xa0f5501f(Read, snoop local processor/Write, snoop local processor/4G win)

Xilinx AXI-CDMA驱动

采用Xilinx AXI-CDMA IP做了一个简单的测试工程，CDMA在DDR之间的搬运带宽，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -s 0x1100000000 -d 0x1000000000 -m 0xffdc00000 -l 0x400000 -e 64 -p 1000  
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0x1100000000 to 0x1000000000
  cmd type: h2c
  poll time: 1000
  test cycle: 64
  xaxicdma type: mem ipaddr 0xffdc00000
************xaxicdmaRegDump begin*************
* control:                        0x00000000 *
* status:                         0x00000002 *
* current descriptor pointer:     0x00000000 *
* current descriptor pointer msb: 0x00000000 *
* tail descriptor pointer:        0x00000000 *
* tail descriptor pointer msb:    0x00000000 *
* source address:                 0x00000000 *
* source address msb:             0x00000011 *
* destination address:            0x00000000 *
* destination address msb:        0x00000010 *
* bytes to transfer:              0x00400000 *
************xaxicdmaRegDump end***************
speed: 3764.71MB/s, cost times: 68ms
root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -s 0x1080000000 -d 0x1000000000 -m 0xffdc00000 -l 0x400000 -e 64 -p 1000  
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0x1080000000 to 0x1000000000
  cmd type: h2c
  poll time: 1000
  test cycle: 64
  xaxicdma type: mem ipaddr 0xffdc00000
************xaxicdmaRegDump begin*************
* control:                        0x00000000 *
* status:                         0x00000002 *
* current descriptor pointer:     0x00000000 *
* current descriptor pointer msb: 0x00000000 *
* tail descriptor pointer:        0x00000000 *
* tail descriptor pointer msb:    0x00000000 *
* source address:                 0x00000000 *
* source address msb:             0x00000011 *
* destination address:            0x00000000 *
* destination address msb:        0x00000010 *
* bytes to transfer:              0x00400000 *
************xaxicdmaRegDump end***************
speed: 3710.14MB/s, cost times: 69ms

c2h方向（Inbound?）可以正确传输，PCIe2.0x4，带宽410MB/s，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c c2h -s 0x1000000000 -d cmem -m 0xffdc00000 -l 0x400000 -e 8 -p 1000
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0x1000000000 to 0xe0000000
  cmd type: c2h
  poll time: 1000
  test cycle: 8
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fffb59e2000
speed: 410.26MB/s, cost times: 78ms

h2c方向（Outbound?），报错DMA Decode Error，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -s cmem -d 0x1000000000 -m 0xffdc00000 -l 0x400000 -e 8 -p 1000
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0xe0000000 to 0x1000000000
  cmd type: h2c
  poll time: 1000
  test cycle: 8
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fffa7f6b000
error: xaxicdmaWait sr 0x00005042 reset...

Peer to Peer，cdma -> ssd，报错DMA Slave Error，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c c2h -s 0x1000000000 -d 0xc41100000 -m 0xffdc00000 -l 0x1000 -e 1 -p 1000          
global params: 
  dma type: xaxicdma trans 0x1000 bytes from 0x1000000000 to 0xc41100000
  cmd type: c2h
  poll time: 1000
  test cycle: 1
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fff91b6a000
error: xaxicdmaWait sr 0x00005022 reset...

Peer to Peer，ssd -> cdma，报错DMA Decode Error，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c c2h -s 0xc41100000 -d 0x1000000000 -m 0xffdc00000 -l 0x1000 -e 1 -p 1000                 
global params: 
  dma type: xaxicdma trans 0x1000 bytes from 0xc41100000 to 0x1000000000
  cmd type: c2h
  poll time: 1000
  test cycle: 1
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fffaa49e000
error: xaxicdmaWait sr 0x00005042 reset...

查看手册，

显然，DMA内存的地址和PCIe memory重合导致，把DMA内存地址改为0x60000000，h2c和c2h都正常了。

root@t2080rdb:~# dma_benchmark -i xaxicdma -c c2h -s 0x1000000000 -d cmem -m 0xffdc00000 -l 0x400000 -e 8 -p 1000
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0x1000000000 to 0x60000000
  cmd type: c2h
  poll time: 1000
  test cycle: 8
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fff800e4000
speed: 1600.00MB/s, cost times: 20ms
root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -s cmem -d 0x1000000000 -m 0xffdc00000 -l 0x400000 -e 16 -p 1000
global params: 
  dma type: xaxicdma trans 0x400000 bytes from 0x60000000 to 0x1000000000
  cmd type: h2c
  poll time: 1000
  test cycle: 16
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fff7fcb7000
************xaxicdmaRegDump begin*************
* control:                        0x00000000 *
* status:                         0x00000002 *
* current descriptor pointer:     0x00000000 *
* current descriptor pointer msb: 0x00000000 *
* tail descriptor pointer:        0x00000000 *
* tail descriptor pointer msb:    0x00000000 *
* source address:                 0x60000000 *
* source address msb:             0x00000000 *
* destination address:            0x00000000 *
* destination address msb:        0x00000010 *
* bytes to transfer:              0x00400000 *
************xaxicdmaRegDump end***************
speed: 1560.98MB/s, cost times: 41ms

Peer to Peer，

root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -s 0xe4100000 -d 0x1000000000 -m 0xffdc00000 -l 0x4000 -e 8 -p 1000         
global params: 
  dma type: xaxicdma trans 0x4000 bytes from 0xe4100000 to 0x1000000000
  cmd type: h2c
  poll time: 1000
  test cycle: 8
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fffac72a000
speed: 125.00MB/s, cost times: 1ms
root@t2080rdb:~# dma_benchmark -i xaxicdma -c h2c -d 0xe4100000 -s 0x1000000000 -m 0xffdc00000 -l 0x4000 -e 8 -p 1000  
global params: nvme 0000:08:00.0: async event result 00010000

  dma type: xaxicdma trans 0x4000 bytes from 0x1000000000 to 0xe4100000
  cmd type: h2c
  poll time: 1000
  test cycle: 8
  xaxicdma type: mem ipaddr 0xffdc00000
regmap: pa 0xffdc00000 size 0x2c va 0x3fff98cf6000
speed: 125.00MB/s, cost times: 1ms
root@t2080rdb:~# 

Xilinx xdma

2017.4驱动在翻译用户态地址时出错，修改采用32位dma，否则RC一直进错误中断，

[    5.287316] set_dma_mask():sizeof(dma_addr_t) == 8
[    5.290860] set_dma_mask():Could not set 64-bit DMA mask.
[    5.295017] set_dma_mask():Using a 32-bit DMA mask.

测试，

root@t2080rdb:/usr/xdma/tests# ./dma_from_device -d /dev/xdma0_c2h_0 -a 0x1000000000 -s 4096 -v
sscanf() = 1, value = 0x00001000
device = /dev/xdma0_c2h_0, address = 0x1000000000, size = 0x00001000, offset = 0x00000000, count = 1
host memory buffer = 0x10014000

传输失败，completed_desc_count为0，

char_sgdma_open(): C2H0: char_sgdma_open(0xc0000000f1ba8d28, 0xc00000007e6e7600)
char_sgdma_llseek():char_sgdma_llseek: pos=68719476736
char_sgdma_read_write(): C2H0: seq:0 file=0xc00000007e6e7600, buf=0x0000000010014000, count=4096, pos=68719476736
char_sgdma_read_write(): C2H0: seq:0 dir_to_dev=0 read request
char_sgdma_read_write(): C2H0: C2H engine channel 0 (engine num 1)= 0xc0000000f1988600
char_sgdma_read_write(): C2H0: lro = 0xc0000000de429000
char_sgdma_read_write(): C2H0: res = 0, remaining = 4096
transfer_create():transfer_create()
transfer_create():mapped_pages=1.
transfer_create():sgl = 0xc00000007e3124c0.
transfer_create():hwnents=1.
transfer_create():sg_page(&sgl[0])=0xc0000000f79bf280.
transfer_create():sg_dma_address(&sgl[0])=0x0000000035b30000.
transfer_create():sg_dma_len(&sgl[0])=0x00001000.
transfer_create():transfer_create():
transfer_create():transfer->desc_bus = 0x35e40000.
transfer_build():SGLE    0: addr=0x0000000035b30000 length=0x00001000
transfer_build():DESC    0: cont_addr=0x35b30000 cont_len=0x00001000 ep_addr=0x1000000000
transfer_create():transfer 0xc0000000f18837c0 has 1 descriptors
transfer_data(): C2H0: seq:0 transfer=0xc0000000f18837c0.
transfer_dump():Descriptor Entry (Pre-Transfer)
dump_desc():0xc000000035e40000/0x00: 0x13004bad 0xad4b0013 magic|extra_adjacent|control
dump_desc():0xc000000035e40004/0x04: 0x00100000 0x00001000 bytes
dump_desc():0xc000000035e40008/0x08: 0x00000000 0x00000000 src_addr_lo
dump_desc():0xc000000035e4000c/0x0c: 0x10000000 0x00000010 src_addr_hi
dump_desc():0xc000000035e40010/0x00: 0x0000b335 0x35b30000 dst_addr_lo
dump_desc():0xc000000035e40014/0x04: 0x00000000 0x00000000 dst_addr_hi
dump_desc():0xc000000035e40018/0x08: 0x00000000 0x00000000 next_addr
dump_desc():0xc000000035e4001c/0x0c: 0x00000000 0x00000000 next_addr_pad
dump_desc():
transfer_queue(): C2H0: transfer_queue(transfer=0xc0000000f18837c0).
transfer_queue(): C2H0: transfer_queue(): starting C2H engine.
engine_start(): C2H0: engine_start(C2H): transfer=0xc0000000f18837c0.
engine_start(): C2H0: iowrite32(0x0000e435 to 0x8000080090ac5080) (first_desc_lo)
engine_start(): C2H0: iowrite32(0x00000000 to 0x8000080090ac5084) (first_desc_hi)
engine_start(): C2H0: iowrite32(0x00000000 to 0x8000080090ac5088) (first_desc_adjacent)
engine_start(): C2H0: ioread32(0x8000080090ac1040) (dummy read flushes writes).
engine_start_mode_config(): C2H0: iowrite32(0x00f83e1f to 0x8000080090ac1004) (control)
engine_start_mode_config(): C2H0: ioread32(0x8000080090ac1040) = 0x00000001 (dummy read flushes writes).
xdma_isr():(irq=67) <<<< INTERRUPT SERVICE ROUTINE
xdma_isr():ch_irq = 0x00000002
xdma_isr():user_irq = 0x00000000
xdma_isr(): C2H0: schedule_work(engine=c0000000f1988600)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1000).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1000) returned 0x1fc10006.
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1040) returned 0x00000010 (status).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1004) returned 0x00f83e1e (control)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5080) returned 0x0000e435 (first_desc_lo)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5084) returned 0x00000000 (first_desc_hi)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5088) returned 0x00000000 (first_desc_adjacent).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1048) returned 0x00000000 (completed_desc_count).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1090) returned 0x00f83e1e (interrupt_enable_mask)
engine_status_read(): C2H0: Status of SG DMA C2H engine:
engine_status_read(): C2H0: ioread32(0x8000080090ac1040).
engine_status_read(): C2H0: status = 0x00000010: IDLE MAGIC_STOPPED 
engine_start(): C2H0: C2H engine 0xc0000000f1988600 now running
transfer_queue(): C2H0: transfer=0xc0000000f18837c0 started C2H engine with transfer 0xc0000000f18837c0.
transfer_queue(): C2H0: engine->running = 1
engine_service_work(): C2H0: engine_service() for C2H engine c0000000f1988600
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1000).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1000) returned 0x1fc10006.
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1040) returned 0x00000010 (status).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1004) returned 0x00f83e1e (control)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5080) returned 0x0000e435 (first_desc_lo)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5084) returned 0x00000000 (first_desc_hi)
engine_reg_dump(): C2H0: ioread32(0x8000080090ac5088) returned 0x00000000 (first_desc_adjacent).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1048) returned 0x00000000 (completed_desc_count).
engine_reg_dump(): C2H0: ioread32(0x8000080090ac1090) returned 0x00f83e1e (interrupt_enable_mask)
engine_status_read(): C2H0: Status of SG DMA C2H engine:
engine_status_read(): C2H0: ioread32(0x8000080090ac1040).
engine_status_read(): C2H0: status = 0x00000010: IDLE MAGIC_STOPPED 
engine_service_shutdown(): C2H0: engine just went idle, resetting RUN_STOP.
xdma_engine_stop(): C2H0: xdma_engine_stop(engine=c0000000f1988600)
xdma_engine_stop(): C2H0: Stopping SG DMA C2H engine; writing 0x00f83e1e to 0x8000080090ac1004.
xdma_engine_stop(): C2H0: xdma_engine_stop(C2H) done
engine_service(): C2H0: desc_count = 0
engine_service(): C2H0: head of queue transfer 0xc0000000f18837c0 has 1 descriptors
engine_service(): C2H0: Engine completed 0 desc, 0 not yet dequeued
engine_err_handle(): C2H0: Aborted C2H engine transfer 0xc0000000f18837c0
engine_err_handle(): C2H0: C2H engine was 0 descriptors into transfer (with 1 desc)
engine_err_handle(): C2H0: C2H engine status = 16
xdma_engine_stop(): C2H0: xdma_engine_stop(engine=c0000000f1988600)
xdma_engine_stop(): C2H0: Stopping SG DMA C2H engine; writing 0x00f83e1e to 0x8000080090ac1004.
xdma_engine_stop(): C2H0: xdma_engine_stop(C2H) done
engine_start(): C2H0: engine_start(C2H): transfer=0xc0000000f18837c0.
engine_start(): C2H0: iowrite32(0x0000e435 to 0x8000080090ac5080) (first_desc_lo)
engine_start(): C2H0: iowrite32(0x00000000 to 0x8000080090ac5084) (first_desc_hi)
engine_start(): C2H0: iowrite32(0x00000000 to 0x8000080090ac5088) (first_desc_adjacent)
engine_start(): C2H0: ioread32(0x8000080090ac1040) (dummy read flushes writes).

查看寄存器发现，magic_stopped被置位，理论上应该是0x04，表示descriptor_completed，

打印描述符函数，

static void dump_desc(struct xdma_desc *desc_virt)
{
	int j;
	u32 *p = (u32 *)desc_virt;
	static char * const field_name[] = {
		"magic|extra_adjacent|control", "bytes", "src_addr_lo",
		"src_addr_hi", "dst_addr_lo", "dst_addr_hi", "next_addr",
		"next_addr_pad"};
	char *dummy;

	/* remove warning about unused variable when debug printing is off */
	dummy = field_name[0];

	for (j = 0; j < 8; j += 1) {
		dbg_desc("0x%08lx/0x%02lx: 0x%08x 0x%08x %s\n",
			 (uintptr_t)p, (uintptr_t)p & 15, (int)*p,
			 le32_to_cpu(*p), field_name[j]);
		p++;
	}
	dbg_desc("\n");
}

描述符，

函数调用，每个函数都用了cpu_to_le32和le32_to_cpu，没什么问题，

sg_aio_read_write -> transfer_create -> xdma_desc_alloc -> transfer_build -> xdma_desc_link -> xdma_desc_control -> xdma_desc_adjacent

怀疑32位dma的物理地址翻译sgm_get_user_pages也不对，即使是在4GB下，增加一个ioctl，采用用户态的DMA内存来测试一下，64位dma打印0x0000010000d23000都超过4GB了，垃圾，

transfer_create():transfer_create()
sg_create_mapper():Allocated 24 bytes for page pointer array for 3 pages @0xc0000000f1a6c8c0.
sg_create_mapper():Allocated 96 bytes for scatterlist for 3 pages @0xc00000007e136c40.
sg_create_mapper():sg_mapping_t *sgm=0xc00000007eb48a60
sg_create_mapper():sgm->pages=0xc0000000f1a6c8c0
sgm_get_user_pages():sgm_map_user_pages()
sgm_get_user_pages():sgl = 0xc00000007e136c40.
sgm_get_user_pages():pages=0xc0000000f1a6c8c0
sgm_get_user_pages():start = 0x10014000.
sgm_get_user_pages():first = 65556, last = 65556
sgm_get_user_pages():get_user_pages_fast(268517376, nr_pages = 1) == 1.
sgm_get_user_pages():sgm->mapped_pages = 1
sgm_get_user_pages():0000: page=0xc0000000f6e2dfa8
sgm_get_user_pages():sg_page(&sgl[0]) = 0xc0000000f6e2dfa8 (pfn = 3363).
sgm_get_user_pages():sg_dma_address(&sgl[0])=0x0000000000000000.
sgm_get_user_pages():sg_dma_len(&sgl[0])=0x00000000.
sgm_get_user_pages():0000: page=0x          (null), pfn=3363, offset=0, length=4096 (SINGLE/FIRST/LAST)
transfer_create():mapped_pages=1.
transfer_create():sgl = 0xc00000007e136c40.
transfer_create():hwnents=1.
transfer_create():sg_page(&sgl[0])=0xc0000000f6e2dfa8.
transfer_create():sg_dma_address(&sgl[0])=0x0000010000d23000.
transfer_create():sg_dma_len(&sgl[0])=0x00001000.
transfer_create():transfer_create():
transfer_create():transfer->desc_bus = 0x100f1cec000.
transfer_build():SGLE    0: addr=0x0000010000d23000 length=0x00001000
transfer_build():DESC    0: cont_addr=0x10000d23000 cont_len=0x00001000 ep_addr=0x1000000000

NVMe SSD

在T2080平台上测试，T2080用一个PCIe2.0x4接到PLX的Switch上，

root@t2080rdb:~# blk_benchmark -r /dev/nvme0n1 -s 0 -l 0x400000 -c 32 
speed: 414.24MB/s, cost times: 309ms
root@t2080rdb:~# blk_benchmark -r /dev/nvme0n1 -s 0 -l 0x400000 -c 128
speed: 420.36MB/s, cost times: 1218ms
root@t2080rdb:~# blk_benchmark -w /dev/nvme0n1 -s 0 -l 0x400000 -c 128 
speed: 387.59MB/s, cost times: 1321ms

采用PCIe3.0x4直连，速度不变，所以瓶颈不在这里，

root@t2080rdb:~# blk_benchmark -r /dev/nvme0n1 -l 0x400000 -c 128
speed: 421.75MB/s, cost times: 1214ms
root@t2080rdb:~# blk_benchmark -w /dev/nvme0n1 -l 0x400000 -c 128
speed: 389.65MB/s, cost times: 1314ms

读FPGA BAR空间时报错，

PCIe error(s) detected
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000

写FPGA BAR空间时报错，

PCIe error(s) detected
nvme 0000:09:00.0: async event result 00010300
PCIe ERR_DR register: 0x80100000
PCIe ERR_CAP_STAT register: 0x80000001
PCIe ERR_CAP_R0 register: 0x00000800
PCIe ERR_CAP_R1 register: 0x00000000
PCIe ERR_CAP_R2 register: 0x00000000
PCIe ERR_CAP_R3 register: 0x00000000
nvme 0000:09:00.0: Failed status: 3, reset controller
nvme 0000:09:00.0: Cancelling I/O 4 QID 2

参考T2080RM 20.4.36.4小节，ERR_DR 0x80100000对应如下两位，


参考T2080RM 20.6.1.12.4小节，outbound ERR_CAP_R0 0x00000800对应，FMT=0，TYPE=4，


Peer to Peer的memory范围设置错误，读盘正确，

root@t2080rdb:~# nvmeqe_benchmark -r /dev/nvme0n1 -p 0xe2000000 -s 0 -l 0x400000 -c 16
speed: 1488.37MB/s, cost times: 43ms

写盘有问题，

root@t2080rdb:~# nvmeqe_benchmark -w /dev/nvme0n1 -p 0xe2000000 -s 0 -l 0x400000 -c 16
nvme 0000:05:00.0: Failed status: 3, reset controller
nvme 0000:05:00.0: Cancelling I/O 3 QID 6
irq 19: nobody cared (try booting with the "irqpoll" option)
CPU: 6 PID: 0 Comm: swapper/6 Tainted: G           O    4.1.35-rt41-fdk-1.0.0-20190116.1935 #39
Call Trace:
[c0000000fff87af0] [c000000000872e6c] .dump_stack+0xac/0xec (unreliable)
[c0000000fff87b80] [c0000000000852d0] .__report_bad_irq+0x4c/0x138
[c0000000fff87c20] [c000000000085a3c] .note_interrupt+0x2f0/0x34c
[c0000000fff87cd0] [c000000000082054] .handle_irq_event_percpu+0x150/0x200
[c0000000fff87da0] [c00000000008215c] .handle_irq_event+0x58/0xa4
[c0000000fff87e20] [c0000000000861bc] .handle_fasteoi_irq+0xd4/0x280
[c0000000fff87ea0] [c0000000000813b0] .generic_handle_irq+0x4c/0x70
[c0000000fff87f20] [c000000000005ac8] .__do_irq+0x5c/0xa8
[c0000000fff87f90] [c000000000013018] .call_do_irq+0x14/0x24
[c0000000f10e79c0] [c000000000005b98] .do_IRQ+0x84/0x118
[c0000000f10e7a50] [c000000000000d28] restore_check_irq_replay+0x2c/0x70
--- interrupt: 501 at .book3e_idle+0x24/0x4c
    LR = .book3e_idle+0x24/0x4c
[c0000000f10e7d40] [c00000000000920c] .arch_cpu_idle+0x34/0xa0 (unreliable)
[c0000000f10e7db0] [c000000000076550] .cpu_startup_entry+0x204/0x2ac
[c0000000f10e7eb0] [c00000000001e904] .start_secondary+0x3f8/0x538
[c0000000f10e7f90] [c00000000000046c] start_secondary_prolog+0x10/0x14
handlers:
[<8000000000b93500>] .nvme_irq [nvmeqe]
Disabling IRQ #19
nvme 0000:05:00.0: Failed status: 3, reset controller
nvme 0000:05:00.0: Cancelling I/O 3 QID 6
nvme 0000:05:00.0: Failed status: 3, reset controller
nvme 0000:05:00.0: Cancelling I/O 0 QID 1
irq 19: nobody cared (try booting with the "irqpoll" option)
CPU: 0 PID: 797 Comm: kworker/u16:3 Tainted: G           O    4.1.35-rt41-fdk-1.0.0-20190116.1935 #39
Workqueue: nvme .nvme_reset_workfn [nvmeqe]
Call Trace:
[c0000000fffb7af0] [c000000000872e6c] .dump_stack+0xac/0xec (unreliable)
[c0000000fffb7b80] [c0000000000852d0] .__report_bad_irq+0x4c/0x138
[c0000000fffb7c20] [c000000000085a3c] .note_interrupt+0x2f0/0x34c
[c0000000fffb7cd0] [c000000000082054] .handle_irq_event_percpu+0x150/0x200
[c0000000fffb7da0] [c00000000008215c] .handle_irq_event+0x58/0xa4
[c0000000fffb7e20] [c0000000000861bc] .handle_fasteoi_irq+0xd4/0x280
[c0000000fffb7ea0] [c0000000000813b0] .generic_handle_irq+0x4c/0x70
[c0000000fffb7f20] [c000000000005ac8] .__do_irq+0x5c/0xa8
[c0000000fffb7f90] [c000000000013018] .call_do_irq+0x14/0x24
[c0000000f14e6dc0] [c000000000005b98] .do_IRQ+0x84/0x118
[c0000000f14e6e50] [c000000000000d28] restore_check_irq_replay+0x2c/0x70
--- interrupt: 501 at .arch_local_irq_restore+0x60/0x70
    LR = .arch_local_irq_restore+0x60/0x70
[c0000000f14e7140] [c000000000068900] .vtime_common_account_irq_enter+0x34/0x60 (unreliable)
[c0000000f14e71b0] [c00000000003d43c] .__do_softirq+0xd8/0x314
[c0000000f14e72b0] [c00000000003dba8] .irq_exit+0xb8/0xe4
[c0000000f14e7320] [c00000000000f004] .timer_interrupt+0x94/0xc4
[c0000000f14e73a0] [c000000000000d50] restore_check_irq_replay+0x54/0x70
--- interrupt: 901 at .arch_local_irq_restore+0x60/0x70
    LR = .arch_local_irq_restore+0x60/0x70
[c0000000f14e7690] [0000000000000001] 0x1 (unreliable)
[c0000000f14e7700] [c00000000086ffb4] ._raw_spin_unlock_irqrestore+0x60/0x74
[c0000000f14e7770] [c000000000084514] .__setup_irq+0x478/0x7d4
[c0000000f14e7820] [c000000000084a58] .request_threaded_irq+0x110/0x23c
[c0000000f14e78d0] [8000000000b8d1e0] .queue_request_irq+0x4c/0x8c [nvmeqe]
[c0000000f14e7940] [8000000000b8f168] .nvme_dev_start.part.45+0x1ac/0x4e4 [nvmeqe]
[c0000000f14e7a10] [8000000000b8f4fc] .nvme_dev_resume+0x5c/0x170 [nvmeqe]
[c0000000f14e7aa0] [8000000000b8f654] .nvme_reset_failed_dev+0x30/0x170 [nvmeqe]
[c0000000f14e7b30] [8000000000b8a05c] .nvme_reset_workfn+0x24/0x38 [nvmeqe]
[c0000000f14e7ba0] [c000000000053424] .process_one_work+0x1f8/0x438
[c0000000f14e7c40] [c0000000000537e4] .worker_thread+0x180/0x5b0
[c0000000f14e7d30] [c0000000000595a4] .kthread+0xf0/0x110
[c0000000f14e7e30] [c000000000000998] .ret_from_kernel_thread+0x58/0xc0
handlers:
[<8000000000b93500>] .nvme_irq [nvmeqe]
Disabling IRQ #19

但是把另外一个SSD的BAR空间写入盘没有问题，而且正确性没有问题，

root@t2080rdb:~# nvmeqe_benchmark -w /dev/nvme0n1 -p 0xe4100000 -s 0 -l 0x4000 -c 16      
speed: 250.00MB/s, cost times: 1ms
root@t2080rdb:~# dd if=/dev/zero of=/dev/nvme0n1 bs=4K count=1
1+0 records in
1+0 records out
4096 bytes (4.0KB) copied, 0.000223 seconds, 17.5MB/s
root@t2080rdb:~# dd if=/dev/nvme0n1 of=a.bin bs=4K count=1    
1+0 records in
1+0 records out
4096 bytes (4.0KB) copied, 0.000303 seconds, 12.9MB/s
root@t2080rdb:~# hexdump -C a.bin                             
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000
root@t2080rdb:~# memtool md 0xc44100000+0x40
c44100000: ff3f033c 2000f000 00020100 00000000                .?.< ...........
c44100010: 00000000 01004600 00000000 01000000                ......F.........
c44100020: 00000000 ff00ff00 00803cef 00010000                ..........<.....
c44100030: 002003ef 00010000 00000000 00000000                . ..............
root@t2080rdb:~# nvmeqe_benchmark -w /dev/nvme0n1 -p 0xe4100000 -s 0 -l 0x1000 -c 1  
speed: infMB/s, cost times: 0ms
root@t2080rdb:~# dd if=/dev/nvme0n1 of=a.bin bs=4K count=1                                  
1+0 records in
1+0 records out
4096 bytes (4.0KB) copied, 0.000676 seconds, 5.8MB/s
root@t2080rdb:~# hexdump -C a.bin 
00000000  ff 3f 03 3c 20 00 f0 00  00 02 01 00 00 00 00 00  |.?.< ...........|
00000010  00 00 00 00 01 00 46 00  00 00 00 00 01 00 00 00  |......F.........|
00000020  00 00 00 00 ff 00 ff 00  00 80 3c ef 00 01 00 00  |..........<.....|
00000030  00 20 03 ef 00 01 00 00  00 00 00 00 00 00 00 00  |. ..............|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000
root@t2080rdb:~# 

4盘和8盘读FPGA，FPGA PCIe2.0x8，理论带宽4GB/s，实际P2P带宽2.4GB/s，

root@t2080rdb:~# ./test2.sh 
speed: 761.90MB/s, cost times: 84ms
speed: 547.01MB/s, cost times: 117ms
speed: 542.37MB/s, cost times: 118ms
speed: 542.37MB/s, cost times: 118ms
root@t2080rdb:~# ./test2.sh 
speed: 432.43MB/s, cost times: 148ms
speed: 283.19MB/s, cost times: 226ms
speed: 281.94MB/s, cost times: 227ms
speed: 281.94MB/s, cost times: 227ms
speed: 272.34MB/s, cost times: 235ms
speed: 273.50MB/s, cost times: 234ms
speed: 273.50MB/s, cost times: 234ms
speed: 271.19MB/s, cost times: 236ms