[SPDK/NVMe存储技术分析]007 - 初识UIO

https://www.cnblogs.com/vlhn/p/7761869.html
注： 要进一步搞清楚SSD盘对应的PCI的BAR寄存器的映射，有必要先了解一下UIO(Userspace I/O)。
UIO(Userspace I/O)是运行在用户空间的I/O技术。在Linux系统中，一般的设备驱动都是运行在内核空间，而在用户空间使用应用程序调用即可。而UIO则是将设备驱动的很少一部分运行在内核空间，而在用户空间实现驱动的绝大多数功能。那么，在内核空间UIO要做的事情就变得很简单，分为两种：

分配和记录设备需要的资源和注册UIO设备
实现必须在内核空间实现的中断处理函数
为了对UIO有一个直观的认识，先上个图：

了解了UIO 驱动在Linux系统中的位置后，让我们对参考资料(Linux User Space Device Drivers)的部分内容做一个中英文对照翻译以加深对UIO的理解。

1 Device Driver Architectures | 设备驱动架构

Linux device drivers are typically designed as kernel drivers running in kernel space
典型的Linux设备驱动都是被设计为运行在内核空间的内核驱动
User space I/O is another alternative device driver architecture that has been supported by the Linux kernel since 2.6.24
从Linux内核版本2.6.24开始，就支持另一种可作为内核设备驱动的替代方案的设备驱动架构，也就是用户空间I/O
People in the Linux kernel community may not always agree on the need to have user space I/O
在Linux内核社区的人们不总是赞成使用用户空间I/O
Industrial I/O cards have been taking advantage of user space I/O for quite some time
在工业中使用的I/O卡利用用户空间I/O的优点已经有一阵子了
For some types of devices, creating a Linux kernel driver may be overkill
对某些类型的设备来说，创建对应的Linux内核驱动很可能代价太高
Soft IP for FPGAs can have unique requirements that don’t always fit the mold
FPGA的软IP有独特的需求，将驱动放在内核实现并不总是适合的
2 Legacy User Space Driver Methods (/dev/mem) | 传统的用户态驱动实现方法(/dev/mem)

A character driver referred to as /dev/mem exists in the kernel that will map device memory into user space
With this driver user space applications can access device memory
Memory access can be disabled in the kernel configuration as this is a big security hole (CONFIG_STRICT_DEVMEM)
Must be root user
A great tool for prototyping or maybe testing new hardware, but is not considered to be an acceptable production solution for a user space device driver
Since it can map any address into user space a buggy user space driver could crash the kernel
3 Introduction to UIO | UIO概述

The Linux kernel provides a framework for doing user space drivers called UIO
The framework is a character mode kernel driver (in drivers/uio) which runs as a layer under a user space driver
UIO helps to offload some of the work to develop a driver
The “U” in UIO is not for universal

• Devices well handled by kernel frameworks should ideally stay in the kernel (if you ask many kernel developers)
• Networking is one area where semiconductor vendors are doing user space I/O to get improved performance
  UIO handles simple device drivers really well
• Simple driver: Device access and interrupt processing with no need to access kernel frameworks
  4 Kernel Space Driver Characteristics | 内核空间驱动的特点

4.1 Advantages | 优点

Runs in kernel space in the highest privilege mode to allow access to interrupts and hardware resources
There are a lot of kernel services such that kernel space drivers can be designed for complex devices
The kernel provides an API to user space which allows multiple applications to access a kernel space driver simultaneously

• Larger and more scalable software systems can be architected
  Many drivers tend to be kernel space
• Asking questions in the open source community is going to be easier
• Pushing drivers to the open source community is likely easier
  4.2 Disadvantages | 缺点

System call overhead to access drivers

• A switch from user space to kernel space (and back) is required
• Overhead can be non-deterministic having impact on real time applications
  Challenging learning curve for developers
• The kernel API is different from the application level API such that it takes time to become productive
  Bugs can be fatal causing a kernel crash
  Challenging to debug
• Kernel code is highly optimized and there are different debug tools
  Frequent kernel API changes
• Kernel drivers built for one kernel version may not build for another
  5 User Space Device Driver Characteristics | 用户空间驱动的特点

5.1 Advantages | 优点

Less challenging to debug as debug tools are more readily available and common to normal application development
User space services such as floating point are available
Device access is very efficient as there is no system call required
The application API of Linux is very stable
The driver can be written in any language, not just “C”
5.2 Disadvantages | 缺点

No access to the kernel frameworks and services

• Contiguous memory allocation, direct cache control, and DMA are not available
• May have to duplicate kernel code or use a kernel driver to supplement
  Interrupt handling cannot be done in user space
• It must be handled by a kernel driver which notifies user space causing some delay
  There is no predefined API to allow applications to access the device driver
• Concurrency must also be considered if multiple applications access a driver
  6 UIO Framework Features | UIO框架的特性

There are two distinct UIO device drivers provided by Linux in drivers/uio
UIO Driver (drivers/uio.c)

• For more advanced users as a minimal kernel space driver is required to setup the UIO framework
• This is the most universal and likely to handle all situations since the kernel space driver can be very custom
• The majority of work can be accomplished in the user space driver
  UIO Platform Device Driver (drivers/uio_pdev_irqgen.c)
  This driver augments the UIO driver such that no kernel space driver is required
  It provides the required kernel space driver for uio
  It works with device tree making it easy to use
  The device tree node for the device needs to use “generic uio” in it’s compatible
  Best starting point since no kernel space code is needed
  7 UIO Driver Kernel Configuration | 支持UIO驱动所需要的内核配置

UIO drivers must be configured in the Linux kernel
CONFIG_UIO=y
CONFIG_UIO_PDRV_GENIRQ=y
8 UIO Platform Device Driver Details | UIO平台服务驱动详解

The user provides only a user space driver
The UIO platform device driver configures from the device tree and registers a UIO device
The user space driver has direct access to the hardware
The user space driver gets notified of an interrupt by reading the UIO device file descriptor

9 Kernel UIO API - Sys Filesystem

The UIO driver in the kernel creates file attributes in the sys filesystem describing the UIO device
/sys/class/uio is the root directory for all the file attributes
A separate numbered directory structure is created under /sys/class/uio for each UIO device

• First UIO device: /sys/class/uio/uio0
• /sys/class/uio/uio0/name contains the name of the device which correlates to the name in the uio_info structure
• /sys/class/uio/uio0/maps is a directory that has all the memory ranges for the device
• Each numbered map directory has attributes to describe the device memory including the address, name, offset and size
  /sys/class/uio/uio0/maps/map0
  10 User Space Driver Flow | 用户态驱动工作流程

01 - The kernel space UIO device driver(s) must be loaded before the user space driver is started (if using modules)
02 - The user space application is started and the UIO device file is opened (/dev/uioX where X is 0, 1, 2 …)

• From user space, the UIO device is a device node in the file system just like any other device
  03 - The device memory address information is found from the relevant sysfs directory, only the size is needed
  04 - The device memory is mapped into the process address space by calling the mmap() function of the UIO driver
  05 - The application accesses the device hardware to control the device
  06 - The device memory is unmapped by calling munmap()
  07 - The UIO device file is closed
  11 User Space Driver Example | 用户态驱动示例

复制代码

 1 #define UIO_SIZE "/sys/class/uio/uio0/maps/map0/size"
 2 
 3 int main(int argc, char **argv)
 4 {
 5         int             uio_fd;
 6         unsigned int    uio_size;
 7         FILE            *size_fp;
 8         void            *base_address;
 9 
10         /*
11          * 1. Open the UIO device so that it is ready to use
12          */
13         uio_fd = open("/dev/uio0", O_RDWR);
14 
15         /*
16          * 2. Get the size of the memory region from the size sysfs file
17          *    attribute
18          */
19         size_fp = fopen(UIO_SIZE, O_RDONLY);
20         fscanf(size_fp, "0x%08X", &uio_size);
21 
22         /*
23          * 3. Map the device registers into the process address space so they
24          *    are directly accessible
25          */
26         base_address = mmap(NULL, uio_size,
27                            PROT_READ|PROT_WRITE,
28                            MAP_SHARED, uio_fd, 0);
29 
30         // Access to the hardware can now occur ...
31 
32         /*
33          * 4. Unmap the device registers to finish
34          */
35         munmap(base_address, uio_size);
36 
37         ...
38 }

复制代码
12 Mapping Device Memory Details | 设备内存映射详解

The character device driver framework of Linux provides the ability to map device memory into a user space process address space
A character driver may implement the mmap() function which a user space application can call
The mmap() function creates a new mapping in the virtual address space of the calling process

• A virtual address, corresponding to the physical address specified is returned
• It can also be used to map a file into a memory space such that the contents of the file are accessed by memory reads and writes
  Whenever the user space program reads or writes in the virtual address range it is accessing the device
  This provides improved performance as no system calls are required
  13 Mapping Device Memory Flow | 设备内存映射流程
  

14 User Space Application Interrupt Processing | 用户空间应用程序中断处理

Interrupts are never handled directly in user space
The interrupt can be handled by the UIO kernel driver which then relays it on to user space via the UIO device file descriptor
The user space driver that wants to be notified when interrupts occur calls select() or read() on the UIO device file descriptor

• The read can be done as blocking or non-blocking mode
  read() returns the number of events (interrupts)
  A thread could be used to handle interrupts
  Alternatively a user provided kernel driver can handle the interrupt and then communicate data to the user space driver through other mechanisms like shared memory
• This may be necessary for devices which have very fast interrupts
  15 User Space Application Interrupt Processing Example | 用户空间应用程序中断处理示例

复制代码

 1 int pending = 0;

 2 int reenable = 1;
 3 
 4 /*
 5  * 1. The UIO device is opened as previously described
 6  */
 7 int uio_fd = open("/dev/uio0", O_RDWR);
 8 
 9 /*
10  * 2. Read the UIO device file descriptor to wait for an interrupt,
11  *    the read blocks by default, a non blocking read can also be used
12  *
13  *    NOTE: The pending variable contains the number of interrupts that have
14  *          occurred if multiple
15  */
16 read(uio_fd, (void *)&pending, sizeof(int));
17 
18 
19 //
20 // add device specific processing like acking the interrupt in the device here
21 //
22 
23 
24 /*
25  * 3. Re-enable the interrupt at the interrupt controller level
26  */
27 write(uio_fd, (void *)&reenable, sizeof(int));

复制代码

Part II: Advanced UIO With Both User Space Application and Kernel Space Driver

16 UIO Driver Details | UIO驱动详解

The user provides a kernel driver and a user space driver
The kernel space driver is a platform driver configuring from the device tree and registering a UIO device
The kernel space driver can also provide an interrupt handler in kernel space
The user space driver has direct access to the hardware
17 Kernel UIO API - Basics | 内核UIO API基础

The API is small and simple to use API小且易用
复制代码
struct uio_info
– name : device name
– version : device driver version
– irq : interrupt number
– irq_flags : flags for request_irq()
– handler : driver irq handler (optional)
– mem[] : memory regions that can be mapped to user space
o addr : memory address
o memtype : type of memory region (physical, logical, virtual)
复制代码
18 Kernel UIO API - Registration | 内核UIO API - 注册

The function uio_register_device() connects the driver to the UIO framework
Requires a struct uio_info as an input
Typically called from the probe() function of a platform device driver
Creates device file /dev/uio# (#starting from 0) and all associated sysfs file attributes
The function uio_unregister_device() disconnects the driver from the UIO framework
Typically called from the cleanup function of a platform device driver
Deletes the device file /dev/uio#
19 Kernel Space Driver Example | 内核空间驱动示例

复制代码
1 probe()
2 {
3 /*
4 * 1. Platform device driver initialization in the driver probe() function
5 /
6 dev = devm_kzalloc(&pdev->dev, (sizeof(struct uio_timer_dev)), GFP_KERNEL);
7 res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
8 dev->regs = devm_ioremap_resource(&pdev->dev, res);
9 irq = platform_get_irq(pdev, 0);
10
11 /
12 * 2. Add basic UIO structure initialization
13 /
14 dev->uio_info.name = “uio_timer”;
15 dev->uio_info.version = 1;
16 dev->uio_info.priv = dev;
17
18 /
19 * 3. Add the memory region initialization for the UIO
20 /
21 dev->uio_info.mem[0].name = “registers”;
22 dev->uio_info.mem[0].addr = res->start;
23 dev->uio_info.mem[0].size = resource_size(res);
24 dev->uio_info.mem[0].memtype = UIO_MEM_PHYS;
25
26 /
27 * 4. Add the interrupt initialization for the UIO
28 /
29 dev->uio_info.irq = irq;
30 dev->uio_info.handler = uio_irq_handler;
31
32 /
33 * 5. Register the UIO device with the kernel framework
34 */
35 uio_register_device(&pdev->dev, &dev->info);
36 }
复制代码
20 UIO Framework Details | UIO框架详解

UIO Driver

• The device tree node for the device can use whatever you want in the compatible property as it only has to match what is used in the kernel space driver as with any platform device driver
  UIO Platform Device Driver
• The device tree node for the device needs to use “generic - uio” in it’s compatible property

参考资料

Linux User Space Device Drivers
UIO: user-space drivers
Howto: Accessing PCI devices from userspace
Kernel space: the UIO interface for device drivers
Paper: Userspace I/O drivers in a realtime context
Send a wise man on an errand, and say nothing to him. | 智者当差，无须交待。