linux V4L2子系统——v4l2的结构体（5）之videobuf2(vb2)
备注：
1. Kernel版本：5.4
2. 使用工具：Source Insight 4.0
简介
videobuf2 用于连接 V4L2 驱动层与用户空间层，提供数据交流的通道，它可以分配并管理视频帧数据。videobuf 层实现了很多 ioctl 函数，包括 buffer 分配、入队、出队和数据流控制。
详细说明请见：V4L2框架-videobuf2
数据类型
不是所有的视频设备都使用同一种类型的 buffer，事实上，buffer 类型可以概括为三种：
- buffer 在虚拟地址以及物理地址上面都是分散的。几乎所有的用户空间 buffer 都是这种类型，在内核空间中，这种类型的 buffer 并不总是能够满足需要，因为这要求硬件可以进行分散的 DMA 操作。
- 物理地址分散，虚拟地址连续。也就是通过 vmalloc 分配的 buffer，换句话说很难使用 DMA 来操作这些buffer。
- 物理地址连续（虚拟地址不关心）。在分段式系统上面分配这种类型的 buffer 是不可靠的（分段式-也即页式系统上面有可能在长时间运行之后内存出现大量碎片，从而导致连续的内存空间很难获得，所以说不可靠），但是简单 DMA 控制器只能够适用于这种类型的 buffer。也有分段式的 DMA 操作，不过那种需要用到 IOMMU 来实现。
这三种类型的 buffer 对应的相关操作函数分别对应于 videobuf2-dma-sg.c,videobuf2-vmalloc.c,videobuf2-dma-contig.c。
根据buffer的不同，，需要用到的内核头文件也不同：
<media/videobuf-dma-sg.h> /* 物理地址分散 */
<media/videobuf-vmalloc.h> /* vmalloc() 分配的buffer */
<media/videobuf-dma-contig.h> /* 物理地址连续 */
数据结构定义
videobuf2的核心数据结构是一个缓冲区队列，用来管理所有的缓冲区。
缓冲区队列用 struct vb2_queue 描述，缓冲区用 struct vb2_buffer 描述。
struct vb2_ops 和 struct vb2_mem_ops 结构体中的函数指针需要驱动实现。
数据结构的拓扑关系图如下图所示：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UpqgCbM8-1655629130291)(en-resource://database/1456:1)]
视频帧数据保存和管理的数据结构由 struct vb2_buffer 描述。
buf_struct_size 定义了缓冲区的大小，需要驱动设置；
num_buffers 定义了缓冲区的数量，不能大于 VIDEO_MAX_FRAME ，不能小于 min_buffers_needed。
缓冲区的大小由 buf_struct_size 定义，驱动可以定义自己的缓冲区，同时设置 buf_struct_size，若为0表示驱动不定义自己缓冲结构，则使用 sizeof(struct vb2_buffer) 初始化 buf_struct_size。
缓冲区类型由 enum v4l2_buf_type 枚举定义，图像采集（摄像头）使用 V4L2_BUF_TYPE_VIDEO_CAPTURE 类型。
struct vb2_queue 和 struct vb2_buffer 数据结构的关系可用下图描述，所以动态分配的 struct vb2_buffer 结构体保存到 bufs[VIDEO_MAX_FRAME] 数组中，由 struct vb2_queue 统一管理。struct vb2_buffer 结构体的最大数量由 VIDEO_MAX_FRAME 宏定义；queued_list 保存所有从用户空间入队的缓冲区；done_list 保存准备出队到用户空间的缓冲区。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rcCaVJt6-1655629130302)(en-resource://database/1457:1)]
vb2_queue
struct vb2_queue
该结构体可以当做是 videobuf2 的一个整体的抽象，要想使用 videobuf2 这个模块，就去初始化一个 vb2_queue 结构体吧，它会帮助我们建立起一个基于 videobuf2 的数据流管理模块的。
videobuf2类型枚举：
//源码： include/uapi/linux/videodev2.h
enum v4l2_buf_type {
V4L2_BUF_TYPE_VIDEO_CAPTURE = 1, // 图像采集
V4L2_BUF_TYPE_VIDEO_OUTPUT = 2, // 图像输出
V4L2_BUF_TYPE_VIDEO_OVERLAY = 3,
V4L2_BUF_TYPE_VBI_CAPTURE = 4,
V4L2_BUF_TYPE_VBI_OUTPUT = 5,
V4L2_BUF_TYPE_SLICED_VBI_CAPTURE = 6,
V4L2_BUF_TYPE_SLICED_VBI_OUTPUT = 7,
V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY = 8,
V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE = 9,
V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE = 10,
V4L2_BUF_TYPE_SDR_CAPTURE = 11,
V4L2_BUF_TYPE_SDR_OUTPUT = 12,
V4L2_BUF_TYPE_META_CAPTURE = 13,
V4L2_BUF_TYPE_META_OUTPUT = 14,
/* Deprecated, do not use */
V4L2_BUF_TYPE_PRIVATE = 0x80,
};
缓冲区类型：
//源码： include/uapi/linux/videobuf2.h
enum v4l2_memory {
V4L2_MEMORY_MMAP = 1, // 内存映射
V4L2_MEMORY_USERPTR = 2, // 用户空间指针
V4L2_MEMORY_OVERLAY = 3, // 内存重叠
V4L2_MEMORY_DMABUF = 4, // DMABUF
};
时间戳类型：
//源码： include/uapi/linux/videobuf2.h
/* Timestamp type */
#define V4L2_BUF_FLAG_TIMESTAMP_MASK 0x0000e000
#define V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN 0x00000000
#define V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC 0x00002000
#define V4L2_BUF_FLAG_TIMESTAMP_COPY 0x00004000
缓存访问方式：
视频图像缓冲区的I/O模型由 enum vb2_io_modes 描述，这里需要特别说明。
Linux系统分为用户空间和内核空间，应用程序处于用户空间，而内核运行在内核空间。V4L2子系统属于内核的组件，也运行在内核空间，其采集的数据也保存在内核空间的内存中。应用程序无法直接访问内核空间的内存，需要借助一些方法才能访问。缓冲区的I/O模式主要有以下三种方式：
（1）VB2_READ 和 VB2_WRITE
使用传统的I/O接口函数 read 和 write 进行读写。这种方式虽然最简单，但会在用户空间和内核空间之间来回复制数据，效率低，且在用户空间和内核空间都占用内存，开销大。
（2）VB2_MMAP
内存映射方式。mmap 把驱动程序中 videobuf2 管理的内存映射到用户空间，应用程序可直接访问 videobuf2 管理的内存。这种方式效率高，内存占用低。但是 videobuf2 管理的内存属于内核空间，不能被交换到SWAP分区中，若同时运行大量的进程，会占用较多的内存，而这些内存又不能被交换，会使系统的性能降低。
（3）VB2_USERPTR
用户指针模式。内存由应用程序分配，并将内存地址传递到内核V4L2驱动程序中，然后由V4L2驱动程序将数据填充到用户空间的内存中。
//源码： include/media/videobuf2-core.h
/**
* enum vb2_io_modes - queue access methods.
* @VB2_MMAP: driver supports MMAP with streaming API.
* @VB2_USERPTR: driver supports USERPTR with streaming API.
* @VB2_READ: driver supports read() style access.
* @VB2_WRITE: driver supports write() style access.
* @VB2_DMABUF: driver supports DMABUF with streaming API.
*/
enum vb2_io_modes {
VB2_MMAP = BIT(0), // 驱动程序支持mmap的流式API
VB2_USERPTR = BIT(1), // 驱动程序支持用户指针模式的流式API
VB2_READ = BIT(2), // 驱动程序支持read()方式访问
VB2_WRITE = BIT(3), // 驱动程序支持write()方式访问
VB2_DMABUF = BIT(4)), // 驱动程序支持DMABUF的流式API
};
//源码： include/media/videobuf2-core.h
/**
* struct vb2_queue - a videobuf queue.
*
* @type: private buffer type whose content is defined by the vb2-core
* caller. For example, for V4L2, it should match
* the types defined on &enum v4l2_buf_type.
* @io_modes: supported io methods (see &enum vb2_io_modes).
* @alloc_devs: &struct device memory type/allocator-specific per-plane device
* @dev: device to use for the default allocation context if the driver
* doesn't fill in the @alloc_devs array.
* @dma_attrs: DMA attributes to use for the DMA.
* @bidirectional: when this flag is set the DMA direction for the buffers of
* this queue will be overridden with %DMA_BIDIRECTIONAL direction.
* This is useful in cases where the hardware (firmware) writes to
* a buffer which is mapped as read (%DMA_TO_DEVICE), or reads from
* buffer which is mapped for write (%DMA_FROM_DEVICE) in order
* to satisfy some internal hardware restrictions or adds a padding
* needed by the processing algorithm. In case the DMA mapping is
* not bidirectional but the hardware (firmware) trying to access
* the buffer (in the opposite direction) this could lead to an
* IOMMU protection faults.
* @fileio_read_once: report EOF after reading the first buffer
* @fileio_write_immediately: queue buffer after each write() call
* @allow_zero_bytesused: allow bytesused == 0 to be passed to the driver
* @quirk_poll_must_check_waiting_for_buffers: Return %EPOLLERR at poll when QBUF
* has not been called. This is a vb1 idiom that has been adopted
* also by vb2.
* @supports_requests: this queue supports the Request API.
* @requires_requests: this queue requires the Request API. If this is set to 1,
* then supports_requests must be set to 1 as well.
* @uses_qbuf: qbuf was used directly for this queue. Set to 1 the first
* time this is called. Set to 0 when the queue is canceled.
* If this is 1, then you cannot queue buffers from a request.
* @uses_requests: requests are used for this queue. Set to 1 the first time
* a request is queued. Set to 0 when the queue is canceled.
* If this is 1, then you cannot queue buffers directly.
* @lock: pointer to a mutex that protects the &struct vb2_queue. The
* driver can set this to a mutex to let the v4l2 core serialize
* the queuing ioctls. If the driver wants to handle locking
* itself, then this should be set to NULL. This lock is not used
* by the videobuf2 core API.
* @owner: The filehandle that 'owns' the buffers, i.e. the filehandle
* that called reqbufs, create_buffers or started fileio.
* This field is not used by the videobuf2 core API, but it allows
* drivers to easily associate an owner filehandle with the queue.
* @ops: driver-specific callbacks
* @mem_ops: memory allocator specific callbacks
* @buf_ops: callbacks to deliver buffer information.
* between user-space and kernel-space.
* @drv_priv: driver private data.
* @buf_struct_size: size of the driver-specific buffer structure;
* "0" indicates the driver doesn't want to use a custom buffer
* structure type. for example, ``sizeof(struct vb2_v4l2_buffer)``
* will be used for v4l2.
* @timestamp_flags: Timestamp flags; ``V4L2_BUF_FLAG_TIMESTAMP_*`` and
* ``V4L2_BUF_FLAG_TSTAMP_SRC_*``
* @gfp_flags: additional gfp flags used when allocating the buffers.
* Typically this is 0, but it may be e.g. %GFP_DMA or %__GFP_DMA32
* to force the buffer allocation to a specific memory zone.
* @min_buffers_needed: the minimum number of buffers needed before
* @start_streaming can be called. Used when a DMA engine
* cannot be started unless at least this number of buffers
* have been queued into the driver.
*/
/*
* Private elements (won't appear at the uAPI book):
* @mmap_lock: private mutex used when buffers are allocated/freed/mmapped
* @memory: current memory type used
* @dma_dir: DMA mapping direction.
* @bufs: videobuf buffer structures
* @num_buffers: number of allocated/used buffers
* @queued_list: list of buffers currently queued from userspace
* @queued_count: number of buffers queued and ready for streaming.
* @owned_by_drv_count: number of buffers owned by the driver
* @done_list: list of buffers ready to be dequeued to userspace
* @done_lock: lock to protect done_list list
* @done_wq: waitqueue for processes waiting for buffers ready to be dequeued
* @streaming: current streaming state
* @start_streaming_called: @start_streaming was called successfully and we
* started streaming.
* @error: a fatal error occurred on the queue
* @waiting_for_buffers: used in poll() to check if vb2 is still waiting for
* buffers. Only set for capture queues if qbuf has not yet been
* called since poll() needs to return %EPOLLERR in that situation.
* @is_multiplanar: set if buffer type is multiplanar
* @is_output: set if buffer type is output
* @copy_timestamp: set if vb2-core should set timestamps
* @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
* last decoded buffer was already dequeued. Set for capture queues
* when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
* @fileio: file io emulator internal data, used only if emulator is active
* @threadio: thread io internal data, used only if thread is active
*/
struct vb2_queue {
unsigned int type; //v4l2_buf_type 枚举类型，表明设备的 buffer 类型
unsigned int io_modes; //vb2_io_modes 枚举类型，表明驱动支持哪一种streaming，实际应用中 VB2_MMAP 类型的居多
struct device *dev;
unsigned long dma_attrs;
unsigned bidirectional:1;
unsigned fileio_read_once:1; // 读取第一个缓冲区后报告EOF
unsigned fileio_write_immediately:1; // write写入的数据都添加到缓冲队列中
unsigned allow_zero_bytesused:1;
unsigned quirk_poll_must_check_waiting_for_buffers:1;
unsigned supports_requests:1;
unsigned requires_requests:1;
unsigned uses_qbuf:1;
unsigned uses_requests:1;
// 保护struct vb2_queue的互斥锁，使缓冲队列的操作串行化，
// 若驱动实有互斥锁，则可设置为NULL，videobuf2核心层API不使用此锁
// 可以与 video_device 共用一个 lock，
// 更推荐使用独立的 lock，独立的 lock 一定程度上可以减少锁竞争
struct mutex *lock;
void *owner;
const struct vb2_ops *ops; //指向 vb2_ops，驱动实现的回调函数
//根据 buffer 类型「物理连续、vmalloc、物理分散的 sg-dma
//选择 vb2_dma_contig_memops，vb2_dma_sg_memops,vb2_vmalloc_memops 三种之一，
//与我来说，最常用的就是第一个啦，当然也可以自己动手丰衣足食，不过常规使用那三个就能够满足了
const struct vb2_mem_ops *mem_ops;
const struct vb2_buf_ops *buf_ops;
// 一般指向驱动特定的结构体，也就是驱动自定义维护的结构体
void *drv_priv;
//自定义的 buffer 结构体大小，
//自定义的 buffer 结构体必须将 vb2_v4l2_buffer 成员放在第一个。
//可以在 vb2 的 queue 回调中通过 container_of
//来获取自定义的buffer结构体数据
unsigned int buf_struct_size;
//V4L2_BUF_FLAG_TIMESTAMP_MASK 或者 V4L2_BUF_FLAG_TSTAMP_SRC_MASK 类型的 flag 之一，
//两种 flag 可以是或的关系同时存在，表明时间戳的类型，
//一般使用 V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC 即可，
//表明是递增类型的，它是在内核的 monotonic 时钟时间轴上面生成的时间戳
u32 timestamp_flags;
// 分配缓冲区时的内存标志，通常为0，
// 也有使用GFP_DMA或__GFP_DMA32强制将内存分配到明确的内存区域
gfp_t gfp_flags;
// 在处理数据流之前，需要最小的缓冲区数目
u32 min_buffers_needed;
struct device *alloc_devs[VB2_MAX_PLANES];
/* private: internal use only */
// 私有锁，保护缓冲区的分配、释放、映射
struct mutex mmap_lock;
unsigned int memory;
enum dma_data_direction dma_dir;
// 保存分配缓冲区的地址
struct vb2_buffer *bufs[VB2_MAX_FRAME];
// 分配的缓冲区数量
unsigned int num_buffers;
// 用户空间入队的缓冲区链表
struct list_head queued_list;
// 入队的就绪缓冲区数量
unsigned int queued_count;
// 属于驱动的缓冲区数量
atomic_t owned_by_drv_count;
// 此链表中的缓冲区已填充数据，可以出队被用户空间使用
struct list_head done_list;
// 保护done_list链表的自旋锁
spinlock_t done_lock;
// 等待缓冲区出队的等待队列
wait_queue_head_t done_wq;
unsigned int streaming:1; // 当前流的状态
unsigned int start_streaming_called:1; // start_streaming()被成功调用
unsigned int error:1; // struct vb2_queue发生了致命错误
// 在poll函数中使用，以检查是否还在等待数据
unsigned int waiting_for_buffers:1;
unsigned int waiting_in_dqbuf:1;
......
};
该结构体的内容需驱动自行实现，以sun6i-csi为例，示例如下：
//源码：drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
/* Initialize videobuf2 queue */
vidq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
vidq->io_modes = VB2_MMAP | VB2_DMABUF;
vidq->drv_priv = video;
vidq->buf_struct_size = sizeof(struct sun6i_csi_buffer);
vidq->ops = &sun6i_csi_vb2_ops;
vidq->mem_ops = &vb2_dma_contig_memops;
vidq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
vidq->lock = &video->lock;
/* Make sure non-dropped frame */
vidq->min_buffers_needed = 3;
vidq->dev = csi->dev;
ret = vb2_queue_init(vidq);
if (ret) {
v4l2_err(&csi->v4l2_dev, "vb2_queue_init failed: %d\n", ret);
goto clean_entity;
}
上面一顿操作之后要记得使用 vb2_queue_init 对 vb2_queue 进一步初始化，里面主要是一大堆的 WARN_ON 来对我们的设置进行检查，获取 v4l2_buf_ops 这个 vb2_buf_ops 类型的 buffer 操作函数，并且初始化一些锁与一个等待队列，该等待队列主要是用来等待 buffer 变为可 dequeue 状态。
vb2_ops
驱动需要实现struct vb2_ops中的函数，当然也可以实现一部分，也可以直接使用内核提供的函数。
//源码：include/media/videobuf2-core.h
/**
* struct vb2_ops - driver-specific callbacks.
*
* These operations are not called from interrupt context except where
* mentioned specifically.
*
* @queue_setup: called from VIDIOC_REQBUFS() and VIDIOC_CREATE_BUFS()
* handlers before memory allocation. It can be called
* twice: if the original number of requested buffers
* could not be allocated, then it will be called a
* second time with the actually allocated number of
* buffers to verify if that is OK.
* The driver should return the required number of buffers
* in \*num_buffers, the required number of planes per
* buffer in \*num_planes, the size of each plane should be
* set in the sizes\[\] array and optional per-plane
* allocator specific device in the alloc_devs\[\] array.
* When called from VIDIOC_REQBUFS(), \*num_planes == 0,
* the driver has to use the currently configured format to
* determine the plane sizes and \*num_buffers is the total
* number of buffers that are being allocated. When called
* from VIDIOC_CREATE_BUFS(), \*num_planes != 0 and it
* describes the requested number of planes and sizes\[\]
* contains the requested plane sizes. In this case
* \*num_buffers are being allocated additionally to
* q->num_buffers. If either \*num_planes or the requested
* sizes are invalid callback must return %-EINVAL.
* @wait_prepare: release any locks taken while calling vb2 functions;
* it is called before an ioctl needs to wait for a new
* buffer to arrive; required to avoid a deadlock in
* blocking access type.
* @wait_finish: reacquire all locks released in the previous callback;
* required to continue operation after sleeping while
* waiting for a new buffer to arrive.
* @buf_out_validate: called when the output buffer is prepared or queued
* to a request; drivers can use this to validate
* userspace-provided information; this is required only
* for OUTPUT queues.
* @buf_init: called once after allocating a buffer (in MMAP case)
* or after acquiring a new USERPTR buffer; drivers may
* perform additional buffer-related initialization;
* initialization failure (return != 0) will prevent
* queue setup from completing successfully; optional.
* @buf_prepare: called every time the buffer is queued from userspace
* and from the VIDIOC_PREPARE_BUF() ioctl; drivers may
* perform any initialization required before each
* hardware operation in this callback; drivers can
* access/modify the buffer here as it is still synced for
* the CPU; drivers that support VIDIOC_CREATE_BUFS() must
* also validate the buffer size; if an error is returned,
* the buffer will not be queued in driver; optional.
* @buf_finish: called before every dequeue of the buffer back to
* userspace; the buffer is synced for the CPU, so drivers
* can access/modify the buffer contents; drivers may
* perform any operations required before userspace
* accesses the buffer; optional. The buffer state can be
* one of the following: %DONE and %ERROR occur while
* streaming is in progress, and the %PREPARED state occurs
* when the queue has been canceled and all pending
* buffers are being returned to their default %DEQUEUED
* state. Typically you only have to do something if the
* state is %VB2_BUF_STATE_DONE, since in all other cases
* the buffer contents will be ignored anyway.
* @buf_cleanup: called once before the buffer is freed; drivers may
* perform any additional cleanup; optional.
* @start_streaming: called once to enter 'streaming' state; the driver may
* receive buffers with @buf_queue callback
* before @start_streaming is called; the driver gets the
* number of already queued buffers in count parameter;
* driver can return an error if hardware fails, in that
* case all buffers that have been already given by
* the @buf_queue callback are to be returned by the driver
* by calling vb2_buffer_done() with %VB2_BUF_STATE_QUEUED.
* If you need a minimum number of buffers before you can
* start streaming, then set
* &vb2_queue->min_buffers_needed. If that is non-zero
* then @start_streaming won't be called until at least
* that many buffers have been queued up by userspace.
* @stop_streaming: called when 'streaming' state must be disabled; driver
* should stop any DMA transactions or wait until they
* finish and give back all buffers it got from &buf_queue
* callback by calling vb2_buffer_done() with either
* %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
* vb2_wait_for_all_buffers() function
* @buf_queue: passes buffer vb to the driver; driver may start
* hardware operation on this buffer; driver should give
* the buffer back by calling vb2_buffer_done() function;
* it is always called after calling VIDIOC_STREAMON()
* ioctl; might be called before @start_streaming callback
* if user pre-queued buffers before calling
* VIDIOC_STREAMON().
* @buf_request_complete: a buffer that was never queued to the driver but is
* associated with a queued request was canceled.
* The driver will have to mark associated objects in the
* request as completed; required if requests are
* supported.
*/
struct vb2_ops {
// 设置缓冲区队列相关参数
int (*queue_setup)(struct vb2_queue *q,
unsigned int *num_buffers, unsigned int *num_planes,
unsigned int sizes[], struct device *alloc_devs[]);
//释放所有在 ioctl 操作函数执行时被持有的锁；
//该回调在 ioctl 需要等待一个新的 buffer 到达的时候被调用；
//需要在阻塞访问的时候避免死锁。
void (*wait_prepare)(struct vb2_queue *q);
//请求所有的在上面 wait_prepare 回调锁释放的锁；
//需要在睡眠等待新的 buffer 到来之后继续运行。
void (*wait_finish)(struct vb2_queue *q);
int (*buf_out_validate)(struct vb2_buffer *vb);
//在 MMAP 方式下，分配完 buffer 或者 USERPTR 情况下请求完 buffer 之后被调用一次，
//（一个 buffer 调用一次）。如果该 ioctl 返回失败，将会导致 queue_setup 执行失败。
//该函数的主要目的是让我们在 buffer 申请完毕之后对 buffer 做一些初始化工作，但是实际上好像并不是经常用到。
//在该函数里面可以获取到 buffer 的 type，memory 类型，index，planesnum 等
int (*buf_init)(struct vb2_buffer *vb);
//每次 buffer 重新入队「就是在用户调用 QBUF 的时候」以及 VIDIOC_PREPARE_BUF 操作的时候被调用，
//驱动可以做一些硬件操作「通常数据都是由硬件产生的」之前的初始化工作。
//如果该回调返回失败，那么 buffer 将不会执行 queue 动作。
//一般在这里需要设置 plane 的 payload，也就是每个 plane 的使用内存长度，
//单位为 Byte，可以使用 vb2_set_plane_payload 来帮助完成
int (*buf_prepare)(struct vb2_buffer *vb);
//每次 buffer 被取出的时候被调用，
//并且是在 buffer 到达用户空间之前，
//所以驱动可以访问/修改 buffer 的内容。
void (*buf_finish)(struct vb2_buffer *vb);
//buffer 被释放之前调用一次，
//每个 buffer 仅有一次，驱动可以在这里面做一些额外的操作。
void (*buf_cleanup)(struct vb2_buffer *vb);
//进入 streaming 状态时被调用一次，一般情况下，
//驱动在该回调函数执行之前，通过 buf_queue 回调来接收 buffer，
//这里驱动需要把 buffer 放到驱动自己维护的一个 buffer 队列里面。
//count 参数存放已经被 queue 的 buffer 数量，驱动可以由此获取它。
//如果发生硬件错误，驱动可以通过该回调返回一个错误，
//此时所有的buffer都会被归还给 videobuf2（调用 vb2_buffer_done(VB2_BUF_STATE_QUEUED)）。
//如果需要设置开始 start_streaming 需要的 buffer 最小数量，
//比如驱动只有在满足 buffer 数量大于等于 3 的时候才能够开启数据流，
//那么就可以在该函数被调用之前设置 vb2_queue->min_buffers_needed 成员为自己想要的值，
//此时该回调函数会在满足最小 buffer 数量之后才被调用。
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
//在 streaming 被禁止的时候调用，驱动需要关掉 DMA 或者等待 DMA 结束，
//调用 vb2_buffer_done 来归还所有驱动持有的 buffers
//（参数使用 VB2_BUF_STATE_DONE 或者 VB2_BUF_STATE_ERR），
//可能需要用到 vb2_wait_for_all_buffers 来等待所有的 buffer，
//该函数是用来等待所有的 buffer 被归还给 videobuf2 的。
void (*stop_streaming)(struct vb2_queue *q);
//用来传递 vb（vb2_buffer 结构体） 给驱动，
//驱动可以在这里开启硬件操作（DMA 等等）。
//驱动填充 buffer 完毕之后需要调用 vb2_buffer_done 归还 buffer，
//该函数总是在 VIDIOC_STREAMON 操作之后调用。
//但是也有可能在 VIDIOC_STREAMON 之前被调用，
//比如用户空间中 querybuf 之后的 queue 操作。
//videobuf2 允许自定义一个 buffer 结构体而不是直接使用 vb2_buffer，
//但是自定义的结构体的第一个成员必须是 vb2_v4l2_buffer 结构体，
//该结构体的大小需要在 vb2_queue 的 buf_struct_size 成员中指定。
void (*buf_queue)(struct vb2_buffer *vb);
void (*buf_request_complete)(struct vb2_buffer *vb);
};
queue_setup 由ioctl命令 VIDIOC_REQBUFS 和 VIDIOC_CREATE_BUFS 调用，经常使用的是 VIDIOC_REQBUFS ，在实际分配内存之前调用一次，若分配的 buffer数量 不足 num_buffers，则会再次调用。该函数需要将分配的 buffer数量 保存到 num_buffers，将 buffer的planes的数量 保存到 num_planes 中，每个 plans的大小 存放在 sizes数组 中。planes和视频的像素格式有关，如YUV420SP格式的 planes 为2。alloc_ctxs数组 保存每一个 plane的特定数据。
wait_prepare 和 wait_finish 函数内核提供了默认的实现，可以直接使用，分别对应函数 vb2_ops_wait_prepare 和 vb2_ops_wait_finish 。这两个函数实现很简单，vb2_ops_wait_prepare 释放互斥锁 vb2_ops_wait_finish 获取互斥锁。用户调用ioctl并使用 VIDIOC_QBUF 命令时，内核会判断是否是阻塞调用，如果是阻塞调用并且没有准备好数据，内核此时会调用 wait_prepare 释放锁并进行休眠等待，直到数据准备好被唤醒，然后再调用 wait_finish 重新持有锁。
buf_init 在分配缓冲区之后调用或获取了新的 USERPTR 之后调用(in MMAP case)，驱动需要完成一些缓冲区初始化的工作，若初始化失败，则返回不为0的数，此时 queue_setup 将失败，一般用不到。
buf_prepare 缓冲区每次从用户空间入队都需要调用或被ioctl的 VIDIOC_PREPARE_BUF 命令调用，驱动需要执行一些初始化工作或获取、修改缓冲区，若驱动支持 VIDIOC_CREATE_BUFS，则需要验证缓冲区的大小，若有错误发生，则缓冲区不会入队。
start_streaming 调用后流进入开启状态，在调用之前驱动必须先调用 buf_queue 接收缓冲区。如果发生硬件错误，驱动可以通过该回调返回一个错误，此时所有的 buffer 都会被归还给 videobuf2 （调用vb2_buffer_done(*vb, VB2_BUF_STATE_QUEUED) ）。如果需要设置 start_streaming 时 buffer的最小数量 ，那么应该在调用该函数之前设置最少的buffer数量值 vb2_queue->min_buffers_needed ，只有 buffer数量 大于 vb2_queue->min_buffers_needed 时 start_streaming 才能被成功调用。
stop_streaming 调用后流进入关闭状态，驱动需要停止 DMA传输 或 等待工作完成 和 归还全部buffers（还给videobuf2） 。调用 vb2_buffer_done 来归还所有驱动持有的buffers，可使用 VB2_BUF_STATE_DONE（操作完成）和 VB2_BUF_STATE_ERR（操作出错） 参数。若要等待完成，可使用 vb2_wait_for_all_buffers。
该结构体的内容可有驱动实现部分，另一部分直接使用内核提供的接口函数，以sun6i-csi为例，示例如下：
//源码：drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
static const struct vb2_ops sun6i_csi_vb2_ops = {
.queue_setup = sun6i_video_queue_setup,
.wait_prepare = vb2_ops_wait_prepare,
.wait_finish = vb2_ops_wait_finish,
.buf_prepare = sun6i_video_buffer_prepare,
.start_streaming = sun6i_video_start_streaming,
.stop_streaming = sun6i_video_stop_streaming,
.buf_queue = sun6i_video_buffer_queue,
};
vb2_buffer
struct vb2_buffer 是保存视频数据和信息的核心结构体，每一帧的图像都对应一个 struct vb2_buffer 结构体，图像信息保存在 struct v4l2_buffer 结构中，如时间戳、编号、序列号等信息。
应用使用ioctl的 VIDIOC_REQBUFS 命令请求缓冲区时，分配 struct vb2_buffer。
struct vb2_buffer 的状态由 enum vb2_buffer_state 枚举定义描述。
若是 V4L2_MEMORY_MMAP 类型，则会额外分配内存，图像数据则保存在额外分配的内存中，额外分配的内存指针保存在 planes[VIDEO_MAX_PLANES] 数组中。
缓冲区枚举状态：
//源码：include/media/videobuf2-core.h
/**
* enum vb2_buffer_state - current video buffer state.
* @VB2_BUF_STATE_DEQUEUED: buffer under userspace control.
* @VB2_BUF_STATE_IN_REQUEST: buffer is queued in media request.
* @VB2_BUF_STATE_PREPARING: buffer is being prepared in videobuf.
* @VB2_BUF_STATE_QUEUED: buffer queued in videobuf, but not in driver.
* @VB2_BUF_STATE_ACTIVE: buffer queued in driver and possibly used
* in a hardware operation.
* @VB2_BUF_STATE_DONE: buffer returned from driver to videobuf, but
* not yet dequeued to userspace.
* @VB2_BUF_STATE_ERROR: same as above, but the operation on the buffer
* has ended with an error, which will be reported
* to the userspace when it is dequeued.
*/
enum vb2_buffer_state { // 缓冲区状态枚举
VB2_BUF_STATE_DEQUEUED, // 缓冲区出队，处于用户空间的控制下，默认状态
VB2_BUF_STATE_IN_REQUEST, // 缓冲区在排队申请中
VB2_BUF_STATE_PREPARING, // videobuf2正在准备缓冲区
VB2_BUF_STATE_QUEUED, // 缓冲区入队，处于videobuf2中，不处于驱动中
VB2_BUF_STATE_ACTIVE, // 缓冲区位于驱动中
VB2_BUF_STATE_DONE, // 缓冲区从驱动返回到videobuf2，但还没出队到用户空间
VB2_BUF_STATE_ERROR, // 出错，dequeued到用户空间会报错
};
vb2_buffer定义：
//源码：include/media/videobuf2-core.h
/**
* struct vb2_buffer - represents a video buffer.
* @vb2_queue: pointer to &struct vb2_queue with the queue to
* which this driver belongs.
* @index: id number of the buffer.
* @type: buffer type.
* @memory: the method, in which the actual data is passed.
* @num_planes: number of planes in the buffer
* on an internal driver queue.
* @timestamp: frame timestamp in ns.
* @request: the request this buffer is associated with.
* @req_obj: used to bind this buffer to a request. This
* request object has a refcount.
*/
struct vb2_buffer { //这个是设备内部帧缓存描述，v4l2_buffer是用户态
struct vb2_queue *vb2_queue; //包含外部指针
unsigned int index; //buf的编号
unsigned int type;
unsigned int memory;
unsigned int num_planes; // 该buffer有多少个planes
u64 timestamp;
struct media_request *request;
struct media_request_object req_obj;
/* private: internal use only
*
* state: current buffer state; do not change
* synced: this buffer has been synced for DMA, i.e. the
* 'prepare' memop was called. It is cleared again
* after the 'finish' memop is called.
* prepared: this buffer has been prepared, i.e. the
* buf_prepare op was called. It is cleared again
* after the 'buf_finish' op is called.
* copied_timestamp: the timestamp of this capture buffer was copied
* from an output buffer.
* queued_entry: entry on the queued buffers list, which holds
* all buffers queued from userspace
* done_entry: entry on the list that stores all buffers ready
* to be dequeued to userspace
* vb2_plane: per-plane information; do not change
*/
enum vb2_buffer_state state;//buffer的状态，buffer轮转的时候要设置
unsigned int synced:1;
unsigned int prepared:1;
unsigned int copied_timestamp:1;
// 私有的per-plane信息，驱动禁止修改
struct vb2_plane planes[VB2_MAX_PLANES];
// queued buffer链表，保存所有从userspace queued进的buffers
struct list_head queued_entry;
// 保存所有准备dequeued到userspace的buffers链表
struct list_head done_entry;
......
};
vb2_plane：
//源码：include/media/videobuf2-core.h
/**
* struct vb2_plane - plane information.
* @mem_priv: private data with this plane.
* @dbuf: dma_buf - shared buffer object.
* @dbuf_mapped: flag to show whether dbuf is mapped or not
* @bytesused: number of bytes occupied by data in the plane (payload).
* @length: size of this plane (NOT the payload) in bytes.
* @min_length: minimum required size of this plane (NOT the payload) in bytes.
* @length is always greater or equal to @min_length.
* @m: Union with memtype-specific data.
* @m.offset: when memory in the associated struct vb2_buffer is
* %VB2_MEMORY_MMAP, equals the offset from the start of
* the device memory for this plane (or is a "cookie" that
* should be passed to mmap() called on the video node).
* @m.userptr: when memory is %VB2_MEMORY_USERPTR, a userspace pointer
* pointing to this plane.
* @m.fd: when memory is %VB2_MEMORY_DMABUF, a userspace file
* descriptor associated with this plane.
* @data_offset: offset in the plane to the start of data; usually 0,
* unless there is a header in front of the data.
*
* Should contain enough information to be able to cover all the fields
* of &struct v4l2_plane at videodev2.h.
*/
struct vb2_plane {
void *mem_priv; // 存放一帧图片数据(针对MMAP类型模式)
struct dma_buf *dbuf; // (针对DMA类型模式)
unsigned int dbuf_mapped;// 标记是否映射
unsigned int bytesused; // 占用大小
unsigned int length; // plane大小
unsigned int min_length;
union {
unsigned int offset; // mmap记录偏移
unsigned long userptr; // 针对userptr模式
int fd; // dmabuf，记录fd
} m;
unsigned int data_offset;// 这个plane中数据开始的偏移值
};
v4l2_buffer：
struct v4l2_buffer用来指定与描述一帧帧缓冲，应用可以设置
//源码：include/uapi/linux/videodev2.h
/**
* struct v4l2_buffer - video buffer info
* @index: id number of the buffer
* @type: enum v4l2_buf_type; buffer type (type == *_MPLANE for
* multiplanar buffers);
* @bytesused: number of bytes occupied by data in the buffer (payload);
* unused (set to 0) for multiplanar buffers
* @flags: buffer informational flags
* @field: enum v4l2_field; field order of the image in the buffer
* @timestamp: frame timestamp
* @timecode: frame timecode
* @sequence: sequence count of this frame
* @memory: enum v4l2_memory; the method, in which the actual video data is
* passed
* @offset: for non-multiplanar buffers with memory == V4L2_MEMORY_MMAP;
* offset from the start of the device memory for this plane,
* (or a "cookie" that should be passed to mmap() as offset)
* @userptr: for non-multiplanar buffers with memory == V4L2_MEMORY_USERPTR;
* a userspace pointer pointing to this buffer
* @fd: for non-multiplanar buffers with memory == V4L2_MEMORY_DMABUF;
* a userspace file descriptor associated with this buffer
* @planes: for multiplanar buffers; userspace pointer to the array of plane
* info structs for this buffer
* @length: size in bytes of the buffer (NOT its payload) for single-plane
* buffers (when type != *_MPLANE); number of elements in the
* planes array for multi-plane buffers
* @request_fd: fd of the request that this buffer should use
*
* Contains data exchanged by application and driver using one of the Streaming
* I/O methods.
*/
// struct v4l2_buffer用来指定与描述一帧帧缓冲，应用可以设置
struct v4l2_buffer {
__u32 index; // buffer的编号
__u32 type; // buffer的类型，由enum v4l2_buf_type定义
// 数据在缓冲区(有效负载)中所占的字节数，对于多平面缓冲区未使用(设置为0)
__u32 bytesused;
// 标志位，见V4L2_BUF_FLAG_XX宏定义，常见值有:
// V4L2_BUF_FLAG_MAPPED —— 代表当前缓存已经映射
// V4L2_BUF_FLAG_QUEUED —— 缓存可以采集数据
// V4L2_BUF_FLAG_DONE —— 缓存可以提取数据
__u32 flags;
__u32 field;
struct timeval timestamp; // 视频帧时间戳
struct v4l2_timecode timecode; // 时间码
__u32 sequence; // 该帧的序列号
/* memory location */
__u32 memory; // enum v4l2_memory枚举定义
union {
// V4L2_MEMORY_MMAP，从将要mapping的device memory头到数据头的offset
__u32 offset;
// V4L2_MEMORY_USERPTR，用户空间指针指向此buffer
unsigned long userptr;
// for multiplanar buffers; userspace pointer to the array of plane
// info structs for this buffer
struct v4l2_plane *planes;
// V4L2_MEMORY_DMABUF，用户空间的描述符关联此描述符
__s32 fd;
} m;
// 对于single-plane，表示buffer的字节数
// 对于multi-plane buffers，则表示planes array中的元素数量
__u32 length;
__u32 reserved2;
union {
__s32 request_fd;
__u32 reserved;
};
};
vb2_mem_ops
struct vb2_mem_ops 是 buffer内存 分配 和 处理 的操作函数集合，这些函数和 buffer 的类型有关系，即 enum v4l2_memory 枚举定义的类型。具体如下。
（1）get_userptr 和 put_userptr 函数用于处理 USERPTR类型 的buffer。
（2）alloc、put、num_users 和 mmap 函数用于处理 MMAP类型 的buffer。
（3）alloc、put、num_users 和 vaddr 函数用于处理 read/write 访问类型的buffer。
（4）attach_dmabuf、detach_dmabuf、map_dmabuf 和 unmap_dmabuf 函数用于处理 DMABUF类型 的buffer。
//源码：include/media/videobuf2-core.h
/**
* struct vb2_mem_ops - memory handling/memory allocator operations.
* @alloc: allocate video memory and, optionally, allocator private data,
* return ERR_PTR() on failure or a pointer to allocator private,
* per-buffer data on success; the returned private structure
* will then be passed as @buf_priv argument to other ops in this
* structure. Additional gfp_flags to use when allocating the
* are also passed to this operation. These flags are from the
* gfp_flags field of vb2_queue. The size argument to this function
* shall be *page aligned*.
* @put: inform the allocator that the buffer will no longer be used;
* usually will result in the allocator freeing the buffer (if
* no other users of this buffer are present); the @buf_priv
* argument is the allocator private per-buffer structure
* previously returned from the alloc callback.
* @get_dmabuf: acquire userspace memory for a hardware operation; used for
* DMABUF memory types.
* @get_userptr: acquire userspace memory for a hardware operation; used for
* USERPTR memory types; vaddr is the address passed to the
* videobuf layer when queuing a video buffer of USERPTR type;
* should return an allocator private per-buffer structure
* associated with the buffer on success, ERR_PTR() on failure;
* the returned private structure will then be passed as @buf_priv
* argument to other ops in this structure.
* @put_userptr: inform the allocator that a USERPTR buffer will no longer
* be used.
* @attach_dmabuf: attach a shared &struct dma_buf for a hardware operation;
* used for DMABUF memory types; dev is the alloc device
* dbuf is the shared dma_buf; returns ERR_PTR() on failure;
* allocator private per-buffer structure on success;
* this needs to be used for further accesses to the buffer.
* @detach_dmabuf: inform the exporter of the buffer that the current DMABUF
* buffer is no longer used; the @buf_priv argument is the
* allocator private per-buffer structure previously returned
* from the attach_dmabuf callback.
* @map_dmabuf: request for access to the dmabuf from allocator; the allocator
* of dmabuf is informed that this driver is going to use the
* dmabuf.
* @unmap_dmabuf: releases access control to the dmabuf - allocator is notified
* that this driver is done using the dmabuf for now.
* @prepare: called every time the buffer is passed from userspace to the
* driver, useful for cache synchronisation, optional.
* @finish: called every time the buffer is passed back from the driver
* to the userspace, also optional.
* @vaddr: return a kernel virtual address to a given memory buffer
* associated with the passed private structure or NULL if no
* such mapping exists.
* @cookie: return allocator specific cookie for a given memory buffer
* associated with the passed private structure or NULL if not
* available.
* @num_users: return the current number of users of a memory buffer;
* return 1 if the videobuf layer (or actually the driver using
* it) is the only user.
* @mmap: setup a userspace mapping for a given memory buffer under
* the provided virtual memory region.
*
* Those operations are used by the videobuf2 core to implement the memory
* handling/memory allocators for each type of supported streaming I/O method.
*
* .. note::
* #) Required ops for USERPTR types: get_userptr, put_userptr.
*
* #) Required ops for MMAP types: alloc, put, num_users, mmap.
*
* #) Required ops for read/write access types: alloc, put, num_users, vaddr.
*
* #) Required ops for DMABUF types: attach_dmabuf, detach_dmabuf,
* map_dmabuf, unmap_dmabuf.
*/
struct vb2_mem_ops {
// 分配video内存和私有数据（可选）分配器
void *(*alloc)(struct device *dev, unsigned long attrs,
unsigned long size,
enum dma_data_direction dma_dir,
gfp_t gfp_flags);
// 告诉分配器此缓冲区不再使用，若没有其他使用者使用，则分配器会释放此块内存
void (*put)(void *buf_priv);
// 获取DMABUF句柄，在V4L2_MEMORY_DMABUF模式中使用
struct dma_buf *(*get_dmabuf)(void *buf_priv, unsigned long flags);
// 获取用户空间指针指向的内存，在V4L2_MEMORY_USERPTR模式中使用
void *(*get_userptr)(struct device *dev, unsigned long vaddr,
unsigned long size,
enum dma_data_direction dma_dir);
// 告诉分配器USERPTR缓冲区不再使用
void (*put_userptr)(void *buf_priv);
// 缓冲区每次从用户空间添加到队列中就会被调用，对缓存同步很有用
void (*prepare)(void *buf_priv);
// 缓冲区每次从内核队列添加到用户空间就会被调用
void (*finish)(void *buf_priv);
// 为硬件操作添加共享的struct dma_buf，在V4L2_MEMORY_DMABUF模式中使用
// alloc_ctx-分配上下文，dbuf-共享的dma_buf
void *(*attach_dmabuf)(struct device *dev,
struct dma_buf *dbuf,
unsigned long size,
enum dma_data_direction dma_dir);
// 通知缓冲区的exporter目前的DMABUF不再使用
void (*detach_dmabuf)(void *buf_priv);
// 从分配器请求访问DMABUF，此DMABUF的分配器将通知驱动该DMABUF将要被使用
int (*map_dmabuf)(void *buf_priv);
// 释放访问DMABUF的控制权，此DMABUF的分配器将通知驱动该DMABUF已经使用完毕
void (*unmap_dmabuf)(void *buf_priv);
// 返回给定缓冲区的内核虚拟地址，该缓冲区与私有数据结构向关联
void *(*vaddr)(void *buf_priv);
// 返回给定缓冲区的分配器定义的cookie
void *(*cookie)(void *buf_priv);
// 返回此缓冲区的当前使用者，若只有videobuf2层使用，则返回1
unsigned int (*num_users)(void *buf_priv);
// 建立用户空间到给定缓冲区虚拟地址区域的映射
int (*mmap)(void *buf_priv, struct vm_area_struct *vma);
};
实时上，Linux内核中有3中不同类型的videobuf2：
（1）缓冲区物理地址和虚拟地址不连续——vb2_dma_sg_memops
//源码： drivers/media/common/videobuf2/videobuf2-dma-sg.c
const struct vb2_mem_ops vb2_dma_sg_memops = {
.alloc = vb2_dma_sg_alloc,
.put = vb2_dma_sg_put,
.get_userptr = vb2_dma_sg_get_userptr,
.put_userptr = vb2_dma_sg_put_userptr,
.prepare = vb2_dma_sg_prepare,
.finish = vb2_dma_sg_finish,
.vaddr = vb2_dma_sg_vaddr,
.mmap = vb2_dma_sg_mmap,
.num_users = vb2_dma_sg_num_users,
.get_dmabuf = vb2_dma_sg_get_dmabuf,
.map_dmabuf = vb2_dma_sg_map_dmabuf,
.unmap_dmabuf = vb2_dma_sg_unmap_dmabuf,
.attach_dmabuf = vb2_dma_sg_attach_dmabuf,
.detach_dmabuf = vb2_dma_sg_detach_dmabuf,
.cookie = vb2_dma_sg_cookie,
};
EXPORT_SYMBOL_GPL(vb2_dma_sg_memops);
（2）缓冲区物理地址不连续但虚拟地址连续——vb2_vmalloc_memops
//源码： drivers/media/common/videobuf2/videobuf2-vmalloc.c
const struct vb2_mem_ops vb2_vmalloc_memops = {
.alloc = vb2_vmalloc_alloc,
.put = vb2_vmalloc_put,
.get_userptr = vb2_vmalloc_get_userptr,
.put_userptr = vb2_vmalloc_put_userptr,
#ifdef CONFIG_HAS_DMA
.get_dmabuf = vb2_vmalloc_get_dmabuf,
#endif
.map_dmabuf = vb2_vmalloc_map_dmabuf,
.unmap_dmabuf = vb2_vmalloc_unmap_dmabuf,
.attach_dmabuf = vb2_vmalloc_attach_dmabuf,
.detach_dmabuf = vb2_vmalloc_detach_dmabuf,
.vaddr = vb2_vmalloc_vaddr,
.mmap = vb2_vmalloc_mmap,
.num_users = vb2_vmalloc_num_users,
};
EXPORT_SYMBOL_GPL(vb2_vmalloc_memops);
（3）缓冲区物理地址和虚拟地址都连续——vb2_dma_contig_memops
//源码： drivers/media/common/videobuf2/videobuf2-dma-contig.c
const struct vb2_mem_ops vb2_dma_contig_memops = {
.alloc = vb2_dc_alloc,
.put = vb2_dc_put,
.get_dmabuf = vb2_dc_get_dmabuf,
.cookie = vb2_dc_cookie,
.vaddr = vb2_dc_vaddr,
.mmap = vb2_dc_mmap,
.get_userptr = vb2_dc_get_userptr,
.put_userptr = vb2_dc_put_userptr,
.prepare = vb2_dc_prepare,
.finish = vb2_dc_finish,
.map_dmabuf = vb2_dc_map_dmabuf,
.unmap_dmabuf = vb2_dc_unmap_dmabuf,
.attach_dmabuf = vb2_dc_attach_dmabuf,
.detach_dmabuf = vb2_dc_detach_dmabuf,
.num_users = vb2_dc_num_users,
};
EXPORT_SYMBOL_GPL(vb2_dma_contig_memops);
234
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
