v4l2文档之——media framework

Linux kernel media framework
============================
 
This document describes the Linux kernel media framework, its data structures,
functions and their usage.

 Linux 内核多媒体驱动框架
 这个文档描述linux内核多媒体驱动框架，包括它的数据结构、函数以及它们的用法。

Introduction
------------
 
The media controller API is documented in DocBook format in
Documentation/DocBook/v4l/media-controller.xml. This document will focus on
the kernel-side implementation of the media framework.
 
一、 简介
Media 控制器API已记录在Docbook格式的Documentation/DocBook/v4l/media-controller.xml文件中，这个文档主要描述内核中media 框架的实现。

Abstract media device model
---------------------------
Discovering a device internal topology, and configuring it at runtime, is one of the goals of the media framework. To achieve this, hardware devices are modeled as an oriented graph of building blocks called entities connected through pads.
An entity is a basic media hardware building block. It can correspond to a large variety of logical blocks such as physical hardware devices (CMOS sensor for instance), logical hardware devices (a building block in a System-on-Chip image processing pipeline), DMA channels or physical connectors.
  A pad is a connection endpoint through which an entity can interact with other entities. Data (not restricted to video) produced by an entity flows from the entity's output to one or more entity inputs. Pads should not be confused with physical pins at chip boundaries.
A link is a point-to-point oriented connection between two pads, either on the same entity or on different entities. Data flows from a source pad to a sink pad.
二、  Media 设备模型摘要
发现一个设备内部的拓扑图，并在运行时配置它，是整个media框架的一部分。为了实现这个，硬件设备被建模为一个导向图表的构建快，并称其为通过pad连接的实体。
entity是一个基本的media硬件构件块。它可以和许多逻辑块通信，例如物理硬件设备CMOS摄像头实例、逻辑硬件设备片上图像处理通道构件、DMA通道或者物理连接器。
Pad是一个连接端点，通过它可以使一个entity和其他entities进行通信。通过entity产生的数据（不仅仅是视频数据）从entity的输出端流到entity的一个或者多个输入端。Pad在芯片中不需要配置物理的引脚。
 Link是在两个pads之前点到点的定向连接件，这种连接可以是在相同或不同的entities上。数据从source pad流到sink pad。
Media device
------------
A media device is represented by a struct media_device instance, defined in include/media/media-device.h. Allocation of the structure is handled by the media device driver,usually by embedding the media_device instance in a larger driver-specific structure.
Drivers register media device instances by calling
         media_device_register(struct media_device *mdev);
 
The caller is responsible for initializing the media_device structure before registration. The following fields must be set:
  - dev : must point to the parent device (usually a pci_dev, usb_interface orplatform_device instance).
  - model : must be filled with the device model name as a NUL-terminated UTF-8 string. The device/model revision must not be stored in this field.

The following fields are optional:
  - serial is a unique serial number stored as a NUL-terminated ASCII string. The field is big enough to store a GUID in text form. If the hardware doesn't provide a unique serial number this field must be left empty.
  - bus_info represents the location of the device in the system as a NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices, the usb_make_path() function must be used. This field is used by applications to distinguish between otherwise identical devices that don't provide a serial number.
  - hw_revision is the hardware device revision in a driver-specific format. When possible the revision should be formatted with the KERNEL_VERSION macro.
  - driver_version is formatted with the KERNEL_VERSION macro. The version minor must be incremented when new features are added to the userspace API without breaking binary compatibility. The version major must be incremented when binary compatibility is broken.
Upon successful registration a character device named media[0-9]+ is created. The device major and minor numbers are dynamic. The model name is exported as a sysfs attribute.
Drivers unregister media device instances by calling
         media_device_unregister(struct media_device *mdev);
Unregistering a media device that hasn't been registered is *NOT* safe.
 

三、 Meida 设备

通过include/media/media-device.h中的结构体来代表一个media设备。Media设备驱动来处理该结构体的分配工作，通常情况下，media_device实例被嵌套在一个大的驱动特定结构体中。

1、media设备注册
驱动通过调用下边函数注册media 设备实例：
         media_device_register(struct media_device *mdev);

2、media_device结构体成员分析
在注册之前，调用者负责初始化media_device结构体，下边的成员必须设置：
-dev：必须指向父设备（通常是pci_dev, usb_interface or platform_device 实例）
-model：必须设置为设备模型的名称，是一个空终止的UTF-8字符串。设备或模型的版本号不能存储在该元素中。
下边的成员是可选的：
-serial是一个独特的连续序号，被保存成NUL-terminated ASCII格式的字符串，这个成员很大，足够以文本格式保存GUID。如果硬件不提供独特的连续序号，这个成员将被暂时设置为空。
-bus_info用一个NUL-terminated ASCII的字符串来代表这个设备在系统中的位置，例如PCI/PCIe设备bus_info必须设置为“PCI:”或者“PCIe:”，后边紧跟pci_name（）的值；或者USB设备，usb_make_path()函数必须被用。这个成员通常被应用程序来区分其他没有提供连续序号的相同设备。
-hw_revision是一个用驱动指定的格式来呈现硬件设备版本号，可能的版本号将通过KERNEL_VERSION宏来格式化。
-deriver_version被KERNEL_VERSION宏格式化。在不破坏二进制兼容性的情况下，当新的特性被添加在用户空间API，版本子设备号必须增加。当二进制兼容性被破坏时，版本的主设备号也必须增加。
成功注册一个名称为media[0-9]的字符设备被创建。设备的主设备号和从设备号会自动分配，模型的名称作为一个sysfs的属性被导出。
通过下面的函数注销media 设备：
 media_device_unregister(struct media_device *mdev);
注销一个没被注册过的media设备是安全的。

Entities, pads and links
------------------------
 
- Entities
 
Entities are represented by a struct media_entity instance, defined in include/media/media-entity.h. The structure is usually embedded into a higher-level structure, such as a v4l2_subdev or video_device instance, although drivers can allocate entities directly.
Drivers initialize entities by calling
         media_entity_init(struct media_entity *entity, u16 num_pads,
                           struct media_pad *pads, u16 extra_links);
  The media_entity name, type, flags, revision and group_id fields can be initialized before or after calling media_entity_init. Entities embedded in higher-level standard structures can have some of those fields set by the higher-level framework.
As the number of pads is known in advance, the pads array is not allocated dynamically but is managed by the entity driver. Most drivers will embed the pads array in a driver-specific structure, avoiding dynamic allocation.
  Drivers must set the direction of every pad in the pads array before calling media_entity_init. The function will initialize the other pads fields.
  Unlike the number of pads, the total number of links isn't always known in advance by the entity driver. As an initial estimate, media_entity_init pre-allocates a number of links equal to the number of pads plus an optional number of extra links. The links array will be reallocated if it grows beyond the initial estimate.
Drivers register entities with a media device by calling
         media_device_register_entity(struct media_device *mdev,
                                      struct media_entity *entity);
 
  Entities are identified by a unique positive integer ID. Drivers can provide an ID by filling the media_entity id field prior to registration, or request the media controller framework to assign an ID automatically. Drivers that provide IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be contiguous even when they are all assigned automatically by the framework.
Drivers unregister entities by calling
         media_device_unregister_entity(struct media_entity *entity);
 
  Unregistering an entity will not change the IDs of the other entities, and the ID will never be reused for a newly registered entity.
 When a media device is unregistered, all its entities are unregistered automatically. No manual entities unregistration is then required.
 Drivers free resources associated with an entity by calling
         media_entity_cleanup(struct media_entity *entity);
 
  This function must be called during the cleanup phase after unregistering the entity. Note that the media_entity instance itself must be freed explicitly by the driver if required.
 Entities have flags that describe the entity capabilities and state.
         MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
         This can be used to report the default audio and video devices or the
         default camera sensor.
Logical entity groups can be defined by setting the group ID of all member entities to the same non-zero value. An entity group serves no purpose in the kernel, but is reported to userspace during entities enumeration. The group_id field belongs to the media device driver and must not by touched by entity drivers.
Media device drivers should define groups if several entities are logically bound together. Example usages include reporting
 
         - ALSA, VBI and video nodes that carry the same media stream
         - lens and flash controllers associated with a sensor
四、 Entity
Entities被include/media/media-entity.h文件中定义的结构体media_entity实例代表，通常嵌在一个更大的结构体中，例如v4l2_subdev或者video_device实例，尽管驱动可以直接分配entities。
1、entity初始化
通过下列函数初始化entities：
         media_entity_init(struct media_entity *entity, u16 num_pads,
                           struct media_pad *pads, u16 extra_links);
Media_entity的名称、类型、标记、版本号和组ID成员可以在调用media_entity_init()函数之前或之后被初始化。可以通过更高级的框架来设置包含entities的更高级标准结构体的其他成员。
 因为pads的数量是预先知道的，pads队列不是被自动分配，但是它们被entity驱动管理。大部分驱动将内嵌pads 队列到驱动特定的结构体，避免自动分配。
 在调用media_entity_init函数之前，驱动必须设置pads队列中每个pad的方向。这个函数将初始化pads的其他成员。
 不同于pads的数量，不能通过entity驱动预先确定所有links的数量。作为初步估计，media_entity_init函数预先分配了和pads数量相同的links，还分配了一些可选择数量的额外links。如果在初始判断之外有新的links，那么links队列将重新初始化。

2、media设备中entity注册
Media 设备通过调用下边的函数注册media entities:
         media_device_register_entity(struct media_device *mdev,
                                      struct media_entity *entity);

 Entities通过唯一的整数ID来识别。在之前注册时通过驱动提供的ID来设置media_entity的id成员，或者通过media 控制器框架来自动分配ID。必须保证驱动手动提供的IDs都是唯一的，当它们被框架自动分配时不能保证IDs是连续的。
驱动通过下边的函数注销entities：
         media_device_unregister_entity(struct media_entity *entity);
 
注销一个entity将不能改变其他entities的IDs，这个ID将不能被新注册的entity使用。
当一个media设备被注销时，所有的entities将被自动注销，没有必要手动注销entities。
驱动通过调用下边函数释放分配的entity资源：
         media_entity_cleanup(struct media_entity *entity);
 
这个函数必须在注销entity之后的清楚阶段被调用。注意，如果需要， media_entity实例自己必须被明确的释放。

3、entity的一些属性
Entities用flags来描述entity的功能和状态：
 MEDIA_ENT_FL_DEFAULT表明是entity是当前默认的。这个可以用来报告默认的audio、video设备和camera传感器。
 通过设置所有entities成员组的ID为相同的非零值来定义逻辑entity组。在内核空间一个entity组没有任何作用，但是在entities枚举器件时会被报告到用户空间。Group_id成员属于media设备驱动，它不能被entity驱动操作。
 如果几个entities合乎一定的逻辑范围，media设备驱动会把他们定义为一组，比如ALSA、VBI 和视频节点可以传送相同的多媒体数据流，镜头和闪光控制器与传感器有关。

 - Pads
 
  Pads are represented by a struct media_pad instance, defined in include/media/media-entity.h. Each entity stores its pads in a pads array managed by the entity driver. Drivers usually embed the array in a driver-specific structure.
Pads are identified by their entity and their 0-based index in the pads array. Both information are stored in the media_pad structure, making the media_pad pointer the canonical way to store and pass link references.
Pads have flags that describe the pad capabilities and state.
         MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
         MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.
cortexOne and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set foreach pad.
 

五、Pad
Pads通过定义在include/media/media-entity.h文件中的结构体media_pad实例来表现。每个entity保存它自己的pads在pads队列中被entity驱动管理。entity驱动通常内嵌pads队列到驱动特定的结构体中。
 Pads通过它们的entity和它们在pads队列中的0-based索引标识。这两个信息存储在media_pad结构体中。使media_pad指针依照规定的方法存储和传送link的参数。
Pads有下边的标志来描述它的功能和状态
   MEDIA_PAD_FL_SINK表明这个pad支持sinking数据
MEDIA_PAD_FL_SOURCE表明这个pad支持sourcing数据
每一个pad必须设置唯一MEDIA_PAD_FL_SINK或MEDIA_PAD_FL_SOURCE标记。

 - Links
  Links are represented by a struct media_link instance, defined in include/media/media-entity.h. Each entity stores all links originating at or targeting any of its pads in a links array. A given link is thus stored twice, once in the source entity and once in the target entity. The array is pre-allocated and grows dynamically as needed.
 Drivers create links by calling
 
         media_entity_create_link(struct media_entity *source, u16 source_pad,
                                  struct media_entity *sink,   u16 sink_pad,
                                  u32 flags);
 
 An entry in the link array of each entity is allocated and stores pointers to source and sink pads.
 Links have flags that describe the link capabilities and state.
         MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
         to transfer media data. When two or more links target a sink pad, only
         one of them can be enabled at a time.
         MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
         modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
         MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
         enabled.

六、Links
Links 由include/media/media-entity.h文件中的结构体media_link实例代表。每个entity保存links队列中所有links的起源或者它的pad的目标。因此一个给定的link要被存储两次，一次存储在source entity中，另一个存储在target entity中。队列根据需求预先分配或者自动分配。
驱动可以通过下变函数来创建links
         media_entity_create_link(struct media_entity *source, u16 source_pad,
                                  struct media_entity *sink,   u16 sink_pad,
                                  u32 flags);
每个entity中links队列的entity被分配和存储在source和sink pads中。
Link 用下边的标志来描述它的功能和状态：
 MEDIA_LNK_FL_ENABLED表明link是使能的的，并能够传输media数据，当一个或多个链接目标是sink pad，在同一时间仅只能使能它们中的一个。
 MEDIA_LNK_FL_IMMUTABLE表明link使能状态不能在运行时修改。如果MEDIA_LNK_FL_IMMUTABLE被设置，那么MEDIA_LNK_FL_ENABLED必须也被设置，因此一个不变link一直被使能。

 Graph traversal
 ---------------
The media framework provides APIs to iterate over entities in a graph.
To iterate over all entities belonging to a media device, drivers can use themedia_device_for_each_entity macro, defined in include/media/media-device.h.
Drivers might also need to iterate over all entities in a graph that can be reached only through enabled links starting at a given entity. The media framework provides a depth-first graph traversal API for that purpose.
Note that graphs with cycles (whether directed or undirected) are *NOT* supported by the graph traversal API. To prevent infinite loops, the graph traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH, currently defined as 16.
Drivers initiate a graph traversal by calling
         media_entity_graph_walk_start(struct media_entity_graph *graph,
                                       struct media_entity *entity);
 

The graph structure, provided by the caller, is initialized to start graph traversal at the given entity.
Drivers can then retrieve the next entity by calling
         media_entity_graph_walk_next(struct media_entity_graph *graph);
 
  When the graph traversal is complete the function will return NULL.。Graph traversal can be interrupted at any moment. No cleanup function call is required and the graph structure can be freed normally.
Helper functions can be used to find a link between two given pads, or a pad connected to another pad through an enabled link
         media_entity_find_link(struct media_pad *source,  struct media_pad *sink);
 
         media_entity_remote_source(struct media_pad *pad);
 
 Refer to the kerneldoc documentation for more information.
六、网络遍历
 Media 框架提供一些用来遍历整个网络中所有entities的APIs。
  为了遍历所有属于media设备的entities，驱动用被定义在include/media/media-device.h.文件中的media_device_for_each_entity宏：
         struct media_entity *entity;
 
         media_device_for_each_entity(entity, mdev) {
                 /* entity will point to each entity in turn */
                 ...
         }
 驱动可能需要遍历网络中所有的entities，要实现这种遍历只能通过使能起始于给定entity的links。为了这种目的，media框架提供了一个depth-first图标遍历API。
  注意：环形的网络（没有方向）不支持通过网络遍历API，为了阻住无限循环，网络遍历代码限制最大遍历深度为MEDIA_ENTITY_ENUM_MAX_DEPTH，当前定义的值为16。
驱动通过下边函数初始化一个网络遍历：
         media_entity_graph_walk_start(struct media_entity_graph *graph,
                                       struct media_entity *entity);
 
上边的graph结构体，通过调用者提供，它在给定entity开始graph遍历之前已经被初始化。驱动通过调用下边函数检索下一个entity：
         media_entity_graph_walk_next(struct media_entity_graph *graph);
当graph遍历完成时，上边函数返回NULL。 Graph 遍历可以再任何时候中断，graph结构体可以正常被释放，清除函数没有必要调用。
辅助函数可以被用作寻找两个给定pads之间的link，或者通过一个使能的link连接两个pads。
         media_entity_find_link(struct media_pad *source,  struct media_pad *sink);
 
         media_entity_remote_source(struct media_pad *pad);
 相关的内核文档提供更多的信息。

 

 Use count and power handling
 ----------------------------
Due to the wide differences between drivers regarding power management needs,
 the media controller does not implement power management. However, the
 media_entity structure includes a use_count field that media drivers can use to
 track the number of users of every entity for power management needs.
 
 The use_count field is owned by media drivers and must not be touched by entity
 drivers. Access to the field must be protected by the media device graph_mutex
 lock.

七、引用计数和电源管理：
 因为在驱动之间关于电源管理需求完全不同，media控制器不能完成电源管理。然而，media_entity结构体包含一个use_count成员，media驱动可以用它来追踪每一个entity的用户引用计数，从而进行电源管理。
Use_count成员属于media驱动，entity驱动不能访问它。访问这个成员必须通过media 设备的graph_mutex加锁。

Links setup
 -----------
Link properties can be modified at runtime by calling
         media_entity_setup_link(struct media_link *link, u32 flags);
 
The flags argument contains the requested new link flags.
The only configurable property is the ENABLED link flag to enable/disable a link. Links marked with the IMMUTABLE link flag can not be enabled or disabled.
  When a link is enabled or disabled, the media framework calls the link_setup operation for the two entities at the source and sink of the link,in that order. If the second link_setup call fails, another link_setup call is made on the first entity to restore the original link flags.
Media device drivers can be notified of link setup operations by setting the media_device::link_notify pointer to a callback function. If provided, the notification callback will be called before enabling and after disabling links.
  Entity drivers must implement the link_setup operation if any of their links is non-immutable. The operation must either configure the hardware or store the configuration information to be applied later.
  Link configuration must not have any side effect on other links. If an enabled link at a sink pad prevents another link at the same pad from being enabled, the link_setup operation must return -EBUSY and can't implicitly disable the first enabled link.

八、Link设置
Link属性可以通过下边函数在运行时被修改：
         media_entity_setup_link(struct media_link *link, u32 flags);
  Flags参数包含被请求新的link的标志。唯一可配置属性的是已近被使能的link，可以把它们配置为使能或禁止。 被标记为IMMUTABLE link标志的links不能被使能或禁止。
  当一个link被使能或者禁止时，media框架会调用link_setup顺序操作在source和sink link上的两个entities。如果第二个link_setup调用失败，另一个link_setup调用设置的第一个entity会恢复到原来的link标志。
Media设备驱动可以通过设定media_device::link_notify指针到一个回调函数，来通知link设置操作。如果提供了通知回调函数，它将在使能links之前和禁止links之后被调用。
如果任何entity links属性是可改变的，entity驱动必须完成link_setup操作。该操作既可以配置硬件，也可以保存配置信息到随后的应用中。
 Link配置必须不能影响旁边的其他links。如果一个在sink pad上的link被使能，为了防止在相同pad上的另一个link被使能，link_setup操作必须返回-EBUSY，并且不能禁止第一个使能的link。

Pipeline and media stream
 ---------------------------
  When starting streaming, drivers must notify all entities in the pipeline to prevent link states from being modified during streaming by calling
         media_entity_pipeline_start(struct media_entity *entity,
                                     struct media_pipeline *pipe);
 
  The function will mark all entities connected to the given entity through enabled links, either directly or indirectly, as streaming.
The media_pipeline instance pointed to by the pipe argument will be stored in every entity in the pipeline. Drivers should embed the media_pipeline structure in higher-level pipeline structures and can then access the pipeline through the media_entity pipe field.
  Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must be identical for all nested calls to the function.
 When stopping the stream, drivers must notify the entities with
         media_entity_pipeline_stop(struct media_entity *entity);
 
  If multiple calls to media_entity_pipeline_start() have been made the same number of media_entity_pipeline_stop() calls are required to stop streaming. The media_entity pipe field is reset to NULL on the last nested stop call.
  Link configuration will fail with -EBUSY by default if either end of the link is a streaming entity. Links that can be modified while streaming must be marked with the MEDIA_LNK_FL_DYNAMIC flag.
If other operations need to be disallowed on streaming entities (such as changing entities configuration parameters) drivers can explicitly check the media_entity stream_count field to find out if an entity is streaming. This operation must be done with the media_device graph_mutex held.

九、 管道和media流：
  当开始传输视频流时，驱动必须通知在官道上的所有entities，为了防止link状态在数据流传输期间被改变。
         media_entity_pipeline_start(struct media_entity *entity,
                                     struct media_pipeline *pipe);
 
  函数将标记通过使能的links连接给定的所有entities，不管直接的还是间接的，只要是视频流数据。
  Media_pipeline实例指出，管道参数将被存储在管道中的每个entity中。驱动将在高一级的管道结构体中内嵌media_pipeline结构体，然后通过media_entity的pipe成员访问pipeline。
  调用media_entity_pipeline_start()函数可能被嵌套。Pipeline指针必须被相同的嵌套函数调用。
  当要停止数据流时，驱动通过下边函数通知entities：
         media_entity_pipeline_stop(struct media_entity *entity);
 
  如果有多个calls to media_entity_pipeline_start()调用，那么必须有相同数量的media_entity_pipeline_stop()被调用来停止数据流。在最后一个嵌套停止被掉时，media_entity的pipe成员必须复位成NULL。
如果 Link的任一端是一个流entity，link配置将失败，返回默认的-EBUSY。当流必须被标记为MEDIA_LNK_FL_DYNAMIC标志时，links可以被修改。
如果其他的操作需要禁止一个流entity(例如改变entity的配置参数)，驱动可以明确地检测media_entity 结构的成员stream_count来发现一个entity上运行的流。这个操作必须用media_device的graph_mutex锁。