目录
一、背景
在 FFmpeg 中,滤镜(filter)处理的是未压缩的原始音视频数据(RGB/YUV视频帧,PCM音频帧等)。一个滤镜的输出可以连接到另一个滤镜的输入,多个滤镜可以连接起来,构成滤镜链/滤镜图,各种滤镜的组合为 FFmpeg 提供了丰富的音视频处理功能。
二、FFmpeg常用滤镜
比较常用的滤镜有:scale、trim、overlay、rotate、movie、yadif。scale 滤镜用于缩放,trim 滤镜用于帧级剪切,overlay 滤镜用于视频叠加,rotate 滤镜实现旋转,movie 滤镜可以加载第三方的视频,yadif 滤镜可以去隔行。
三、FFmpeg滤镜API简介
FFmpeg滤镜API是libavfilter库中的一部分。它提供了一个框架,该框架可以在视频和音频数据流上执行各种操作,包括但不限于:
- 图像大小和比例调整(scaling,cropping)
- 颜色校正
- 音频噪声减少和音量调整
- 特效添加,如模糊和锐化
- 视频转场效果
- 字幕、时间码和水印的叠加
1. 滤镜的使用方法
要使用FFmpeg滤镜API,我们需要对FFmpeg有一定的了解,包括如何安装FFmpeg和它的基本命令行操作。
2.滤镜的使用流程
2.1、初始化:
创建并初始化滤镜图,这是处理媒体文件时的数据结构。
2.2、添加滤镜:
向滤镜图中添加所需的滤镜。每个滤镜都可能有自己的参数。
2.3、配置:
配置滤镜图,确保所有的滤镜都正确连接。
2.4、处理数据:
将解码的帧送入滤镜图,然后获取处理后的帧进行编码或其他操作。
3. 滤镜数据结构与API简介
2.1 struct AVFilter
/**
* Filter definition. This defines the pads a filter contains, and all the
* callback functions used to interact with the filter.
*/
typedef struct AVFilter {
const char *name;
const char *description;
const AVFilterPad *inputs;
const AVFilterPad *outputs;
const AVClass *priv_class;
int flags;
// private API
......
} AVFilter;
2.2 struct AVFilterContext
/** An instance of a filter */
struct AVFilterContext {
const AVClass *av_class; ///< needed for av_log() and filters common options
const AVFilter *filter; ///< the AVFilter of which this is an instance
char *name; ///< name of this filter instance
AVFilterPad *input_pads; ///< array of input pads
AVFilterLink **inputs; ///< array of pointers to input links
unsigned nb_inputs; ///< number of input pads
AVFilterPad *output_pads; ///< array of output pads
AVFilterLink **outputs; ///< array of pointers to output links
unsigned nb_outputs; ///< number of output pads
void *priv; ///< private data for use by the filter
struct AVFilterGraph *graph; ///< filtergraph this filter belongs to
......
};
2.3 struct AVFilterGraph
typedef struct AVFilterGraph {
const AVClass *av_class;
AVFilterContext **filters;
unsigned nb_filters;
......
} AVFilterGraph;
2.4 struct AVFilterLink
/**
* A link between two filters. This contains pointers to the source and
* destination filters between which this link exists, and the indexes of
* the pads involved. In addition, this link also contains the parameters
* which have been negotiated and agreed upon between the filter, such as
* image dimensions, format, etc.
*
* Applications must not normally access the link structure directly.
* Use the buffersrc and buffersink API instead.
* In the future, access to the header may be reserved for filters
* implementation.
*/
struct AVFilterLink {
AVFilterContext *src; ///< source filter
AVFilterPad *srcpad; ///< output pad on the source filter
AVFilterContext *dst; ///< dest filter
AVFilterPad *dstpad; ///< input pad on the dest filter
......
}
2.5 struct AVFilterInOut
/**
* A linked-list of the inputs/outputs of the filter chain.
*
* This is mainly useful for avfilter_graph_parse() / avfilter_graph_parse2(),
* where it is used to communicate open (unlinked) inputs and outputs from and
* to the caller.
* This struct specifies, per each not connected pad contained in the graph, the
* filter context and the pad index required for establishing a link.
*/
typedef struct AVFilterInOut {
/** unique name for this input/output in the list */
char *name;
/** filter context associated to this input/output */
AVFilterContext *filter_ctx;
/** index of the filt_ctx pad to use for linking */
int pad_idx;
/** next input/input in the list, NULL if this is the last */
struct AVFilterInOut *next;
} AVFilterInOut;
2.6 avfilter_graph_create_filter()
/**
* Create and add a filter instance into an existing graph.
* The filter instance is created from the filter filt and inited
* with the parameters args and opaque.
*
* In case of success put in *filt_ctx the pointer to the created
* filter instance, otherwise set *filt_ctx to NULL.
*
* @param name the instance name to give to the created filter instance
* @param graph_ctx the filter graph
* @return a negative AVERROR error code in case of failure, a non
* negative value otherwise
*/
int avfilter_graph_create_filter(AVFilterContext **filt_ctx, const AVFilter *filt,
const char *name, const char *args, void *opaque,
AVFilterGraph *graph_ctx);
2.7 avfilter_graph_parse_ptr()
/**
* Add a graph described by a string to a graph.
*
* In the graph filters description, if the input label of the first
* filter is not specified, "in" is assumed; if the output label of
* the last filter is not specified, "out" is assumed.
*
* @param graph the filter graph where to link the parsed graph context
* @param filters string to be parsed
* @param inputs pointer to a linked list to the inputs of the graph, may be NULL.
* If non-NULL, *inputs is updated to contain the list of open inputs
* after the parsing, should be freed with avfilter_inout_free().
* @param outputs pointer to a linked list to the outputs of the graph, may be NULL.
* If non-NULL, *outputs is updated to contain the list of open outputs
* after the parsing, should be freed with avfilter_inout_free().
* @return non negative on success, a negative AVERROR code on error
*/
int avfilter_graph_parse_ptr(AVFilterGraph *graph, const char *filters,
AVFilterInOut **inputs, AVFilterInOut **outputs,
void *log_ctx);
2.8 avfilter_graph_config()
/**
* Check validity and configure all the links and formats in the graph.
*
* @param graphctx the filter graph
* @param log_ctx context used for logging
* @return >= 0 in case of success, a negative AVERROR code otherwise
*/
int avfilter_graph_config(AVFilterGraph *graphctx, void *log_ctx);
2.9 av_buffersrc_add_frame_flags()
/**
* Add a frame to the buffer source.
*
* By default, if the frame is reference-counted, this function will take
* ownership of the reference(s) and reset the frame. This can be controlled
* using the flags.
*
* If this function returns an error, the input frame is not touched.
*
* @param buffer_src pointer to a buffer source context
* @param frame a frame, or NULL to mark EOF
* @param flags a combination of AV_BUFFERSRC_FLAG_*
* @return >= 0 in case of success, a negative AVERROR code
* in case of failure
*/
av_warn_unused_result
int av_buffersrc_add_frame_flags(AVFilterContext *buffer_src,
AVFrame *frame, int flags);
2.10 av_buffersink_get_frame()
/**
* Get a frame with filtered data from sink and put it in frame.
*
* @param ctx pointer to a context of a buffersink or abuffersink AVFilter.
* @param frame pointer to an allocated frame that will be filled with data.
* The data must be freed using av_frame_unref() / av_frame_free()
*
* @return
* - >= 0 if a frame was successfully returned.
* - AVERROR(EAGAIN) if no frames are available at this point; more
* input frames must be added to the filtergraph to get more output.
* - AVERROR_EOF if there will be no more output frames on this sink.
* - A different negative AVERROR code in other failure cases.
*/
int av_buffersink_get_frame(AVFilterContext *ctx, AVFrame *frame);
四、使用举例
假设我们想要将一个视频文件转换为特定大小,并且添加一个水印。我们可以使用命令行如下:
ffmpeg -i input.mp4 -vf "scale=1280:720, watermark=watermark.png" output.mp4
如果要用FFmpeg滤镜API实现同样的功能,我们就需要创建scale滤镜和overlay滤镜,并将它们添加到滤镜图中,然后将视频帧通过滤镜图处理。
在实现这些功能时,我们需要考虑到的不仅仅是如何将滤镜插入到流程中,还需要考虑如何读取和解码输入文件,以及如何编码和写入输出文件。
五、使用注意事项
1、版本兼容性
FFmpeg的API可能会随版本更新而改变。在使用时确保对应的API与当前安装的FFmpeg版本相兼容。
2、内存管理
FFmpeg中的很多操作都涉及到动态内存的分配和释放。需要仔细管理,防止内存泄漏。
3、线程安全
在多线程环境中使用FFmpeg时要确保线程安全。某些函数可能不是线程安全的,需要额外的锁来保护。
4、错误处理
FFmpeg的函数大多数会返回错误码。调用时需要检查返回值,并做适当的错误处理。
参考文章