FFmpeg In Android - tutorial-1-Making Screencaps

Overview 概述

Movie files have a few basic components. First, the file itself is called a container, and the type of container determines where the information in the file goes. Examples of containers are AVI and Quicktime. Next, you have a bunch of streams; for example, you usually have an audio stream and a video stream. (A “stream” is just a fancy word for “a succession of data elements made available over time”.)
The data elements in a stream are called frames. Each stream is encoded by a different kind of codec. The codec defines how the actual data is COded and DECoded - hence the name CODEC.Examples of codecs are DivX and MP3. Packets are then read from thestream. Packets are pieces of data that can contain bits of data that are decoded into raw frames that we can finally manipulate for our application.
For our purposes, each packet contains complete frames, or multiple frames in the case of audio.

At its very basic level, dealing with video and audio streams is very easy:

电影文件有些基本的组成部分。首先,文件本身被称为容器(container),容器的类型决定了信息被存放在文件中的位置。AVI 和 Quicktime 就是容器的例子。接着,你有一组流(streams),例如,你通常有的是一个音频流和一个视频流。(一个“流”只是一种想像出来的词语,用来表示“一连串的通过时间来串连的数据元素”)。
在流中的数据元素被称为帧(frames)。每个流是由不同的编码器(codec)来编码生成的。编解码器描述了实际的数据是如何被编码 COded 和解码 DECoded 的,因此它的名字叫做 CODEC。Divx 和 MP3 就是编解码器的例子。接着从流中被读出来的叫做包(packets)。包是一段数据,它包含了一段数据可以被解码成我们在应用程序中操作的原始帧。根据我们的目的,每个包包含了完整的帧, 对于音频来说是多个完整帧。
基本上来说,处理视频和音频流是很容易的:

1. OPEN video_stream FROM video.avi 从video.avi 文件中打开视频流 video_stream
2. READ packet FROM video_stream INTO frame 从视频流中读取包到帧中
3. IF frame NOT COMPLETE GOTO 2 如果这个帧还不完整,跳到2
4. DO SOMETHING WITH frame 对这个帧进行一些操作
5. GOTO 2 跳回到2

Handling multimedia with ffmpeg is pretty much as simple as this program,although some programs might have a very complex “DO SOMETHING” step. So inthis tutorial, we’re going to open a file, read from the video stream insideit, and our DO SOMETHING is going to be writing the frame to a PPM file.

在这个程序中使用 ffmpeg 来处理多种媒体是相当容易的,虽然很多程序可能在一些操作步骤上非常的复杂。因此在这篇教程中,我们将打开一个文件,读取里面的视频流,我们操作步骤将是把这个帧写到一个 PPM 文件中。

Opening the File 打开文件

First, let’s see how we open a file in the first place. With ffmpeg, you have to first initialize the library.
首先,来看一下我们如何打开一个文件。通过 ffmpeg,你必需先初始化这个库。

 #include <libavcodec/avcodec.h>
 #include <libavformat/avformat.h>
    ...
    int main(int argc, charg *argv[]) {
    	av_register_all();

This registers all available file formats and codecs with the library so they will be used automatically when a file with the corresponding format/codec is opened. Note that you only need to call av_register_all() once, so we do it here in main(). If you like, it’s possible to register only certain individual file formats and codecs, but there’s usually no reason why you would have to do that.
这里注册了所有的文件格式和编解码器的库,所以它们将被自动的使用在被打开的合适格式的文件上。注意你只需要调用 av_register_all() 一次,因此我们在主函数 main() 中来调用它。如果你喜欢,也可以只注册特定的格式和编解码器,但是通常你没有必要这样做。

Now we can actually open the file:
现在我们可以真正的打开文件:

	AVFormatContext *pFormatCtx = NULL;
    
    // Open video file
    if(avformat_open_input(&pFormatCtx;, argv[1], NULL, 0, NULL)!=0)
      return -1; // Couldn't open file

We get our filename from the first argument. This function reads the file header and stores information about the file format in the AVFormatContext structure we have given it. The last three arguments are used to specify the file format, buffer size, and format options, but by setting this to NULL or 0, libavformat will auto-detect these. This function only looks at the header, so next we need to check out the stream information in the file.:
我们通过第一个参数来获得文件名。这个函数读取文件的头部并且把信息保存到我们给的 AVFormatContext结构体中。最后三个参数用来指定特殊的文件格式,缓冲大小和格式参数,但如果把它们设置为空 NULL 或者 0, libavformat 将自动检测这些参数。
这个函数只是检测了文件的头部,所以接着我们需要检查在文件中的流的信息:

	// Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0)
      return -1; // Couldn't find stream information

This function populates pFormatCtx->streams with the proper information. We introduce a handy debugging function to show us what’s inside:
这个函数为 pFormatCtx->streams 填充上正确的信息。我们引进一个手工调试的函数来看一下里面有什么:

	// Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);

Now pFormatCtx->streams is just an array of pointers, of size pFormatCtx->nb_streams, so let’s walk through it until we find a video stream.
现在 pFormatCtx->streams 仅仅是一组大小为 pFormatCtx->nb_streams 的数组指针,所以让我们先跳过它直到我们找到一个视频流。

	int i;
    AVCodecContext *pCodecCtxOrig = NULL;
    AVCodecContext *pCodecCtx = NULL;
    
    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
      if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
        videoStream=i;
        break;
      }
    if(videoStream==-1)
      return -1; // Didn't find a video stream
    
    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;

The stream’s information about the codec is in what we call the “codec context.” This contains all the information about the codec that the stream is using, and now we have a pointer to it. But we still have to find the actual codec and open it:
流中关于编解码器的信息就是被我们叫做“编解码器上下文”（codec context）的东西。这里面包含了流中所使用的关于编解码器的所有信息，现在我们有了一个指向它的指针。但是我们必需要找到真正的编解码器并且打开它：

AVCodec *pCodec = NULL;
    
    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
      fprintf(stderr, "Unsupported codec!\n");
      return -1; // Codec not found
    }
    // Copy context
    pCodecCtx = avcodec_alloc_context3(pCodec);
    if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0) {
      fprintf(stderr, "Couldn't copy codec context");
      return -1; // Error copying codec context
    }
    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec)<0)
      return -1; // Could not open codec

Note that we must not use the AVCodecContext from the video stream directly!
So we have to use avcodec_copy_context() to copy the context to a new location
(after allocating memory for it, of course).
注意我们不能直接使用video stream中的AVCodecContext，所以我们调用avcodec_copy_context函数来复制到新的位置(先分配内存)。

Storing the Data 保存数据

Now we need a place to actually store the frame:
现在我们需要找到一个地方来保存帧：

	AVFrame *pFrame = NULL;
    
    // Allocate video frame
    pFrame=av_frame_alloc();

Since we’re planning to output PPM files, which are stored in 24-bit RGB, we’re going to have to convert our frame from its native format to RGB. ffmpeg will do these conversions for us. For most projects (including ours) we’re going to want to convert our initial frame to a specific format. Let’s allocate a frame for the converted frame now.
因为我们准备输出保存 24 位 RGB 色的 PPM 文件，我们必需把帧的格式从原来的转换为 RGB。 FFMPEG 将为我们做这些转换。在大多数项目中（包括我们的这个）我们都想把原始的帧转换成一个特定的格式。让我们先为转换来申请一帧的内存。

uint8_t *video_dst_data[4] = {NULL};
int      video_dst_linesize[4];

 av_image_alloc(video_dst_data, video_dst_linesize,
                             width, height, AV_PIX_FMT_RGB24, 1);

Finally! Now we’re ready to read from the stream!
终于，我们已经准备好来从流中读取数据了。

Reading the Data 读取数据

What we’re going to do is read through the entire video stream by reading in the packet, decoding it into our frame, and once our frame is complete, we will convert and save it.
我们将要做的是通过读取包来读取整个视频流，然后把它解码成完成的一帧，最后转换格式并且保存。

	struct SwsContext *sws_ctx = NULL;
    int frameFinished;
    AVPacket packet;
    // initialize SWS context for software scaling
    sws_ctx = sws_getContext(pCodecCtx->width,
        pCodecCtx->height,
        pCodecCtx->pix_fmt,
        pCodecCtx->width,
        pCodecCtx->height,
        PIX_FMT_RGB24,
        SWS_BILINEAR,
        NULL,
        NULL,
        NULL
        );
    
    i=0;
    while(av_read_frame(pFormatCtx, &packet;)>=0) {
      // Is this a packet from the video stream?
      if(packet.stream_index==videoStream) {
    	// Decode video frame
        avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished;, &packet;);
        
        // Did we get a video frame?
        if(frameFinished) {
        // Convert the image from its native format to RGB
            sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
    		  pFrame->linesize, 0, pCodecCtx->height,
    		  video_dst_data, video_dst_linesize);
    	
            // Save the frame to disk
            if(++i<=5)
              SaveFrame(pFrameRGB, pCodecCtx->width, 
                        pCodecCtx->height, i);
        }
      }
        
      // Free the packet that was allocated by av_read_frame
      av_free_packet(&packet;);
    }

A note on packets 关于包 Packets 的注释：
Technically a packet can contain partial frames or other bits of data, but ffmpeg’s parser ensures that the packets we get contain either complete or multiple frames.
从技术上讲一个包可以包含部分或者其它的数据，但是 ffmpeg 的解封装器保证了我们得到的
包 Packets 包含的要么是完整的要么是多个完整的帧。

The process, again, is simple: av_read_frame() reads in a packet and stores it in the AVPacket struct. Note that we’ve only allocated the packet structure - ffmpeg allocates the internal data for us, which is pointed to by packet.data. This is freed by the av_free_packet() later. avcodec_decode_video() converts the packet to a frame for us. However, we might not have all the information we need for a frame after decoding a packet, so avcodec_decode_video() sets frameFinished for us when we have the next frame. Finally, we use sws_scale() to convert from the native format (pCodecCtx->pix_fmt) to RGB. Finally, we pass the frame and height and width information to our SaveFrame function.
这个循环过程是比较简单的： av_read_frame() 读取一个包并且把它保存到 AVPacket 结构体中。注意我们仅仅申请了一个包的结构体——ffmpeg 为我们申请了内部的数据的内存并通过 packet.data 指针来指向它。这些数据可以在后面通过 av_free_packet() 来释放。函数avcodec_decode_video() 把包转换为帧。然而当解码一个包的时候，我们可能没有得到一个完整帧的信息。因此，当我们得到下一帧的时候， avcodec_decode_video() 为我们设置了帧结束标志 frameFinished。最后，我们使用sws_scale()函数来把帧从原始格（pCodecCtx->pix_fmt）转换成为 RGB 格式。最后，我们把帧和高度宽度信息传递给我们的 SaveFrame 函数。

Now all we need to do is make the SaveFrame function to write the RGB information to a file in PPM format. We’re going to be kind of sketchy on the PPM format itself; trust us, it works.
现在我们需要做的是让 SaveFrame 函数能把 RGB 信息写入到一个 PPM 格式的文件中。我们将生成一个简单的 PPM 格式文件，请相信，它是可以工作的。

	void ppm_save(unsigned char *buf, int wrap, int xsize, int ysize, char *filename) {
    FILE *f;
    int y;

    f = fopen(filename,"w");
    fprintf(f, "P6\n%d %d\n%d\n", xsize, ysize, 255);
    for (y = 0; y < ysize; y++)
        fwrite(buf + y * wrap, 1, xsize * 3, f);
    fclose(f);
}

We do a bit of standard file opening, etc., and then write the RGB data. We write the file one line at a time. A PPM file is simply a file that has RGB information laid out in a long string. If you know HTML colors, it would be like laying out the color of each pixel end to end like #ff0000#ff0000…would be a red screen. (It’s stored in binary and without the separator, but you get the idea.) The header indicated how wide and tall the image is, andthe max size of the RGB values.
我们做了一些标准的文件打开动作，然后写入 RGB 数据。我们一次向文件写入一行数据。 PPM 格式文件的是一种包含一长串的 RGB 数据的文件。如果你了解 HTML 色彩表示的方式，那么它就类似于把每个像素的颜色头对头的展开，就像 #ff0000#ff0000… 就表示了了个红色的屏幕。（它被保存成二进制方式并且没有分隔符，但是你自己是知道如何分隔的）。文件的头部表示了图像的宽度和高度以及最大的 RGB 值的大小。

Now, going back to our main() function. Once we’re done reading from the video stream, we just have to clean everything up:
现在，回顾我们的 main() 函数。一旦我们开始读取完视频流，我们必需清理一切：

	// Free the RGB image
    av_free(buffer);
    av_free(pFrameRGB);
    
    // Free the YUV frame
    av_free(pFrame);
    
    // Close the codecs
    avcodec_close(pCodecCtx);
    avcodec_close(pCodecCtxOrig);
    
    // Close the video file
    avformat_close_input(&pFormatCtx;);
    
    return 0;

That’s it for the code! Now, if you’re on Linux or a similar platform, you’ll run:
就这些！下面，我们将使用 Linux 或者其它类似的平台，运行：

gcc -o tutorial01 tutorial01.c -lavutil -lavformat -lavcodec -lz -lavutil -lm

g++ -std=c++14 -o tuturial02 tutorial02.cpp -I/INCLUDE_PATH -L/LIB_PATH -lavutil -lavformat -lavcodec -lswscale -lswresample -lavdevice -lz -lavutil -lm -lpthread -ldl

Most image programs should be able to open PPM files. Test it on some movie files.
大多数的图像处理函数可以打开 PPM 文件。可以使用一些电影文件来进行测试。

源码下载