Using libavformat and libavcodec(1)

http://www.inb.uni-luebeck.de/~boehme/using_libavcodec.html

 

The libavformat and libavcode libraries that come with ffmpeg are a great way of accessing a large variety of video file formats. Unfortunately, there is no real documentation on using these libraries in your own programs(at least I couldn't find any), and the example programs aren't really very helpful either.

 

 

This situation meant that, when I used libavformat/libavcodec on a recent project, it took quite a lot of experimentation to find out how to use them. Here's what I learned - hopefully I'll be able to save others from having to go through the same trial-and-error process. There's also a small demo program that you can download. The code I'll present works with libavformat/libavcodec as included in version 0.4.8 of ffmpeg (the most recent version as I'm writing this). If you find that later versions break the code, please let me know.

 

In this document, I'll only cover how to read video streams from a file; audio streams work pretty much the same way, but I haven't actually used them, so I can't present any example code.

 

In case you're wondering why there are two libraries, libavformat and libavcodec: Many video file formats(AVI being a prime example) don't actually specify which codec(s) should be used to encode audio and video data; they merely define how an audio and a video stream(or, potentially, several audio/video streams) should be combined into a single file. This  is why sometimes, when you open an AVI file, you get only sound, but no picture - because the right video codec isn't installed on your system. Thus, libavformat deals with parsing video files and separating the streams contained in them, and libavcodec deals with decoding raw audio and video streams.

 

 

Opening a Video File

First things first - let's look at how to open a video file and get at the streams contained in it. The first thing we need to do is to initialize libavformat/libavcodec.

av_register_all() 

This registers all available file formats and codecs with the library so they will be used automatically when a file with the corresponding format/codec is opened. Note that you only need to call av_register_all() once, so it's probably best to do this somewhere in your startup code. If you like, it's possible to register only certain individual file formats and codecs, but there's usually no reason why you would have to do that.

 

Next off, opening the file:

AVFormatContext *pFormatCtx; const char *filename = "myvideo.mpg" // Open video file if (av_open_input_file(&pFormatCtx, filename, NULL, 0, NULL) != 0) handle_error(); // Couldn't open file 

The last three parameters specify the file format, buffer size and format parameters; by simply specifying NULL or 0 we ask libavformat to auto-detect the format and use a default buffer size. Replace handle_error() with appropriate error handling code for your application.

 

Next, we need to retrieve information about the streams contained in the file:

// Retrieve stream information if (av_find_stream_info(pFormatCtx) < 0) handle_error(); // Couldn't find stream information 

This fill the streams field of the AVFormatContext with valid information. As a debugging aid, we'll dump this information onto standard error, but of course you don't have to do this in a a production application

dump_format(pFormatCtx, 0, filename, false); 

As mentioned in the introduction, we'll handle only video streams, not audio streams, To make things nice and easy, we simply use the first video stream we find:

int i, videoStream; AVCodecContext *pCodeCtx; // Find the first video stream videoStream = 1; for (i = 0; i < pFormatCtx->nb_streams; i++) { if (pFormatCtx->streams[i]->codec.codec_type == CODEC_TYPE_VIDEO) { videoStream = i; break; } } if (videoStream == -1) handle_error(); // Didn't find a video stream // Get a pointer to the codec context for the video stream pCodecCtx = &pFormatCtx->streams[videoStream]->codec; 

OK,so now we've got a pointer to the so-called codec context for our video stream, but we still have to find the actual codec and open it:

AVCodec *pCodec; // Find the decoder for the video stream pCodec = avcodec_find_decoder(pCodecCtx->codec_id); if (pCodec == NULL) handle_error(); //Codec not found // Inform the codec that we can handle truncated bitstreams -- i.e., // bitstreams where frame boundaries can fail in the middle of packets if (pCodec->capabilities & CODEC_CAP_TRUNCATED) pCodecCtx->flags |= CODEC_FLAGS_TRUNCATED; // Open codec if (avcodec_open(pCodecCtx, pCodec) < 0) handle_error(); // Counld not open codec 

(So what's up with those "truncated bitstreams"? Well, as we'll see in a moment, the data in a video stream is split up into packets. Since the amount of data per video frame can vary, the boundaires between two video frames need to coincide with a packet boundary. Here, we're telling the codec that we can handle this situation.)

 

One important piece of information that is stored in the AVCodeContext structure is the frame rate of the video. To allow for non-integer frame rates (like NTSC's 29.97 fps), the rate is stored as a fraction, with the numerator in pCodecCtx->frame_rate and the denominator in pCodecCtx->frame_rate_base. While testing the library with different video files, I noticed that some codec(notably ASF) seem to fill these fields incorrectly(frame_rate_base contains 1 instead of 1000). The following hack fixes this:

// Hack to correct wrong frame rates that seem to be generated by some codecs if (pCodecCtx->frame_rate > 1000 && pCodecCtx->frame_rate_base == 1) pCodecCtx->frame_rate_base = 1000; 

Note that it shouldn't be a problem to leave this fix in place even if the bug is corrected some day - it's unlikely that a video would have a frame rate of more than 1000 fps.

One more thing left to do: Allocate a video frame to store the decoded images in:

AVFrame *pFrame; pFrame = avcodec_alloc_frame(); 

That's it!Now let's start decoding some video.