Intel MSDK 硬解码

Intel MSDK 是 Intel 公司提供的基于硬件加速功能的多媒体开发框架，通过 Intel 显卡的硬件加速功能（Intel® Quick Sync Video），可实现快速视频转码和图像处理。

• 硬编码格式：HEVC（h.265），AVC（h.264），MPEG-2，JPEG
• 硬解码格式：HEVC，AVC，VP8，MPEG-2，VC1，JPEG，MJPEG
• Filter：颜色空间转换，去隔行，降噪，缩放，旋转

Key Specifications （支持的硬件平台，操作系统，编译器等） 请参考 Intel 官网。

Intel MSDK samples

下载链接：https://software.intel.com/en-us/media-sdk/documentation/code-samples

Sample Name	Description
Transcoding	sample_multi_transcode Transforms an elementary video stream from one compressed format to another.
Encoding	sample_encode Converts raw video frames into an elementary compressed stream.
Decoding	sample_decode Transforms a compressed video stream into raw frames using HEVC decode and VP8 decode. The plug-in and the included sample_decvpp demonstrate decode functions with color conversion of raw video sequences.
Video Processing	sample_vpp Demonstrates how to use algorithms that process raw frames using denoising, deinterlacing, inverse telecine, and color conversion techniques.
OpenCL™ Video Motion Estimation (VME)	ocl_motion_estimation Provides step-by-step guidelines on using Intel’s motion estimation extension for the OpenCL standard. The motion estimation extension includes a set of host-callable functions for frame-based VME.
OpenCL™ Interoperability	ocl_media_sdk_interop Shows how to use Intel Media SDK and Intel? SDK for Open CL? applications together for efficient video decoding and fast post-processing.
HEVC GPU Assist APIs	sample_h265_gaa Supplies examples of the typical data and control flow for using HEVC GPU Assist APIs effectively.

注：本文的代码基于 Intel_Media_SDK_2017_R1。

Intel MSDK decoding sample

以下是我改编过的 decoding sample 用法，利用 FFmpeg 进行 demux：

>intel_msdk_dec.exe
Intel(R) Media SDK Decoding Sample Version 3.5.915.45327

Usage: intel_msdk_dec.exe -i input_file [-o output_file]
Options:
   [-hw]            - use platform specific SDK implementation, if not specified software implementation is used
   [-d3d]           - work with d3d9 surfaces
   [-d3d11]         - work with d3d11 surfaces

Intel MSDK decoding 代码

以下是整个解码及播放过程的概要代码，略去各个函数的具体实现和资源释放：

    CDecodingPipeline dec_pipeline; // pipeline for decoding, includes decoder and output file writer
    SDL_video_helper sdl_helper;
    ffmpeg_reader g_file_reader;
    
    sts = ParseInputString(argv, (mfxU8)argc, &params);
    sts = g_file_reader.Init( params.strSrcFile );
    
    AVCodecID codec_id = g_file_reader.get_codec_id();
    switch (codec_id) {
    case AV_CODEC_ID_H264:
        params.videoType = MFX_CODEC_AVC;
        break;
    case AV_CODEC_ID_MPEG2VIDEO:
    case AV_CODEC_ID_MPEG2TS:
        params.videoType = MFX_CODEC_MPEG2;
        break;
    }
    
    int width = 0, height = 0;
    hr = g_file_reader.get_frame_size(width, height, g_pitch);
    
    double frame_rate = 0;
    hr = g_file_reader.get_frame_rate(frame_rate);
    
    hr = sdl_helper.init( SDL_PIXELFORMAT_IYUV, width, height, (int)frame_rate, sdl_get_frame_cb );
    sts = dec_pipeline.Init( &params, &g_file_reader );
    
    g_consumed_evt = CreateEvent( NULL, FALSE, FALSE, NULL );
    g_decoded_evt = CreateEvent( NULL, FALSE, FALSE, NULL );
    HANDLE decoding_thread_handle = CreateThread( NULL, 0, decoding_thread, &dec_pipeline, 0, NULL );
    
    hr = sdl_helper.run();

ffmpeg_reader::Init 函数

H.264 有两种封装，一种是 annexb 模式，即传统模式，有 startcode，SPS 和 PPS 是在 ES 中；一种是 mp4 模式，没有 startcode，SPS 和 PPS 以及其它信息被封装在 container 中，每一个 frame 前面是这个 frame 的长度。很多解码器只支持 annexb 这种模式，因此需要将 mp4 做转换。

H.264 的 SPS 和 PPS 串包含了初始化 H.264 解码器所需要的信息参数，包括编码所用的 profile，level，图像的宽和高，deblock 滤波器等。

mfxStatus ffmpeg_reader::Init(const TCHAR *file_path, mfxU32 video_type = 0)
{
    AVCodecContext* dec_ctx = NULL;
    m_video_stream_idx = open_input_file(file_path, AVMEDIA_TYPE_VIDEO, &m_fmt_ctx, NULL);
    RETURN_IF_FALSE_EX(m_video_stream_idx >= 0, MFX_ERR_UNKNOWN);

    if (get_codec_id() == AV_CODEC_ID_H264) {
        // Retrieve required h264_mp4toannexb filter
        const AVBitStreamFilter *bsf = av_bsf_get_by_name("h264_mp4toannexb");
        RETURN_IF_NULL_EX(bsf, MFX_ERR_UNKNOWN);

        int hr = av_bsf_alloc(bsf, &m_bsf_ctx);
        RETURN_IF_FAILED_EX(hr, MFX_ERR_UNKNOWN);

        hr = avcodec_parameters_copy(m_bsf_ctx->par_in, m_fmt_ctx->streams[m_video_stream_idx]->codecpar);
        RETURN_IF_FAILED_EX(hr, MFX_ERR_UNKNOWN);

        hr = av_bsf_init(m_bsf_ctx);
        RETURN_IF_FAILED_EX(hr, MFX_ERR_UNKNOWN);
    }

    m_inited = true;
    return MFX_ERR_NONE;
}

SDL_video_helper::init 函数

请看 这里，一模一样。

CDecodingPipeline::Init 函数

初始化 MSDK 解码用的 pipeline。

mfxStatus CDecodingPipeline::Init( sInputParams *pParams, CSmplBitstreamReader* fileReader )
{
    m_FileReader = fileReader;
    mfxStatus sts = MFX_ERR_NONE;
    mfxVersion version;
    version.Major = 1;
    version.Minor = 0;
    
    if (pParams->bUseHWLib) {             
        mfxIMPL impl = MFX_IMPL_HARDWARE_ANY; // try searching on all display adapters          
        if (D3D11_MEMORY == pParams->memType) 
            impl |= MFX_IMPL_VIA_D3D11;
            
        sts = m_mfxSession.Init(impl, &version);
        if (MFX_ERR_NONE != sts) // Some version may not support multiple adapters, try initialize on the default
            sts = m_mfxSession.Init(impl & !MFX_IMPL_HARDWARE_ANY | MFX_IMPL_HARDWARE, &version);        
    }
    else
        sts = m_mfxSession.Init(MFX_IMPL_SOFTWARE, &version);    
        
    m_pmfxDEC = new MFXVideoDECODE(m_mfxSession);
    m_mfxVideoParams.mfx.CodecId = pParams->videoType; 
    m_memType = pParams->memType;
    
    sts = InitMfxBitstream(&m_mfxBS, 1024 * 1024);
    sts = CreateAllocator();
    sts = InitMfxParams();
    sts = AllocFrames();
    sts = m_pmfxDEC->Init(&m_mfxVideoParams);
    return MFX_ERR_NONE;
}

CDecodingPipeline::InitMfxParams 函数

读文件头并初始化解码器基本信息，如图像宽高、帧率等。

mfxStatus CDecodingPipeline::InitMfxParams()
{
    mfxStatus sts = MFX_ERR_NONE;
    
    // try to find a sequence header in the stream
    while (true) {
        // parse bit stream and fill mfx params
        sts = m_pmfxDEC->DecodeHeader(&m_mfxBS, &m_mfxVideoParams);
        if (MFX_ERR_MORE_DATA == sts) {
            if (m_mfxBS.MaxLength == m_mfxBS.DataLength) {
                sts = ExtendMfxBitstream(&m_mfxBS, m_mfxBS.MaxLength * 2); 
                MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);
            }
            
            sts = m_FileReader->ReadNextFrame(&m_mfxBS);
            continue;
        }
        else
            break;
    }
    
    MSDK_IGNORE_MFX_STS(sts, MFX_WRN_PARTIAL_ACCELERATION);
    MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);
    
    m_mfxVideoParams.IOPattern = (mfxU16)(m_memType != SYSTEM_MEMORY 
        ? MFX_IOPATTERN_OUT_VIDEO_MEMORY : MFX_IOPATTERN_OUT_SYSTEM_MEMORY);
    return MFX_ERR_NONE;
}

decoding_thread 函数

不知疲倦的解码。。

DWORD WINAPI decoding_thread(void* pParam)
{
    HRESULT hr = E_FAIL;
    CDecodingPipeline* pipeline = (CDecodingPipeline*)pParam;
    while (!g_exit) {
        mfxStatus sts = pipeline->Run( on_decoded_frame );
        if (MFX_ERR_INCOMPATIBLE_VIDEO_PARAM == sts || MFX_ERR_DEVICE_LOST == sts || MFX_ERR_DEVICE_FAILED == sts) {
            if (MFX_ERR_INCOMPATIBLE_VIDEO_PARAM != sts) {
                sts = pipeline->ResetDevice();
                GOTO_IF_FALSE(sts >= MFX_ERR_NONE);
            } 
            
            sts = pipeline->ResetDecoder();
            GOTO_IF_FALSE(sts >= MFX_ERR_NONE);
            
            continue;
        }
        else {
            GOTO_IF_FALSE(sts >= MFX_ERR_NONE);
            break;
        }
    }
    
    hr = S_OK;
RESOURCE_FREE:
    g_eof = true;
    return hr;
}

CDecodingPipeline::Run 函数

读取文件然后解码。

mfxStatus CDecodingPipeline::Run(FP_ON_DECODED_SURFACE callback)
{   
    m_pDecodeCallback = callback;
    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_DATA == sts || MFX_ERR_MORE_SURFACE == sts) {
        if (MFX_WRN_DEVICE_BUSY == sts) {
            MSDK_SLEEP(1); // just wait and then repeat the same call to DecodeFrameAsync
        }
        else if (MFX_ERR_MORE_DATA == sts) {
            // read more data to input bit stream
            sts = m_FileReader->ReadNextFrame(&m_mfxBS); 
            MSDK_BREAK_ON_ERROR(sts);            
        }
        
        if (MFX_ERR_MORE_SURFACE == sts || MFX_ERR_NONE == sts) {
            // find new working surface 
            nIndex = GetFreeSurfaceIndex(m_pmfxSurfaces, m_mfxResponse.NumFrameActual); 
        }
        
        BREAK_ON_FAIL( DecodeFrame(sts, &m_mfxBS, &(m_pmfxSurfaces[nIndex])) );
    }
    
    //save the main loop exit status (required for the case of ERR_INCOMPATIBLE_PARAMS)
    mfxStatus mainloop_sts = sts; 
    
    MSDK_IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA); // means that file has ended, need to go to buffering loop
    MSDK_IGNORE_MFX_STS(sts, MFX_ERR_INCOMPATIBLE_VIDEO_PARAM); 
    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_SURFACE == sts) {        
        if (MFX_WRN_DEVICE_BUSY == sts)
            MSDK_SLEEP(1);
            
        mfxU16 nIndex = GetFreeSurfaceIndex(m_pmfxSurfaces, m_mfxResponse.NumFrameActual);
        BREAK_ON_FAIL( DecodeFrame(sts, NULL, &(m_pmfxSurfaces[nIndex])) );
    } 
    
    // MFX_ERR_MORE_DATA is the correct status to exit buffering loop with
    MSDK_IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA); 

    // send out if exited main decoding loop with ERR_INCOMPATIBLE_PARAM
    if (MFX_ERR_INCOMPATIBLE_VIDEO_PARAM == mainloop_sts)
        sts = mainloop_sts; 
    
    return sts; // ERR_NONE or ERR_INCOMPATIBLE_VIDEO_PARAM
}

ffmpeg_reader::ReadNextFrame 函数

读取一帧未解码的数据 → 转 Annex B 格式 → 写入 bit stream。

mfxStatus ffmpeg_reader::ReadNextFrame(mfxBitstream *out_bit_stream) 
{
    while (!video_frame_found) { // Read until video frame is found or no more video frames in container.
        if (0 == av_read_frame(m_fmt_ctx, &packet)) {
            if (packet.stream_index == m_video_stream_idx) {
                switch (codec_id) {
                case AV_CODEC_ID_H264:
                    hr = do_bsf_filter(&packet); // Apply MP4 to H264 Annex B filter on buffer
                    if (m_bsf_filtered_pkts.empty() && (hr == AVERROR(EAGAIN))) 
                        continue;
                        
                    copy_filt_pkt_to_bit_stream(out_bit_stream);
                    break;
                }
                
                out_bit_stream->DataFlag = MFX_BITSTREAM_COMPLETE_FRAME;
                video_frame_found = true;
            } 
            
            av_packet_unref(&packet);
        }
        else 
            return MFX_ERR_MORE_DATA;
    }
    
    return MFX_ERR_NONE;
}

ffmpeg_reader::do_bsf_filter 函数

MP4 格式转 Annex B 格式。

int ffmpeg_reader::do_bsf_filter(AVPacket* pkt)
{
    int hr = av_bsf_send_packet(m_bsf_ctx, pkt);
    RETURN_IF_FAILED(hr);
    
    while (true) {
        AVPacket filt_pkt;
        init_packet(&filt_pkt);
        
        hr = av_bsf_receive_packet(m_bsf_ctx, &filt_pkt);
        if (0 == hr)
            m_bsf_filtered_pkts.push(filt_pkt);
        else {
            av_packet_unref(&filt_pkt);
            break;
        }
    } 

    return hr;
}

CDecodingPipeline::DecodeFrame 函数

解完一帧后通过回调函数处理。

HRESULT CDecodingPipeline::DecodeFrame( mfxStatus& sts, mfxBitstream* bitStream, mfxFrameSurface1* pInSurface )
{
    HRESULT hr = E_FAIL;
    mfxSyncPoint syncp = NULL;
    mfxFrameSurface1 *pOutSurface = NULL;
    
    sts = m_pmfxDEC->DecodeFrameAsync(bitStream, pInSurface, &pOutSurface, &syncp);
    if (MFX_ERR_NONE < sts && syncp) // ignore warnings if output is available
        sts = MFX_ERR_NONE;
        
    if (MFX_ERR_NONE == sts)
        sts = m_mfxSession.SyncOperation(syncp, MSDK_DEC_WAIT_INTERVAL);      
               
    bool isLocked = false;
    if (MFX_ERR_NONE == sts) {                
        if (m_bExternalAlloc) {
            sts = m_pMFXAllocator->Lock(m_pMFXAllocator->pthis, pOutSurface->Data.MemId, &(pOutSurface->Data));
            isLocked = true;
        }
        
        sts = m_pDecodeCallback(pOutSurface);        
    } 
    
    hr = S_OK;
RESOURCE_FREE:
    if (isLocked && m_bExternalAlloc)
        sts = m_pMFXAllocator->Unlock(m_pMFXAllocator->pthis, pOutSurface->Data.MemId, &(pOutSurface->Data));
    return hr;
}

on_decoded_frame 回调函数

解码出来的帧格式是 NV12 的，需要转成 SDL 喜欢的 YUV420P 格式，最后通知 SDL 来拿吧。

// callback from Intel decoding pipeline
mfxStatus on_decoded_frame(mfxFrameSurface1* pSurface)
{
    while ((g_yuv420p_frames.size() >= MAX_QUEUE_SIZE) && (!g_exit))
        WaitForSingleObject(g_consumed_evt, 1000);

    if (g_exit)
        return MFX_ERR_ABORTED;

    BYTE* yuv420p_buf = NULL;
    convert_nv12_to_yuv420p(pSurface, &yuv420p_buf);

    {
        MFUtil::AutoLock lock(g_critSec);
        g_yuv420p_frames.push(yuv420p_buf);
    }

    SetEvent(g_decoded_evt);
    return MFX_ERR_NONE;
}

SDL_video_helper::run 函数

请看 这里，一模一样。

sdl_get_frame_cb 回调函数

SDL 取视频帧的回调函数，经典的 消费者 / 生产者 模式。

// callback from SDL to get a decoded frame to render
int sdl_get_frame_cb(BYTE** pixel, int* pitch)
{
    while (g_yuv420p_frames.empty()) {
        if (g_eof)
            return SDL_ERR_EOF;
            
        WaitForSingleObject(g_decoded_evt, 1000);
    }

    SAFE_DELETE_ARRAY(g_last_render_frame);

    {
        MFUtil::AutoLock lock(g_critSec);
        g_last_render_frame = g_yuv420p_frames.front();
        g_yuv420p_frames.pop();
    }

    *pixel = g_last_render_frame;
    *pitch = g_pitch;

    SetEvent(g_consumed_evt);

    return S_OK;
}

Intel MSDK 硬编码

请参考我的另一篇 文章。

– EOF –