iOS利用VideoToolbox实现视频硬解码

1、总体架构

---

总体思想即将FFmpeg parse到的数据装到CMBlockBuffer中,将extra data分离出的vps,sps,pps装到CMVideoFormatDesc中,将计算好的时间戳装到CMTime中,最后即可拼成完成的CMSampleBuffer以用来提供给解码器.

CMSampleBufferCreate

1.1 简易流程

FFmpeg parse流程

• 创建format context: avformat_alloc_context

• 打开文件流: avformat_open_input

• 寻找流信息: avformat_find_stream_info

• 获取音视频流的索引值: formatContext->streams[i]->codecpar->codec_type == (isVideoStream ? AVMEDIA_TYPE_VIDEO : AVMEDIA_TYPE_AUDIO)

• 获取音视频流: m_formatContext->streams[m_audioStreamIndex]

• 解析音视频数据帧: av_read_frame

• 获取extra data: av_bitstream_filter_filter

VideoToolbox decode流程

• 比较上一次的extra data,如果数据更新需要重新创建解码器

• 分离并保存FFmpeg parse到的extra data中分离vps, sps, pps等关键信息 (比较NALU头)

• 通过CMVideoFormatDescriptionCreateFromH264ParameterSets,CMVideoFormatDescriptionCreateFromHEVCParameterSets装载vps,sps,pps等NALU header信息.

• 指定解码器回调函数与解码后视频数据类型(yuv,RGB...)

• 创建解码器VTDecompressionSessionCreate

• 生成CMBlockBufferRef装载解码前数据,再将其转为CMSampleBufferRef以提供给解码器.

• 开始解码VTDecompressionSessionDecodeFrame

• 在回调函数中CVImageBufferRef即为解码后的数据,可转为CMSampleBufferRef传出.

1.2 文件结构

image

1.3 快速使用

• 初始化preview解码后的视频数据将渲染到该预览层

- (void)viewDidLoad {

[super viewDidLoad];

[self setupUI];

}

​

- (void)setupUI {

self.previewView = [[XDXPreviewView alloc] initWithFrame:self.view.frame];

[self.view addSubview:self.previewView];

[self.view bringSubviewToFront:self.startBtn];

}

• 解析并解码文件中视频数据

- (void)startDecodeByVTSessionWithIsH265Data:(BOOL)isH265 {

NSString *path = [[NSBundle mainBundle] pathForResource:isH265 ? @"testh265" : @"testh264" ofType:@"MOV"];

XDXAVParseHandler *parseHandler = [[XDXAVParseHandler alloc] initWithPath:path];

XDXVideoDecoder *decoder = [[XDXVideoDecoder alloc] init];

decoder.delegate = self;

[parseHandler startParseWithCompletionHandler:^(BOOL isVideoFrame, BOOL isFinish, struct XDXParseVideoDataInfo *videoInfo, struct XDXParseAudioDataInfo *audioInfo) {

if (isFinish) {

[decoder stopDecoder];

return;

}

if (isVideoFrame) {

[decoder startDecodeVideoData:videoInfo];

}

}];

}

• 将解码后数据渲染到屏幕上

注意: 如果数据中含有B帧则需要做一个重排序才能渲染,本例提供两个文件,一个不含B帧的h264类型文件,一个含B帧的h265类型文件.

- (void)getVideoDecodeDataCallback:(CMSampleBufferRef)sampleBuffer {

if (self.isH265File) {

// Note : the first frame not need to sort.

if (self.isDecodeFirstFrame) {

self.isDecodeFirstFrame = NO;

CVPixelBufferRef pix = CMSampleBufferGetImageBuffer(sampleBuffer);

[self.previewView displayPixelBuffer:pix];

}

XDXSortFrameHandler *sortHandler = [[XDXSortFrameHandler alloc] init];

sortHandler.delegate = self;

[sortHandler addDataToLinkList:sampleBuffer];

}else {

CVPixelBufferRef pix = CMSampleBufferGetImageBuffer(sampleBuffer);

[self.previewView displayPixelBuffer:pix];

}

}

​

- (void)getSortedVideoNode:(CMSampleBufferRef)sampleBuffer {

int64_t pts = (int64_t)(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000);

static int64_t lastpts = 0;

NSLog(@"Test marigin - %lld",pts - lastpts);

lastpts = pts;