音视频同步-主要代码逻辑
也就是你追我敢的过程
因为音频格式sample channel rate 都是固定的 所以 某一段音频时间是可以确定的,所以一音频为基准 视频同步到音频
常规做法: 展示一帧视频帧后,得到下一帧视频帧的PTS, 和当前音频的时钟锁一个比较,如果在之前那就立即播放,如果在之后那就设置一个delay ,然后设置进定时器,当定时器超时后,刷新新的视频帧。
av_frame_get_best_effort_timestamp(pFrame)) 可以直接获得一个对应AvFrame 中获得一个合适pts
pts转成秒需要乘以time_base
然后对pts做一定计算 如果可以用就直接 is->video_clock = pts 就行
音视频同步主要具体代码逻辑
在主要结构体里面增加5个参数
audio_clock 当前音频帧时间
video_clock 当前视频帧的下一帧的时间
frame_timer 下次要回调的timer
frame_last_pts 视频上一帧的pts
frame_last_delay 上一帧视频播放的delay
double audio_clock;
double video_clock; ///<pts of last decoded frame / predicted pts of next decoded frame
double frame_timer; //视频帧回调的时间
double frame_last_pts;
double frame_last_delay;
简单逻辑就是!
1 获取 当前真和前面一帧之间的 delay
delay = vp->pts - is->frame_last_pts
2判断delay 并且代替前一步
delay = vp->pts - is->frame_last_pts; /* the pts from last time */
if(delay <= 0 || delay >= 1.0) {
delay = is->frame_last_delay;
}
is->frame_last_delay = delay;
is->frame_last_pts = vp->pts;
3 获得当前音频帧的PTS 和 正在播放帧的下一帧(也就是当前处理帧)计算他们之间的diff
ref_clock = get_audio_clock(is);
diff = vp->pts - ref_clock;
4 sync_threshold 不能让帧小于音频的最小帧率
if(fabs(diff) < AV_NOSYNC_THRESHOLD) 如果前面得到的距离 满足最小帧处理要求那就同步处理
如果diff <= -sync_threshold 那么说明当前帧时间以及过了 delay 设置0 快点播放
如果 diff >= sync_threshold 说明距离播放还有一段时间吧delay 增长
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
if(diff <= -sync_threshold) {
delay = 0;
} else if(diff >= sync_threshold) {
delay = 2 * delay;
}
5 在当前回调时间基础上 加上上面计划好的delay时间
is->frame_timer += delay;
6 计算实际时间中的delay 保证最小值不低于音频帧的帧率
最后传入实际需要时间 去刷新 视频渲染 +0.5是加一个偏执
actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
if(actual_delay < 0.010) {
/* Really it should skip the picture instead */
actual_delay = 0.010;
}
schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;
VideoPicture *vp;
double actual_delay, delay, sync_threshold, ref_clock, diff;
if(is->video_st) {
if(is->pictq_size == 0) {
schedule_refresh(is, 1);
} else {
vp = &is->pictq[is->pictq_rindex];
delay = vp->pts - is->frame_last_pts; /* the pts from last time */
if(delay <= 0 || delay >= 1.0) {
/* if incorrect delay, use previous one */
delay = is->frame_last_delay;
}
/* save for next time */
is->frame_last_delay = delay;
is->frame_last_pts = vp->pts;
/* update delay to sync to audio */
ref_clock = get_audio_clock(is);
diff = vp->pts - ref_clock;
/* Skip or repeat the frame. Take delay into account
FFPlay still doesn't "know if this is the best guess." */
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
if(diff <= -sync_threshold) {
delay = 0;
} else if(diff >= sync_threshold) {
delay = 2 * delay;
}
}
is->frame_timer += delay;
/* computer the REAL delay */
actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
if(actual_delay < 0.010) {
/* Really it should skip the picture instead */
actual_delay = 0.010;
}
schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
/* show the picture! */
video_display(is);
/* update queue for next picture! */
if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
is->pictq_rindex = 0;
}
SDL_LockMutex(is->pictq_mutex);
is->pictq_size--;
SDL_CondSignal(is->pictq_cond);
SDL_UnlockMutex(is->pictq_mutex);
}
} else {
schedule_refresh(is, 100);
}
}