在多媒体应用中,针对Client端的带宽情况,Server端对Video Data要进行不同的处理,当Client带宽比较差时,需要在Server端对不同的视频帧进行是丢弃处理,如在低带宽条件下,对于非关键帧,为了更好的用户体验性和播放的平滑性,可对其进行丢弃,而关键帧,则一般不给予丢弃。下面结合代码,详细分析下Red5是如何根据Client端的带宽进行丢包处理的。
Red5的Video Data丢包处理的代码集中在其PlayEngine 的 PushMessage方法,代码如下:
if (body instanceof VideoData) {
IVideoStreamCodec videoCodec = null;
if (msgIn instanceof IBroadcastScope) {
IBroadcastStream stream = (IBroadcastStream) ((IBroadcastScope) msgIn)
.getAttribute(IBroadcastScope.STREAM_ATTRIBUTE);
if (stream != null && stream.getCodecInfo() != null) {
videoCodec = stream.getCodecInfo().getVideoCodec();
}
}
//dont try to drop frames if video codec is null - related to SN-77
if (videoCodec != null && videoCodec.canDropFrames()) {
if (playlistSubscriberStream.state == State.PAUSED) {
// The subscriber paused the video
log.debug("Dropping packet because we are paused");
videoFrameDropper.dropPacket(rtmpMessage);
return;
}
// Only check for frame dropping if the codec supports it
long pendingVideos = pendingVideoMessages();
if (!videoFrameDropper.canSendPacket(rtmpMessage, pendingVideos)) {
// Drop frame as it depends on other frames that were dropped before.
log.debug("Dropping packet because frame dropper says we cant send it");
return;
}
if (!receiveVideo) {
// The client disabled video or the app doesn't have enough bandwidth
// allowed for this stream.
log.debug("Dropping packet because we cant receive video or token acquire failed");
videoFrameDropper.dropPacket(rtmpMessage);
return;
}
// increment the number of times we had pending video frames sequentially
if (pendingVideos > 1) {
numSequentialPendingVideoFrames++;
} else {
// reset number of sequential pending frames if 1 or 0 are pending.
numSequentialPendingVideoFrames = 0;
}
if (pendingVideos > maxPendingVideoFramesThreshold
|| numSequentialPendingVideoFrames > maxSequentialPendingVideoFrames) {
log.debug("Pending: {} Threshold: {} Sequential: {}", new Object[] { pendingVideos,
maxPendingVideoFramesThreshold, numSequentialPendingVideoFrames });
// We drop because the client has insufficient bandwidth.
long now = System.currentTimeMillis();
if (bufferCheckInterval > 0 && now >= nextCheckBufferUnderrun) {
// Notify client about frame dropping (keyframe)
sendInsufficientBandwidthStatus(currentItem);
nextCheckBufferUnderrun = now + bufferCheckInterval;
}
videoFrameDropper.dropPacket(rtmpMessage);
return;
}
videoFrameDropper.sendPacket(rtmpMessage);
}
}
首先Red5根据Stream的Codec来决定是不是需要启用丢包处理。Red5共定义了三种Codec,分别为ScreenVideo,SorensonVideo和AVCVideo,其中ScreenVideo是用于屏幕截取,Red5的缺省实现是,不可以采取丢包处理,而SorensonVideo和AVCVideo分别用于手机和视频播放,Red5的缺省实现是可以采取丢包处理。如果我们需要在采用ScreenVideo的时候也启用丢包处理,则需要修改ScreenVideo的代码,如下所示:
/** {@inheritDoc} */
public boolean canDropFrames() {
return false;
}
在Red5的代码中,之所有在此处直接返回false,不支持丢包,主要是因为ScreenVideo这种Codec的Video data的Decode依赖于前后帧,如果丢弃了一帧,很可能接下来的好几帧数据都不能正确Decode。当然如果网络非常差的情况下,还是通过修改ScreenVideo的canDropFrames(),使其返回true,则可在低带宽条件下启动丢包处理,只不过这样在播放端就会出现连续的花屏现象。
在判断Codec是否支持Drop frames后,Red5主要根据Mina通道里的pendingVideos的数量来决定是否丢弃Video data。由上面的代码可以得知,pendingVideos 是通过方法pendingVideoMessages()获取的。pendingVideoMessages的返回的是RTMPConnection里面的pendingVideos的值(当往Mina的通道里写入一个),在Red5的代码中,如果pendingVideos超过最大允许的等待帧数,则启动丢包处理:
// increment the number of times we had pending video frames sequentially
if (pendingVideos > 1) {
numSequentialPendingVideoFrames++;
} else {
// reset number of sequential pending frames if 1 or 0 are pending.
numSequentialPendingVideoFrames = 0;
}
if (pendingVideos > maxPendingVideoFramesThreshold
|| numSequentialPendingVideoFrames > maxSequentialPendingVideoFrames) {
log.debug("Pending: {} Threshold: {} Sequential: {}", new Object[] { pendingVideos,
maxPendingVideoFramesThreshold, numSequentialPendingVideoFrames });
// We drop because the client has insufficient bandwidth.
long now = System.currentTimeMillis();
if (bufferCheckInterval > 0 && now >= nextCheckBufferUnderrun) {
// Notify client about frame dropping (keyframe)
sendInsufficientBandwidthStatus(currentItem);
nextCheckBufferUnderrun = now + bufferCheckInterval;
}
videoFrameDropper.dropPacket(rtmpMessage);
return;
}
这里之所以用pendingVideos和numSequentialPendingVideoFrames同时作为是否启用丢包的判断条件,主要是为了保证统计的实时性和准确性。
上面分析了Red5如何判断是否应该丢包,及采取什么样的丢包策略,因此在实际应用的时候可以根据具体的需求,对上述流程进行调节和改动,以适用于不同的应用场景。