Live555用做RTSPClient时，利用RTP时间戳进行音视频同步的解决方案(必须有RTCP支持才可行)

http://www.mworkbox.com/wp/work/551.html

先看来自Live555官网的2个常见问题：
问题1：Why do most RTP sessions use separate streams for audio and video? How can a receiving client synchronize these streams?
回答：Sending audio and video in separate RTP streams provides a great deal of flexibility. For example, this makes it possible for a player to receive only the audio stream, but not video (or vice-versa). It would even be possible to have one computer receive and play audio, and a separate computer receive and play video.

These audio and video streams are synchronized using RTCP “Sender Report” (SR) packets – which map each stream’s RTP timestamp to ‘wall clock’ (NTP) time. For more information, see the IETF’s RTP/RTCP specification.

Receivers can then use this mapping to synchronize the incoming RTP streams. The LIVE555 Streaming Media code does this automatically: For subclasses of “RTPSource”, the “presentationTime” parameter that’s passed to the ‘afterGettingFunc’ of “getNextFrame()” (see “liveMedia/include/FramedSource.hh”) will be an accurate, time-synchronized time. (For this to work, you need to have also created a “RTCPInstance” for each RTP source.)

For example, if you use “openRTSP” to receive RTSP/RTP streams, then the contents of each RTP stream (audio and video) are written into separate files. This is done using the “FileSink” class. If you look at the “FileSink::afterGettingFrame()” member function, you’ll notice that there’s a “presentationTime” parameter for each incoming frame. Some other receiver could use the “presentationTime” parameter to synchronize audio and video.

问题2：But I notice that there’s an abrupt change in a stream’s presentation times after the first RTCP “SR” packet has been received. Is this a bug?
回答：No, this is normal, and expected; there’s no bug here. This happens because the first few presentation times – before RTCP synchronization occurs – are just ‘guesses’ made by the receiving code (based on the receiver’s ‘wall clock’ and the RTP timestamp). However, once RTCP synchronization occurs, all subsequent presentation times will be accurate.

This means is that a receiver should be prepared for the fact that the first few presentation times (until RTCP synchronization starts) will not be accurate. The code, however, can check this by calling “RTPSource:: hasBeenSynchronizedUsingRTCP()”. If this returns False, then the presentation times are not accurate, and should not be used for synchronization. However, once the call to returns True, then the presentation times (from then on) will be accurate.

我的心得：
1. Live555中关于写文件时，需要先同步音视频流的例子见Live555的源码QuickTimeFileSink.cpp：

void QuickTimeFileSink
:: afterGettingFrame ( void * clientData,  unsigned packetDataSize,
             unsigned numTruncatedBytes,
             struct timeval presentationTime,
             unsigned  /*durationInMicroseconds*/ )  {
  SubsessionIOState * ioState  =  (SubsessionIOState * )clientData ;
   if  ( !ioState - >syncOK (presentationTime ) )  {
     // Ignore this data:音视频还未同步，忽略这些数据包,syncOK中调用了hasBeenSynchronizedUsingRTCP()
    ioState - >fOurSink. continuePlaying ( ) ;
     return ;
   }
  ...
   ioState - >afterGettingFrame (packetDataSize, presentationTime ) ;
}

2.1 通过Live555源码搜索，查找RTPSource同步标记(fCurPacketHasBeenSynchronizedUsingRTCP)设置的逻辑:
—- hasBeenSynchronizedUsingRTCP Matches (17 in 9 files) —-
MediaSession.cpp (livemedia): if (!rtpSource()->hasBeenSynchronizedUsingRTCP()) {
MultiFramedRTPSource.cpp (livemedia): fPresentationTime, fCurPacketHasBeenSynchronizedUsingRTCP,
RTPSource.cpp (livemedia): fCurPacketHasBeenSynchronizedUsingRTCP(False), fLastReceivedSSRC(0),
RTPSource.hh (livemedia\include): virtual Boolean hasBeenSynchronizedUsingRTCP();
RTPSource.hh (livemedia\include): Boolean fCurPacketHasBeenSynchronizedUsingRTCP;
testRTSPClient.cpp: if (fSubsession.rtpSource() != NULL && !fSubsession.rtpSource()->hasBeenSynchronizedUsingRTCP()) {
—- fHasBeenSyncedUsingRTCP Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
MultiFramedRTPSource.cpp (livemedia): hasBeenSyncedUsingRTCP = fHasBeenSyncedUsingRTCP;
MultiFramedRTPSource.hh (livemedia\include): Boolean fHasBeenSyncedUsingRTCP;
—- assignMiscParams Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): bPacket->assignMiscParams(rtpSeqNo, rtpTimestamp, presentationTime,
—- hasBeenSyncedUsingRTCP Matches (18 in 4 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
RTPSource.cpp (livemedia): resultHasBeenSyncedUsingRTCP = fHasBeenSynchronized;

2.2 真正进行RTCP时间同步标记判断和保存的地方，最终会通过bPacket->assignMiscParams函数，赋值给RTPSource::fCurPacketHasBeenSynchronizedUsingRTCP变量：

void MultiFramedRTPSource :: networkReadHandler1 ( ) 
{
...
     struct timeval presentationTime ;  // computed by:
    Boolean hasBeenSyncedUsingRTCP ;  // computed by:
    receptionStatsDB ( )
      . noteIncomingPacket (rtpSSRC, rtpSeqNo, rtpTimestamp,
              timestampFrequency ( ),
              usableInJitterCalculation, presentationTime,
              hasBeenSyncedUsingRTCP, bPacket - >dataSize ( ) ) ;

     // Fill in the rest of the packet descriptor, and store it:
     struct timeval timeNow ;
    gettimeofday ( &timeNow,  NULL ) ;
    bPacket - >assignMiscParams (rtpSeqNo, rtpTimestamp, presentationTime,
                  hasBeenSyncedUsingRTCP, rtpMarkerBit,
                  timeNow ) ;
...
}

3.结论：参考Live55源码中的QuickTimeFileSink类中关于音视频同步逻辑相关的部分，即可在自己的FileSink或者MemorySink中实现音视频同步逻辑。

其他：研读源码，发现MediaSubsession::getNormalPlayTime函数可以获取当前rtp包的ntp时间，值得一试：
—- curPacketRTPTimestamp( Matches (3 in 2 files) —-
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
RTPSource.hh (livemedia\include): u_int32_t curPacketRTPTimestamp() const { return fCurPacketRTPTimestamp; }
—- getNormalPlayTime Matches (4 in 3 files) —-
MediaSession.cpp (livemedia):double MediaSubsession::getNormalPlayTime(struct timeval const& presentationTime) {
MediaSession.hh (livemedia\include): double getNormalPlayTime(struct timeval const& presentationTime);
MediaSession.hh (livemedia\include): double fNPT_PTS_Offset; // set by “getNormalPlayTime()”; add this to a PTS to get NPT
testRTSPClient.cpp: envir() << “\tNPT: ” << fSubsession.getNormalPlayTime(presentationTime);