ExoPlayer 中的音频时间戳计算

ExoPlayer 中的音频时间戳计算

视频播放 音频视频对齐 有三种方式

1. 以视频为基准
2. 以音频为基准
3. 以系统时钟为基准

一 Audio 时间戳的计算

以音频为基准最为常见， ExoPlayer中也是采用以音频为基准的方式。DefaultAudioSink 负责音频数据的渲染，音频数据的时间戳也是在处理的。DefaultAudioSink 时间戳处理涉及这几个概念。

1. AudioTrackPositionTracker
2. MediaPositionParameters
3. Configuration

1.1 handleBuffer

ExoPlayer 对PCM 数据的处理通过 DefaultAudioSink#handleBuffer 函数，hanleBuffer 函数 第二个参数为presentationTimeUs， 这个时间并不是真正的音频时间戳，

presentationTimeUs=1_000_000_000 + 音频时间戳。

1. 如果第一次调用hanleBuffer， 然后AudioTrack 并没有初始化，会调用 initializeAudioTrack() 函数，在initializeAudioTrack 函数中，初始化AudioTrack, 并且startMediaTimeUsNeedsInit = true;

   DefaultAudioSink.java
     
   public boolean handleBuffer(
       ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount)
       throws InitializationException, WriteException {
   
     if (!isAudioTrackInitialized()) {
       try {
         initializeAudioTrack();
       } catch (InitializationException e) {
         return false;
       }
     }
   
     if (startMediaTimeUsNeedsInit) {
       startMediaTimeUs = max(0, presentationTimeUs);
       startMediaTimeUsNeedsSync = false;
       startMediaTimeUsNeedsInit = false;
   
       applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
     }
   

   DefaultAudioSink.java
   
   
   private void initializeAudioTrack() throws InitializationException {
   
       audioTrack = buildAudioTrack();
       audioSessionId = audioTrack.getAudioSessionId();
       audioTrackPositionTracker.setAudioTrack(
           audioTrack,
           /* isPassthrough= */ configuration.outputMode == OUTPUT_MODE_PASSTHROUGH,
           configuration.outputEncoding,
           configuration.outputPcmFrameSize,
           configuration.bufferSize);
   
       startMediaTimeUsNeedsInit = true;
     }
   

2. initializeAudioTrack 初始化完成后， 在handleBuffer 中判断startMediaTimeUsNeedsInit 为ture, 初始化startMediaTimeUs, 调用applyAudioProcessorPlaybackParametersAndSkipSilence 生成 MediaPositionParameters

3. applyAudioProcessorPlaybackParametersAndSkipSilence 函数中 new 一个MediaPositionParameters 保存当前的mediaTimeUs 和audioTrackPositionUs 。

   private void applyAudioProcessorPlaybackParametersAndSkipSilence(long presentationTimeUs) {
   
     mediaPositionParametersCheckpoints.add(
         new MediaPositionParameters(
             playbackParameters,
             skipSilenceEnabled,
             /* mediaTimeUs= */ max(0, presentationTimeUs),
             /* audioTrackPositionUs= */ configuration.framesToDurationUs(getWrittenFrames())));
     setupAudioProcessors();
   }
   

ExoPlayer 中 DefaultAudioSink 默认负责音频的渲染，获取音频时间戳的函数 getCurrentPositionUs

public long getCurrentPositionUs(boolean sourceEnded) {
    if (!isAudioTrackInitialized() || startMediaTimeUsNeedsInit) {
        return CURRENT_POSITION_NOT_SET;
    }
    long positionUs = audioTrackPositionTracker.getCurrentPositionUs(sourceEnded);
    positionUs = min(positionUs, configuration.framesToDurationUs(getWrittenFrames()));
    return applySkipping(applyMediaPositionParameters(positionUs));
}

1.2 AudioTrackPositionTracker 真正的计算时间戳

AudioTrackPositionTracker用于 DefaultAudioSink 内部计算时间戳，是真正计算时间戳的类。根据android 版本不同AudioTrackPositionTracker 内部有两种计算时间戳的方法，

1. derive a smoothed position by sampling the track’s frame position, 我称之为 平滑偏离平均
2. AudioTrack#getTimestamp(AudioTimestamp timestamp) 计算

在 getCurrentPositionUs 函数中

1. maybeSampleSyncParams 中计算 smoothedPlayheadOffsetUs，

2. 判断是否支持 AudioTimestamp 时间戳计算。

3. 如果支持 AudioTimestamp 时间戳计算

4. 否则使用 smoothedPlayheadOffsetUs 计算时间戳。

1.3 AudioTimestamp 计算时间戳方法

AudioTimestamp 的计算方法，获取AudioTrack 上次采样的播放Frame timestampPositionFrames, 然后

Frame 转换成微秒时间， 计算当前时间和AudioTrack 采样的时间差。播放时间戳就是

positionUs = timestampPositionUs + elapsedSinceTimestampUs

1.4 smoothedPlayheadOffsetUs 计算时间戳方法

smoothedPlayheadOffsetUs 方式中 如果是第一次获取时间戳，smoothedPlayheadOffsetUs 还没有计算，直接使用 audioTrack.getPlaybackHeadPosition() 计算。否则使用平滑偏离平均 方式计算，计算公式为：

positionUs = systemTimeUs + smoothedPlayheadOffsetUs;

smoothedPlayheadOffsetUs 计算公式为：

smoothedPlayheadOffsetUs = (playbackPositionUs - systemTimeUs) +(playbackPositionUs - systemTimeUs) +… /n

positionUs 实际是这么计算的

positionUs = systemTimeUs + avg(playbackPositionUs - systemTimeUs)

最后根据延迟再做个修正

positionUs = max(0, positionUs - latencyUs)

AudioTrackPositionTrackerWrap.java

public long getCurrentPositionUs(boolean sourceEnded) {
  if (Assertions.checkNotNull(this.audioTrack).getPlayState() == PLAYSTATE_PLAYING) {
    maybeSampleSyncParams();
  }

  // If the device supports it, use the playback timestamp from AudioTrack.getTimestamp.
  // Otherwise, derive a smoothed position by sampling the track's frame position.
  long systemTimeUs = System.nanoTime() / 1000;
  long positionUs;
  AudioTimestampPoller audioTimestampPoller = Assertions.checkNotNull(this.audioTimestampPoller);
  boolean useGetTimestampMode = audioTimestampPoller.hasAdvancingTimestamp();
  if (useGetTimestampMode) {
    // Calculate the speed-adjusted position using the timestamp (which may be in the future).
    long timestampPositionFrames = audioTimestampPoller.getTimestampPositionFrames();
    long timestampPositionUs = framesToDurationUs(timestampPositionFrames);
    long elapsedSinceTimestampUs = systemTimeUs - audioTimestampPoller.getTimestampSystemTimeUs();
    elapsedSinceTimestampUs =
        Util.getMediaDurationForPlayoutDuration(elapsedSinceTimestampUs, audioTrackPlaybackSpeed);
    positionUs = timestampPositionUs + elapsedSinceTimestampUs;
  } else {
    if (playheadOffsetCount == 0) {
      // The AudioTrack has started, but we don't have any samples to compute a smoothed position.
      positionUs = getPlaybackHeadPositionUs();
    } else {
      // getPlaybackHeadPositionUs() only has a granularity of ~20 ms, so we base the position off
      // the system clock (and a smoothed offset between it and the playhead position) so as to
      // prevent jitter in the reported positions.
      positionUs = systemTimeUs + smoothedPlayheadOffsetUs;
      Log.i("AudioTrackPositionTracker", "getCurrentPositionUs:"+positionUs);
    }
    
    if (!sourceEnded) {
      positionUs = max(0, positionUs - latencyUs);
    }
  }

  return positionUs;
}

1.5 为什么采用smoothedPlayheadOffsetUs 计算

代码注释中：

// getPlaybackHeadPositionUs() only has a granularity of ~20 ms, so we base the position off
// the system clock (and a smoothed offset between it and the playhead position) so as to
// prevent jitter in the reported positions.

1.6 maybeSampleSyncParams 采样

AudioTimestamp 和smoothedPlayheadOffsetUs 的真正计算在maybeSampleSyncParams 函数中.

采样周期是 MIN_PLAYHEAD_OFFSET_SAMPLE_INTERVAL_US 30ms

private void maybeSampleSyncParams() {
  long playbackPositionUs = getPlaybackHeadPositionUs();

  long systemTimeUs = System.nanoTime() / 1000;
  if (systemTimeUs - lastPlayheadSampleTimeUs >= MIN_PLAYHEAD_OFFSET_SAMPLE_INTERVAL_US) {
    // Take a new sample and update the smoothed offset between the system clock and the playhead.
    playheadOffsets[nextPlayheadOffsetIndex] = playbackPositionUs - systemTimeUs;
    nextPlayheadOffsetIndex = (nextPlayheadOffsetIndex + 1) % MAX_PLAYHEAD_OFFSET_COUNT;
    if (playheadOffsetCount < MAX_PLAYHEAD_OFFSET_COUNT) {
      playheadOffsetCount++;
    }
    lastPlayheadSampleTimeUs = systemTimeUs;
    smoothedPlayheadOffsetUs = 0;
    for (int i = 0; i < playheadOffsetCount; i++) {
      smoothedPlayheadOffsetUs += playheadOffsets[i] / playheadOffsetCount;
    }
  }

  maybePollAndCheckTimestamp(systemTimeUs, playbackPositionUs);
  maybeUpdateLatency(systemTimeUs);
}

1.7 Configuration 获取时间戳

DefaultAudioSink 有一个全局变量 writtenPcmBytes，保存了当前AudioTrack 的handlbuffer 处理的数据大小

Configuration 类保存了当前音频的格式， framesToDurationUs 函数计算出当前的时间戳，这个时间戳通常和从 AudioTrackPositionTracker 获取的时间戳做一个修正。

private static final class Configuration {

  public final Format inputFormat;
  public final int inputPcmFrameSize;
  @OutputMode public final int outputMode;
  public final int outputPcmFrameSize;
  public final int outputSampleRate;
  public final int outputChannelConfig;
  @C.Encoding public final int outputEncoding;
  public final int bufferSize;
  
public long framesToDurationUs(long frameCount) {
  return (frameCount * C.MICROS_PER_SECOND) / outputSampleRate;
}

long positionUs = audioTrackPositionTracker.getCurrentPositionUs(sourceEnded) 后

 positionUs = min(positionUs, configuration.framesToDurationUs(getWrittenFrames()));

和当前handlbuffer 的数据做个比较，取最小值。一般来说 handleBuffer的数据应该大于等于audioTrackPositionTracker.getCurrentPositionUs 获取的数据

二 seek后的时间戳

上面分析了正常起播的情况，seek 后音频时间戳的处理通过 MediaPositionParameters 处理

在 DefaultAudioSink#getCurrentPositionUs AudioTrackPositionTracker 只是计算的是当前AudioTrack 上的时间戳。在Exoplayer 中每次seek 都要通过 DefaultAudioSink#flush 函数重新创建AudioTrack , 这样seek 后getCurrentPositionUs 只能获取seek 后的时间。 因此 ExoPlayer 引入了MediaPositionParameters 修正时间戳。

2.1 重新同步和重建AudioTrack

每次seek 后， AudioRender 需要调用

1. handleDiscontinuity

2. flush 函数。

在 handleDiscontinuity 中 startMediaTimeUsNeedsSync 变量 赋值为 true。

public void handleDiscontinuity() {
  startMediaTimeUsNeedsSync = true;
}

flush 中释放当前的AudioTrack

public void flush() {
  Log.e(TAG, "flush", new Exception("flush"));
  if (isAudioTrackInitialized()) {
    resetSinkStateForFlush();

    if (audioTrackPositionTracker.isPlaying()) {
      audioTrack.pause();
    }

    // AudioTrack.release can take some time, so we call it on a background thread.
    final AudioTrack toRelease = audioTrack;
    audioTrack = null;


    audioTrackPositionTracker.reset();
    releasingConditionVariable.close();
    new Thread("ExoPlayer:AudioTrackReleaseThread") {
      @Override
      public void run() {
        try {
          toRelease.flush();
          toRelease.release();
        } finally {
          releasingConditionVariable.open();
        }
      }
    }.start();
  }
}

2.2 生成MediaPositionParameters

handleBuffer 中如果 startMediaTimeUsNeedsSync 为true, 就会调用applyAudioProcessorPlaybackParametersAndSkipSilence 生成MediaPositionParameters。

handleBuffer：
if (startMediaTimeUsNeedsSync) {

  long adjustmentUs = presentationTimeUs - expectedPresentationTimeUs;
  startMediaTimeUs += adjustmentUs;
  startMediaTimeUsNeedsSync = false;
  applyAudioProcessorPlaybackParametersAndSkipSilence(presentationTimeUs);
  if (listener != null && adjustmentUs != 0) {
    listener.onPositionDiscontinuity();
  }
}

三 时间戳的计算过程

getCurrentPositionUs 函数中对时间戳的处理分为四步：

1. AudioTrackPositionTrackerWrap.getCurrentPositionUs 获取AudioTrack 上播放时间戳positionUs

   1.1 audioTrackPositionTracker获取时间戳第一种方式 audioTrack.getPlaybackHeadPosition()

   1.2 audioTrackPositionTracker获取时间戳第二种方式 AudioTimestamp

   long positionUs = audioTrackPositionTracker.getCurrentPositionUs(sourceEnded);

2. 根据 writtenPcmBytes 计算AudioFrame 的写入值，然后和positionUs 做min 运算，

   positionUs = min(positionUs, configuration.framesToDurationUs(getWrittenFrames()));

3. applyMediaPositionParameters 函数做 seek 后的校验，或者变速播放后的校验

   positionUs == mediaPositionParameters.mediaTimeUs + positionUs -mediaPositionParameters.audioTrackPositionUs

private long applyMediaPositionParameters(long positionUs) {
 long playoutDurationSinceLastCheckpointUs =
     positionUs - mediaPositionParameters.audioTrackPositionUs;
 if (mediaPositionParameters.playbackParameters.equals(PlaybackParameters.DEFAULT)) {
   return mediaPositionParameters.mediaTimeUs + playoutDurationSinceLastCheckpointUs;
}    

4. applySkipping 对AudioProcess 中的数据处理做一下校正

private long applySkipping(long positionUs) {
    return positionUs
        + configuration.framesToDurationUs(audioProcessorChain.getSkippedOutputFrameCount());
}