对等连接级别框架

PeerConnection Level Framework

对等连接级别框架

API

• Fixture
• Fixture factory function

Documentation

文档

The PeerConnection level framework is designed for end-to-end media quality testing through the PeerConnection level public API. The framework uses the Unified plan API to generate offers/answers during the signaling phase. The framework also wraps the video encoder/decoder and inject it into webrtc::PeerConnection to measure video quality, performing 1:1 frames matching between captured and rendered frames without any extra requirements to input video. For audio quality evaluation the standard GetStats() API from PeerConnection is used.

对等连接级框架是为通过对等连接级公共API进行端到端媒体质量测试而设计的。该框架使用统一计划API在信令发送阶段生成offers/answers。该框架还封装了视频编码器/解码器，并将其注入到webrtc::PeerConnection中，以测量视频质量，在捕获的帧和渲染的帧之间执行1:1的帧匹配，而不需要任何额外的输入视频要求。对于音频质量评估，使用来自PeerConnection的标准GetStats（）API。

The framework API is located in the namespace webrtc::webrtc_pc_e2e.

框架API位于命名空间webrtc::webrtc_pc_e2e中。

Supported features

支持的功能

• Single or bidirectional media in the call
• 通话中的单媒体或双向媒体
• RTC Event log dump per peer
• 每个对等端的RTC事件日志转储
• AEC dump per peer
• 每个对等点的AEC转储
• Compatible with webrtc::TimeController for both real and simulated time
• 与webrtc::TimeController兼容，可用于实时和模拟时间
• Media
• 媒体
  • AV sync
  • AV同步
• Video
• 视频
  • Any amount of video tracks both from caller and callee sides
  • 来自主叫方和被叫方的任意数量的视频跟踪
  • Input video from
  • 输入来自的视频
    • Video generator
    • 视频发生器
    • Specified file
    • 指定的文件
    • Any instance of webrtc::test::FrameGeneratorInterface
    • webrtc::test::FrameGeneratorInterface的任何实例
  • Dumping of captured/rendered video into file
  • 将捕获/渲染的视频转储到文件中
  • Screen sharing
  • 屏幕共享
  • Vp8 simulcast from caller side
  • 来自主叫端的Vp8联播
  • Vp9 SVC from caller side
  • 来自呼叫方的Vp9 SVC
  • Choosing of video codec (name and parameters), having multiple codecs negotiated to support codec-switching testing.
  • 选择视频编解码器（名称和参数），协商多个编解码器以支持编解码器切换测试。
  • FEC (ULP or Flex)
  • Forced codec overshooting (for encoder overshoot emulation on some mobile devices, when hardware encoder can overshoot target bitrate)
  • 强制编解码器超调（用于某些移动设备上的编码器超调仿真，当硬件编码器可能超调目标比特率时）
• Audio
• 音频
  • Up to 1 audio track both from caller and callee sides
  • 来自主叫方和被叫方的最多1个音轨
  • Generated audio
  • 生成的音频
  • Audio from specified file
  • 来自指定文件的音频
  • Dumping of captured/rendered audio into file
  • 将捕获/渲染的音频转储到文件中
  • Parameterizing of cricket::AudioOptions
  • 参数化cricket::AudioOptions
  • Echo emulation
  • 回声模拟
• Injection of various WebRTC components into underlying webrtc::PeerConnection or webrtc::PeerConnectionFactory. You can see the full list here
• ​将各种WebRTC组件注入底层的webrtc::PeerConnection或webrtc::PeerConnectionFactory。可以在这里看到完整的列表
• Scheduling of events, that can happen during the test, for example:
• 测试期间可能发生的事件的计划，例如：
  • Changes in network configuration
  • 网络配置的更改
  • User statistics measurements
  • 用户统计测量
  • Custom defined actions
  • 自定义操作
• User defined statistics reporting via webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter interface
• 通过webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter接口进行用户定义的统计报告

Exported metrics

导出的指标

General

通用

• <peer_name>_connected - peer successfully established connection to remote side
• <peer_name>_connected-peer成功建立到远程端的连接
• cpu_usage - CPU usage excluding video analyzer
• cpu_usage-不包括视频分析器的cpu使用情况
• audio_ahead_ms - Used to estimate how much audio and video is out of sync when the two tracks were from the same source. Stats are polled periodically during a call. The metric represents how much earlier was audio played out on average over the call. If, during a stats poll, video is ahead, then audio_ahead_ms will be equal to 0 for this poll.
• audio_ahead_ms-用于估计当两首曲目来自同一来源时有多少音频和视频不同步。通话期间会定期轮询统计信息。该指标表示通话中平均播放音频的时间。如果在统计调查中，视频领先，那么audio_ahead_ms将等于此调查的0。
• video_ahead_ms - Used to estimate how much audio and video is out of sync when the two tracks were from the same source. Stats are polled periodically during a call. The metric represents how much earlier was video played out on average over the call. If, during a stats poll, audio is ahead, then video_ahead_ms will be equal to 0 for this poll.
• video_ahead_ms-用于估计当两首曲目来自同一来源时，有多少音频和视频不同步。通话期间会定期轮询统计信息。该指标表示通话中平均播放视频的时间。如果在统计调查中，音频领先，那么video_ahead_ms将等于此调查的0。

Video

视频

See documentation for DefaultVideoQualityAnalyzer

​请参阅DefaultVideoQualityAnalyzer的文档

Audio

音频

• accelerate_rate - when playout is sped up, this counter is increased by the difference between the number of samples received and the number of samples played out. If speedup is achieved by removing samples, this will be the count of samples removed. Rate is calculated as difference between nearby samples divided on sample interval.
• accelerate_rate—当播放速度加快时，此计数器将按接收到的样本数和播放的样本数之差增加。如果通过移除样本来实现加速，则这将是移除的样本数。速率计算为附近样本之间的差值除以样本间隔。
• expand_rate - the total number of samples that are concealed samples over time. A concealed sample is a sample that was replaced with synthesized samples generated locally before being played out. Examples of samples that have to be concealed are samples from lost packets or samples from packets that arrive too late to be played out
• expand_rate—一段时间内作为隐藏样本的样本总数。隐藏样本是指在播放之前用本地生成的合成样本替换的样本。必须隐藏的样本的示例是丢失数据包的样本或到达太迟而无法播放的数据包的采样
• speech_expand_rate - the total number of samples that are concealed samples minus the total number of concealed samples inserted that are “silent” over time. Playing out silent samples results in silence or comfort noise.
• speech_expand_rate—作为隐藏样本的样本总数减去随时间“静默”插入的隐藏样本总数。播放静音样本会导致静音或舒适噪音。
• preemptive_rate - when playout is slowed down, this counter is increased by the difference between the number of samples received and the number of samples played out. If playout is slowed down by inserting samples, this will be the number of inserted samples. Rate is calculated as difference between nearby samples divided on sample interval.
• preventive_rate—当播放速度减慢时，此计数器会增加接收到的样本数和播放的样本数之间的差值。如果通过插入样本来减慢播放速度，则这将是插入的样本数。速率计算为附近样本之间的差值除以样本间隔。
• average_jitter_buffer_delay_ms - average size of NetEQ jitter buffer.
• average_jitter_buffer_delay_ms—NetEQ抖动缓冲区的平均大小。
• preferred_buffer_size_ms - preferred size of NetEQ jitter buffer.
• preferred_buffer_size_ms—NetEQ抖动缓冲区的首选大小。
• visqol_mos - proxy for audio quality itself.
• visqol_mos-音频质量本身的代理。
• asdm_samples - measure of how much acceleration/deceleration was in the signal.
• asdm_samples-测量信号中的加速度/减速度。
• word_error_rate - measure of how intelligible the audio was (percent of words that could not be recognized in output audio).
• word_error_rate-衡量音频的可懂度（输出音频中无法识别的单词的百分比）。

Network

网络

• bytes_sent - represents the total number of payload bytes sent on this PeerConnection, i.e., not including headers or padding
• bytes_send-表示在此对等连接上发送的有效负载字节总数，即不包括头或填充
• packets_sent - represents the total number of packets sent over this PeerConnection’s transports.
• packets_sent-表示通过该对等连接的传输发送的数据包总数。
• average_send_rate - average send rate calculated on bytes_sent divided by test duration.
• average_send_rate—根据字节发送除以测试持续时间计算的平均发送速率。
• payload_bytes_sent - total number of bytes sent for all SSRC plus total number of RTP header and padding bytes sent for all SSRC. This does not include the size of transport layer headers such as IP or UDP.
• payload_bytes_send-为所有SSRC发送的字节总数加上为所有SSRC发送的RTP报头和填充字节总数。这不包括传输层头（如IP或UDP）的大小。
• sent_packets_loss - packets_sent minus corresponding packets_received.
• sent_packets_loss-packets_sent减去相应的packets_received。
• bytes_received - represents the total number of bytes received on this PeerConnection, i.e., not including headers or padding.
• bytes_received-表示在此对等连接上接收的字节总数，即不包括标头或填充。
• packets_received - represents the total number of packets received on this PeerConnection’s transports.
• packets_received-表示在该对等连接的传输上接收的数据包总数。
• average_receive_rate - average receive rate calculated on bytes_received divided by test duration.
• average_receive_rate—根据字节数received除以测试持续时间计算的平均接收速率。
• payload_bytes_received - total number of bytes received for all SSRC plus total number of RTP header and padding bytes received for all SSRC. This does not include the size of transport layer headers such as IP or UDP.
• payload_bytes_received-所有SSRC接收的字节总数加上所有SSRC收到的RTP报头和填充字节总数。这不包括传输层标头（如IP或UDP）的大小。

Framework stability

框架稳定性

• frames_in_flight - amount of frames that were captured but wasn‘t seen on receiver in the way that also all frames after also weren’t seen on receiver.
• frames_in_flight-捕获但在接收器上看不到的帧的数量，就像之后的所有帧在接收器上也看不到一样。
• bytes_discarded_no_receiver - total number of bytes that were received on network interfaces related to the peer, but destination port was closed.
• bytes_discarded_no_receiver—在与对等端相关的网络接口上接收到的字节总数，但目标端口已关闭。
• packets_discarded_no_receiver - total number of packets that were received on network interfaces related to the peer, but destination port was closed.
• packets_disscarded_no_receiver—在与对等端相关的网络接口上接收到的但目标端口已关闭的数据包总数。

Examples

示例

Examples can be found in

示例可在中找到

• peer_connection_e2e_smoke_test.cc
• pc_full_stack_tests.cc

Stats plotting

统计数据绘图

Description

描述

Stats plotting provides ability to plot statistic collected during the test. Right now it is used in PeerConnection level framework and give ability to see how video quality metrics changed during test execution.

统计数据绘图提供了绘制测试期间收集的统计数据的能力。目前，它被用于PeerConnection级别的框架，并提供了查看视频质量指标在测试执行过程中如何变化的能力。

Usage

用法

To make any metrics plottable you need:

要使需要的任何指标都可绘制：

1.Collect metric data with SamplesStatsCounter which internally will store all intermediate points and timestamps when these points were added.

​1.使用SamplesStatsCounter收集度量数据，该计数器将在添加这些点时在内部存储所有中间点和时间戳。

2.Then you need to report collected data with webrtc::test::PrintResult(...). By using these method you will also specify name of the plottable metric.

​2.然后，需要使用webrtc::test::PrintResult（…）报告收集的数据。通过使用这些方法，还将指定绘图表度量的名称。

After these steps it will be possible to export your metric for plotting. There are several options how you can do this:

经过这些步骤后，将可以导出度量进行打印。有几个选项可供选择：

1.Use webrtc::TestMain::Create() as main function implementation, for example use test/test_main.cc as main function for your test.

​1.使用webrtc::TestMain::Create（）作为主函数实现，例如使用test/test_main.cc作为测试的主函数。

In such case your binary will have flag --plot, where you can provide a list of metrics, that you want to plot or specify all to plot all available metrics.

在这种情况下，二进制文件将具有flag-plot，可以在其中提供要绘制的度量列表，或者指定all来绘制所有可用的度量。

If --plot is specified, the binary will output metrics data into stdout. Then you need to pipe this stdout into python plotter script rtc_tools/metrics_plotter.py, which will plot data.

​如果指定了--plot，二进制文件将把度量数据输出到stdout。然后，您需要将此stdout管道传输到python绘图仪脚本rtc_tools/metrics_plotter.py中，该脚本将绘制数据。

Examples:

示例：

$ ./out/Default/test_support_unittests \
  --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
  --nologs \
  --plot=all \
| python rtc_tools/metrics_plotter.py

$ ./out/Default/test_support_unittests \
  --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
  --nologs \
  --plot=psnr,ssim \
| python rtc_tools/metrics_plotter.py

Example chart: 

图表示例：

2.Use API from test/testsupport/perf_test.h directly by invoking webrtc::test::PrintPlottableResults(const std::vector<std::string>& desired_graphs) to print plottable metrics to stdout. Then as in previous option you need to pipe result into plotter script.

 ​2.通过调用webrtc::test::PrintPlottableResults(const std::vector<std::string>& desired_graphs)，直接使用test/testsupport/perf_test.h中的API将绘图表度量打印到stdout。然后，和前面的选项一样，您需要将结果管道传输到绘图仪脚本中。