播放（播放方式研究）
在上一篇文章中，我们叙述了直播技术的环境配置（包括服务端nginx，nginx-rtmp-module， ffmpeg, Android编译，ios编译）。从本文开始，我们将叙述播放相关的东西，播放是直播技术中关键的一步，它包括很多技术如：解码，缩放，时间基线选择，缓存队列，画面渲染，声音播放等等。我将分为三个部分为大家讲述整个播放流程；
-
Android
第一部分是基于NativeWindow的视频渲染，主要使用的OpenGL ES2通过传入surface来将视频数据渲染到surface上显示出来。第二部分是基于OpenSL ES来音频播放。第三部分，音视频同步。我们使用的都是android原生自带的一些库来做音视频渲染处理。
-
IOS
同样IOS也分成三个部分，第一部分视频渲染：使用OpenGLES.framework，通过OpenGL来渲染视频画面，第二部分是音频播放，基于AudioToolbox.framework做音频播放；第三部分，视音频同步。
利用原生库可以减少资源的利用，降低内存，提高性能；一般而言，如果不是通晓android、ios的程序员会选择一个统一的视频显示和音频播放库（SDL），这个库可以实现视频显示和音频播。但是增加额外的库意味着资源的浪费和性能的降低。
Android
我们首先带来android端的视频播放功能，我们分成三个部分，1、视频渲染；2、音频播放；3、时间基线（音视频同步）来阐述。
1、视频渲染
ffmpeg为我们提供浏览丰富的编解码类型（ffmpeg所具备编解码能力都是软件编解码，不是指硬件编解码。具体之后文章会详细介绍ffmpeg），视频解码包括flv, mpeg, mov 等；音频包括aac, mp3等。对于整个播放，FFmpeg主要处理流程如下：
<code class="language-C++ hljs scss has-numbering"> <span class="hljs-function">av_register_all()</span>; <span class="hljs-comment">// 注册所有的文件格式和编解码器的库,打开的合适格式的文件上会自动选择相应的编解码库</span>
<span class="hljs-function">avformat_network_init()</span>; <span class="hljs-comment">// 注册网络服务</span>
<span class="hljs-function">avformat_alloc_context()</span>; <span class="hljs-comment">// 分配FormatContext内存，</span>
<span class="hljs-function">avformat_open_input()</span>; <span class="hljs-comment">// 打开输入流，获取头部信息，配合av_close_input_file（）关闭流</span>
<span class="hljs-function">avformat_find_stream_info()</span>; <span class="hljs-comment">// 读取packets，来获取流信息，并在pFormatCtx->streams 填充上正确的信息</span>
<span class="hljs-function">avcodec_find_decoder()</span>; <span class="hljs-comment">// 获取解码器，</span>
<span class="hljs-function">avcodec_open2()</span>; <span class="hljs-comment">// 通过AVCodec来初始化AVCodecContext</span>
<span class="hljs-function">av_read_frame()</span>; <span class="hljs-comment">// 读取每一帧</span>
<span class="hljs-function">avcodec_decode_video2()</span>; <span class="hljs-comment">// 解码帧数据</span>
<span class="hljs-function">avcodec_close()</span>; <span class="hljs-comment">// 关闭编辑器上下文</span>
<span class="hljs-function">avformat_close_input()</span>; <span class="hljs-comment">// 关闭文件流</span></code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul>
我们先来看一段代码：
<code class="language-C++ hljs php has-numbering">av_register_all();
avformat_network_init();
pFormatCtx = avformat_alloc_context();
<span class="hljs-keyword">if</span> (avformat_open_input(&pFormatCtx, pathStr, <span class="hljs-keyword">NULL</span>, <span class="hljs-keyword">NULL</span>) != <span class="hljs-number">0</span>) {
LOGE(<span class="hljs-string">"Couldn't open file: %s\n"</span>, pathStr);
<span class="hljs-keyword">return</span>;
}
<span class="hljs-keyword">if</span> (avformat_find_stream_info(pFormatCtx, &dictionary) < <span class="hljs-number">0</span>) {
LOGE(<span class="hljs-string">"Couldn't find stream information."</span>);
<span class="hljs-keyword">return</span>;
}
av_dump_format(pFormatCtx, <span class="hljs-number">0</span>, pathStr, <span class="hljs-number">0</span>);
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul>
这段代码可以算是初始化FFmpeg，首先注册编解码库，为FormatContext分配内存，调用avformat_open_input打开输入流，获取头部信息，配合avformat_find_stream_info来填充FormatContext中相关内容，av_dump_format这个是dump出流信息。这个信息是这个样子的：
<code class="language-text hljs lasso has-numbering">video infomation：
Input <span class="hljs-variable">#0</span>, flv, from <span class="hljs-string">'rtmp:127.0.0.1:1935/live/steam'</span>:
Metadata:
Server : NGINX RTMP (github<span class="hljs-built_in">.</span>com/sergey<span class="hljs-attribute">-dryabzhinsky</span>/nginx<span class="hljs-attribute">-rtmp</span><span class="hljs-attribute">-module</span>)
displayWidth : <span class="hljs-number">320</span>
displayHeight : <span class="hljs-number">240</span>
fps : <span class="hljs-number">15</span>
profile :
level :
<span class="hljs-built_in">Duration</span>: <span class="hljs-number">00</span>:<span class="hljs-number">00</span>:<span class="hljs-number">00.00</span>, start: <span class="hljs-number">15.400000</span>, bitrate: N/A
Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">0</span>: Video: flv1 (flv), yuv420p, <span class="hljs-number">320</span>x240, <span class="hljs-number">15</span> tbr, <span class="hljs-number">1</span>k tbn, <span class="hljs-number">1</span>k tbc
Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">1</span>: Audio: mp3, <span class="hljs-number">11025</span> Hz, stereo, s16p, <span class="hljs-number">32</span> kb/s</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul>
接下来就是找到视频的解码器上下文AVCodecContext和AVCodec解码器。
<code class="language-C++ hljs java has-numbering"> videoStream = -<span class="hljs-number">1</span>;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < pFormatCtx->nb_streams; i++) {
<span class="hljs-keyword">if</span> (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
videoStream = i;
<span class="hljs-keyword">break</span>;
}
}
<span class="hljs-keyword">if</span> (videoStream == -<span class="hljs-number">1</span>) {
LOGE(<span class="hljs-string">"Didn't find a video stream."</span>);
<span class="hljs-keyword">return</span>;
}
<span class="hljs-javadoc">/**
获取视频解码器上下文和解码器
*/</span>
pVideoCodecCtx = pFormatCtx->streams[videoStream]->codec;
pVideoCodec = avcodec_find_decoder(pVideoCodecCtx->codec_id);
<span class="hljs-keyword">if</span> (pVideoCodec == NULL) {
LOGE(<span class="hljs-string">"pVideoCodec not found."</span>);
<span class="hljs-keyword">return</span>;
}
<span class="hljs-javadoc">/**
获取视频宽度和高度
*/</span>
width = pVideoCodecCtx->width;
height = pVideoCodecCtx->height;
<span class="hljs-javadoc">/**
获取设置NativeWindow buffer属性，
*/</span>
ANativeWindow_setBuffersGeometry(nativeWindow, width, height, WINDOW_FORMAT_RGBA_8888);
LOGD(<span class="hljs-string">"the width is %d, the height is %d"</span>, width, height);
<span class="hljs-comment">// 分配每一帧的内存，pFrame原始帧，pFrameRGB为转换帧</span>
pFrame = avcodec_alloc_frame();
pFrameRGB = avcodec_alloc_frame();
<span class="hljs-keyword">if</span> (pFrameRGB == NULL || pFrameRGB == NULL) {
LOGE(<span class="hljs-string">"Could not allocate video frame."</span>);
<span class="hljs-keyword">return</span>;
}
<span class="hljs-keyword">int</span> numBytes = avpicture_get_size(PIX_FMT_RGBA, pVideoCodecCtx->width,
pVideoCodecCtx->height);
buffer = (uint8_t *) av_malloc(numBytes * sizeof(uint8_t));
<span class="hljs-keyword">if</span> (buffer == NULL) {
LOGE(<span class="hljs-string">"buffer is null"</span>);
<span class="hljs-keyword">return</span>;
}
<span class="hljs-comment">// 填充AVPicture信息</span>
avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGBA,
pVideoCodecCtx->width, pVideoCodecCtx->height);
<span class="hljs-comment">// 获取视频缩放上下文</span>
pSwsCtx = sws_getContext(pVideoCodecCtx->width,
pVideoCodecCtx->height,
pVideoCodecCtx->pix_fmt,
width,
height,
PIX_FMT_RGBA,
SWS_BILINEAR,
NULL,
NULL,
NULL);
<span class="hljs-keyword">int</span> frameFinished = <span class="hljs-number">0</span>;
<span class="hljs-comment">// 循环读取每一帧</span>
<span class="hljs-keyword">while</span> (av_read_frame(pFormatCtx, &packet) >= <span class="hljs-number">0</span>) {
<span class="hljs-keyword">if</span> (packet.stream_index == videoStream) {
<span class="hljs-comment">// 解码每一帧数据</span>
avcodec_decode_video2(pVideoCodecCtx, pFrame, &frameFinished, &packet);
<span class="hljs-keyword">if</span> (frameFinished) {
LOGD(<span class="hljs-string">"av_read_frame"</span>);
<span class="hljs-comment">// 调用NativeWindows展示画面</span>
ANativeWindow_lock(nativeWindow, &windowBuffer, <span class="hljs-number">0</span>);
<span class="hljs-comment">// 缩放视频</span>
sws_scale(pSwsCtx, (uint8_t <span class="hljs-keyword">const</span> * <span class="hljs-keyword">const</span> *)pFrame->data,
pFrame->linesize, <span class="hljs-number">0</span>, pVideoCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
uint8_t * dst = (uint8_t *)windowBuffer.bits;
<span class="hljs-keyword">int</span> dstStride = windowBuffer.stride * <span class="hljs-number">4</span>;
uint8_t * src = (pFrameRGB->data[<span class="hljs-number">0</span>]);
<span class="hljs-keyword">int</span> srcStride = pFrameRGB->linesize[<span class="hljs-number">0</span>];
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> h = <span class="hljs-number">0</span>; h < height; h++) {
memcpy(dst + h * dstStride, src + h * srcStride, srcStride);
}
ANativeWindow_unlockAndPost(nativeWindow);
}
}
av_free_packet(&packet); <span class="hljs-comment">// 释放packet</span>
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li></ul>
从整个解码到缩放，再到渲染到nativeWindow思路非常清晰，当然我们这里没有考虑时间基线，这就意味着播放的时候会很快或者很慢（这个取决于视频的帧率）。我们在第三部分详细讨论时间基线（音视频同步问题）。
2、音频渲染
音频渲染我们同样适用的是android原生的库，OpenSL ES。我们会使用一些主要的接口参数如下：
<code class="language-C++ hljs cs has-numbering"><span class="hljs-comment">// engine interfaces</span> <span class="hljs-keyword">static</span> SLObjectItf engineObject; <span class="hljs-keyword">static</span> SLEngineItf engineEngine; <span class="hljs-comment">// output mix interfaces</span> <span class="hljs-keyword">static</span> SLObjectItf outputMixObject; <span class="hljs-keyword">static</span> SLEnvironmentalReverbItf outputMixEnvironmentalReverb; <span class="hljs-keyword">static</span> SLObjectItf bqPlayerObject; <span class="hljs-keyword">static</span> SLEffectSendItf bqPlayerEffectSend; <span class="hljs-keyword">static</span> SLMuteSoloItf bqPlayerMuteSolo; <span class="hljs-keyword">static</span> SLVolumeItf bqPlayerVolume; <span class="hljs-keyword">static</span> SLPlayItf bqPlayerPlay; <span class="hljs-keyword">static</span> SLAndroidSimpleBufferQueueItf bqPlayerBufferQueue; <span class="hljs-comment">// aux effect on the output mix, used by the buffer queue player</span> <span class="hljs-keyword">const</span> <span class="hljs-keyword">static</span> SLEnvironmentalReverbSettings reverbSettings = SL_I3DL2_ENVIRONMENT_PRESET_STONECORRIDOR;</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li></ul>
这些参数都是音频播放过程使用到的，而整个音频播放的流程主要包括以下几个方面：
<code class="language-C++ hljs coffeescript has-numbering">
<span class="hljs-regexp">//</span>创建播放引擎，初始化接口参数。
bool createEngine() {
SLresult result;
<span class="hljs-regexp">//</span> 创建引擎engineObject
result = slCreateEngine(&engineObject, <span class="hljs-number">0</span>, NULL, <span class="hljs-number">0</span>, NULL, NULL);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span> 实现引擎engineObject
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineObject)</span>-></span>Realize(engineObject, SL_BOOLEAN_FALSE);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span> 获取引擎接口engineEngine
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineObject)</span>-></span>GetInterface(engineObject, SL_IID_ENGINE,
&engineEngine);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span>
<span class="hljs-reserved">const</span> SLInterfaceID ids[<span class="hljs-number">1</span>] = {SL_IID_ENVIRONMENTALREVERB};
<span class="hljs-reserved">const</span> SLboolean req[<span class="hljs-number">1</span>] = {SL_BOOLEAN_FALSE};
<span class="hljs-regexp">//</span> 创建混音器outputMixObject
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineEngine)</span>-></span>CreateOutputMix(engineEngine, &outputMixObject, <span class="hljs-number">1</span>,
ids, req);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span> 实现混音器outputMixObject
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixObject)</span>-></span>Realize(outputMixObject, SL_BOOLEAN_FALSE);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span> 获取混音器接口outputMixEnvironmentalReverb
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixObject)</span>-></span>GetInterface(outputMixObject,
SL_IID_ENVIRONMENTALREVERB,
&outputMixEnvironmentalReverb);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS == result) {
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixEnvironmentalReverb)</span>-></span>SetEnvironmentalReverbProperties(
outputMixEnvironmentalReverb, &reverbSettings);
}
<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>;
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul>
在创建OpenSL ES音频播放引擎的时候，我们主要针对引擎和混音器进行初始化。之后我们会创建音频播放缓冲。
<code class="language-C++ hljs coffeescript has-numbering">bool createBufferQueueAudioPlayer(PlayCallBack callback) {
SLresult result;
SLDataLocator_AndroidSimpleBufferQueue loc_bufq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, <span class="hljs-number">2</span>};
<span class="hljs-regexp">//</span> pcm数据格式
SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM, <span class="hljs-number">1</span>, SL_SAMPLINGRATE_44_1,
SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
SL_SPEAKER_FRONT_CENTER, SL_BYTEORDER_LITTLEENDIAN};
SLDataSource audioSrc = {&loc_bufq, &format_pcm};
SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject};
SLDataSink audioSnk = {&loc_outmix, NULL};
<span class="hljs-reserved">const</span> SLInterfaceID ids[<span class="hljs-number">3</span>] = {SL_IID_BUFFERQUEUE, SL_IID_EFFECTSEND, SL_IID_VOLUME};
<span class="hljs-reserved">const</span> SLboolean req[<span class="hljs-number">3</span>] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};
<span class="hljs-regexp">//</span> 创建音频播放器
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineEngine)</span>-></span>CreateAudioPlayer(engineEngine, &bqPlayerObject,
&audioSrc, &audioSnk, <span class="hljs-number">3</span>, ids, req);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 实现音频播放器（bqPlayerObject）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>Realize(bqPlayerObject, SL_BOOLEAN_FALSE);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerPlay（获取播放器）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_PLAY,
&bqPlayerPlay);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerBufferQueue（缓冲buffer）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_BUFFERQUEUE,
&bqPlayerBufferQueue);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 注册播放回调 callback
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerBufferQueue)</span>-></span>RegisterCallback(bqPlayerBufferQueue,
callback, NULL);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) {
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
}
<span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerEffectSend（音效）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_EFFECTSEND,
&bqPlayerEffectSend);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerVolume（音量）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_VOLUME,
&bqPlayerVolume);
<span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result)
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
<span class="hljs-regexp">//</span> 设置播放bqPlayerPlay状态（暂停）
<span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerPlay)</span>-></span>SetPlayState(bqPlayerPlay, SL_PLAYSTATE_STOPPED);
<span class="hljs-keyword">return</span> result == SL_RESULT_SUCCESS;
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li></ul>
整个音频播放流畅其实看起来也是很简单的，主要分：1、创建实现播放引擎；2、创建实现混音器；3、设置缓冲和pcm格式；4、创建实现播放器；5、获取音频播放器接口；6、获取缓冲buffer；7、注册播放回调；8、获取音效接口；9、获取音量接口；10、获取播放状态接口；
做完这10步，整个音频播放器引擎就创建完毕，接下来就是引擎读取数据播放。
<code class="language-C++ hljs objectivec has-numbering"><span class="hljs-keyword">void</span> playBuffer(<span class="hljs-keyword">void</span> *pBuffer, <span class="hljs-keyword">int</span> size) {
<span class="hljs-comment">// 判断数据可用性</span>
<span class="hljs-keyword">if</span> (pBuffer == <span class="hljs-literal">NULL</span> || size == -<span class="hljs-number">1</span>) {
<span class="hljs-keyword">return</span>;
}
LOGV(<span class="hljs-string">"PlayBuff!"</span>);
<span class="hljs-comment">// 数据存放进bqPlayerBufferQueue中</span>
SLresult result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue,
pBuffer, size);
<span class="hljs-keyword">if</span> (result != SL_RESULT_SUCCESS)
LOGE(<span class="hljs-string">"Play buffer error!"</span>);
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul>
这段代码主要阐述的播放的过程，通过将数据放进bqPlayerBufferQueue，供播放引擎读取播放。记得我们在创建缓冲buffer的时候，注册了一个callback，这个callBack的作用就是通知可以向缓冲队列中添加数据，这个callBack的原型如下：
<code class="hljs lasso has-numbering"><span class="hljs-literal">void</span> videoPlayCallBack(SLAndroidSimpleBufferQueueItf bq, <span class="hljs-literal">void</span> <span class="hljs-subst">*</span>context) {
<span class="hljs-comment">// 添加数据到bqPlayerBufferQueue中，通过调用playBuffer方法。</span>
<span class="hljs-literal">void</span><span class="hljs-subst">*</span> <span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> getData();
int size <span class="hljs-subst">=</span> getDataSize();
playBuffer(<span class="hljs-built_in">data</span>, size);
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul>
这样就循环向缓冲队列中添加数据，可以一直播放。有很多简单的音乐播放器就是基于这种模式设计的。这种方式也很可靠，能够非常清楚的展现整个播放流程。
3、时间基线（音视频同步）
为了能够完整的播放视频和音频，我们需要对一些数据进行整合，包括音视频数据，时间参数等。在ffmpeg源码的ffplayer.c使用就是这种方式，对齐时间基线有三种方式：1、对齐音频时间基线；2、对齐视频时间基线；3、对齐第三方时间基线。为了能够能够清楚展现播放速度控制和音视频同步，特意给出了一个流程图。音视频队列从ffmpeg的av_read_frame中读取每一帧数据到音视频队列中（保存的是AVPacket数据），然后音频队列和视频队列从音视频队列中不停的拿数据，期间两者做一个同步，最后展示到画面和播放音频。
<code class="hljs lasso has-numbering"><span class="hljs-literal">void</span> getPacket() {
struct timespec time;
time<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">=</span> <span class="hljs-number">10</span>;<span class="hljs-comment">//网络不好最多等10秒</span>
time<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">=</span> <span class="hljs-number">0</span>;
struct ThreadMsg msg;
<span class="hljs-keyword">while</span> (<span class="hljs-literal">true</span>) {
memset(<span class="hljs-subst">&</span>msg, <span class="hljs-number">0</span>, sizeof(struct ThreadMsg));
msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> <span class="hljs-built_in">NULL</span>;
<span class="hljs-comment">// 等待网络缓冲</span>
ThreadList<span class="hljs-tag">::queueGet</span>(playInstance<span class="hljs-subst">-></span><span class="hljs-built_in">queue</span>, <span class="hljs-subst">&</span>time, <span class="hljs-subst">&</span>msg);
<span class="hljs-keyword">if</span> (msg<span class="hljs-built_in">.</span>msgtype <span class="hljs-subst">==</span> <span class="hljs-subst">-</span><span class="hljs-number">1</span>) {<span class="hljs-comment">//正常退出</span>
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>);
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>);
break;
}
<span class="hljs-keyword">if</span> (msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span> <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) {
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>);
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>);
playInstance<span class="hljs-subst">-></span>timeout_flag <span class="hljs-subst">=</span> <span class="hljs-number">1</span>;
break;
}
AVPacket <span class="hljs-subst">*</span>packet_p <span class="hljs-subst">=</span> (AVPacket <span class="hljs-subst">*</span>) msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span>;
<span class="hljs-keyword">if</span> (packet_p<span class="hljs-subst">-></span>stream_index <span class="hljs-subst">==</span> playInstance<span class="hljs-subst">-></span>videoState<span class="hljs-subst">-></span>videoStream) {
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, packet_p, <span class="hljs-number">1</span>);
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet_p<span class="hljs-subst">-></span>stream_index <span class="hljs-subst">==</span> playInstance<span class="hljs-subst">-></span>videoState<span class="hljs-subst">-></span>audioStream) {
ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, packet_p, <span class="hljs-number">1</span>);
}
}
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul>
<code class="hljs lasso has-numbering">int queueGet(struct ThreadQueue <span class="hljs-subst">*</span><span class="hljs-built_in">queue</span>, const struct timespec <span class="hljs-subst">*</span>timeout,
struct ThreadMsg <span class="hljs-subst">*</span>msg) {
struct MsgList <span class="hljs-subst">*</span>firstrec;
int ret <span class="hljs-subst">=</span> <span class="hljs-number">0</span>;
struct timespec abstimeout;
<span class="hljs-keyword">if</span> (<span class="hljs-built_in">queue</span> <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span> <span class="hljs-subst">||</span> msg <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) {
<span class="hljs-keyword">return</span> EINVAL;
}
<span class="hljs-keyword">if</span> (timeout) {
struct timeval now;
gettimeofday(<span class="hljs-subst">&</span>now, <span class="hljs-built_in">NULL</span>);
abstimeout<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">=</span> now<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">+</span> timeout<span class="hljs-subst">-></span>tv_sec;
abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">=</span> (now<span class="hljs-built_in">.</span>tv_usec <span class="hljs-subst">*</span> <span class="hljs-number">1000</span>) <span class="hljs-subst">+</span> timeout<span class="hljs-subst">-></span>tv_nsec;
<span class="hljs-keyword">if</span> (abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">>=</span> <span class="hljs-number">1000000000</span>) {
abstimeout<span class="hljs-built_in">.</span>tv_sec<span class="hljs-subst">++</span>;
abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">-=</span> <span class="hljs-number">1000000000</span>;
}
}
pthread_mutex_lock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex);
<span class="hljs-comment">/* Will wait until awakened by a signal or broadcast */</span>
<span class="hljs-keyword">while</span> (<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span> <span class="hljs-subst">&&</span> ret <span class="hljs-subst">!=</span> ETIMEDOUT) { <span class="hljs-comment">//Need to loop to handle spurious wakeups</span>
<span class="hljs-keyword">if</span> (timeout) {
ret <span class="hljs-subst">=</span> pthread_cond_timedwait(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>cond, <span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex, <span class="hljs-subst">&</span>abstimeout);
} <span class="hljs-keyword">else</span> {
pthread_cond_wait(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>cond, <span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex);
}
}
<span class="hljs-keyword">if</span> (ret <span class="hljs-subst">==</span> ETIMEDOUT) {
pthread_mutex_unlock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex);
<span class="hljs-keyword">return</span> ret;
}
firstrec <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first;
<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first<span class="hljs-subst">-></span>next;
<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length<span class="hljs-subst">--</span>;
<span class="hljs-keyword">if</span> (<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) {
<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>last <span class="hljs-subst">=</span> <span class="hljs-built_in">NULL</span>; <span class="hljs-comment">// we know this since we hold the lock</span>
<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length <span class="hljs-subst">=</span> <span class="hljs-number">0</span>;
}
msg<span class="hljs-subst">-></span><span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> firstrec<span class="hljs-subst">-></span>msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span>;
msg<span class="hljs-subst">-></span>msgtype <span class="hljs-subst">=</span> firstrec<span class="hljs-subst">-></span>msg<span class="hljs-built_in">.</span>msgtype;
msg<span class="hljs-subst">-></span>qlength <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length;
release_msglist(<span class="hljs-built_in">queue</span>, firstrec);
pthread_mutex_unlock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex);
<span class="hljs-keyword">return</span> <span class="hljs-number">0</span>;
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li></ul>
我们分三个步骤进行操作。
- 整合数据
整合数据指的是讲音频数据和视频数据添加到相应的队列中，以便播放使用，具体如下；这个数据结构主要是针对音视频中一些基本参数的。我们将利用这些东西做解码，播放速度控制，音视频同步等等。
<code class="hljs d has-numbering"><span class="hljs-keyword">typedef</span> <span class="hljs-keyword">struct</span> VideoState { <span class="hljs-comment">//解码过程中的数据结构</span>
AVFormatContext *pFormatCtx;
AVCodecContext *pVideoCodecCtx; <span class="hljs-comment">// 视频解码器上下文</span>
AVCodecContext *pAudioCodecCtx; <span class="hljs-comment">// 音频解码器上下文</span>
AVCodec *pAudioCodec; <span class="hljs-comment">// 音频解码器</span>
AVCodec *pVideoCodec; <span class="hljs-comment">// 视频解码器</span>
AVFrame *pVideoFrame; <span class="hljs-comment">// 视频为转码帧</span>
AVFrame *pVideoFrameRgba; <span class="hljs-comment">// 视频转码为rgba数据的帧</span>
<span class="hljs-keyword">struct</span> SwsContext *sws_ctx; <span class="hljs-comment">// 视频转码工具</span>
<span class="hljs-keyword">void</span> *pVideobuffer; <span class="hljs-comment">// 视频缓存</span>
AVFrame *pAudioFrame;<span class="hljs-comment">// 音频帧</span>
<span class="hljs-keyword">struct</span> SwrContext *swr_ctx; <span class="hljs-comment">// 音频转码工具</span>
<span class="hljs-keyword">int</span> sample_rate_src; <span class="hljs-comment">//音频采样率</span>
<span class="hljs-keyword">int</span> sample_fmt; <span class="hljs-comment">// 音频采用格式</span>
<span class="hljs-keyword">int</span> sample_layout; <span class="hljs-comment">// 音频通道数</span>
int64_t video_start_time; <span class="hljs-comment">// 视频开始时间</span>
int64_t audio_start_time; <span class="hljs-comment">// 音频时间</span>
<span class="hljs-built_in">double</span> video_time_base; <span class="hljs-comment">// 视频基准时间</span>
<span class="hljs-built_in">double</span> audio_time_base; <span class="hljs-comment">// 音频基准时间</span>
<span class="hljs-keyword">int</span> videoStream; <span class="hljs-comment">// FormatContext.streams中视频帧位置</span>
<span class="hljs-keyword">int</span> audioStream; <span class="hljs-comment">// FormatContext.streams中音频帧位置</span>
} VideoState;
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li></ul>
<code class="hljs cpp has-numbering"><span class="hljs-keyword">typedef</span> <span class="hljs-keyword">struct</span> PlayInstance {
ANativeWindow *window; <span class="hljs-comment">// nativeWindow // 通过传入surface构建</span>
<span class="hljs-keyword">int</span> display_width; <span class="hljs-comment">// 显示宽度</span>
<span class="hljs-keyword">int</span> display_height; <span class="hljs-comment">// 显示高度</span>
<span class="hljs-keyword">int</span> stop; <span class="hljs-comment">// 停止</span>
<span class="hljs-keyword">int</span> timeout_flag; <span class="hljs-comment">// 超时标记</span>
<span class="hljs-keyword">int</span> disable_video;
VideoState *videoState;
<span class="hljs-comment">//队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *<span class="hljs-built_in">queue</span>; <span class="hljs-comment">// 音视频帧队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *video_queue; <span class="hljs-comment">// 视频帧队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *audio_queue; <span class="hljs-comment">// 音频帧队列</span>
} PlayInstance;</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul>
-
播放速度控制
播放速度控制可以通过获取音、视频队列中的数据来控制时间，我们是在单独的线程中从音、视频队列中获取相应的数据。然后根据数据时间基准线进行播放速度控制。这一块结合音视频同步一起实现。 -
音视频同步
下面代码是视频展示过程（在单独线程中做的），我们主要考虑的是延时同步那一块。将音频队列中的数据分类存放的音频队列和视频队列。
<code class="hljs haskell has-numbering">
<span class="hljs-title">void</span> video_thread() {
struct timespec time;
time.tv_sec = <span class="hljs-number">10</span>;//网络不好最多等<span class="hljs-number">10</span>秒
time.tv_nsec = <span class="hljs-number">0</span>;
struct <span class="hljs-type">ThreadMsg</span> msg;
int packet_count = <span class="hljs-number">0</span>;
while (true) {
<span class="hljs-keyword">if</span> (playInstance->stop) {
break;
}
msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span> = <span class="hljs-type">NULL</span>;</span>
// 从视频队列中获取数据，并等待
<span class="hljs-type">ThreadList</span>::queueGet(playInstance->video_queue, &time, &msg);
<span class="hljs-keyword">if</span> (msg.msgtype == -<span class="hljs-number">1</span>) {
break;
}
<span class="hljs-keyword">if</span> (msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span> == <span class="hljs-type">NULL</span>) <span class="hljs-container">{
<span class="hljs-type">LOGE</span>("视频线程空循环\<span class="hljs-title">n</span>");
<span class="hljs-title">break</span>;
}</span></span>
<span class="hljs-type">AVPacket</span> *packet_p = (<span class="hljs-type">AVPacket</span> *) msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>;</span>
<span class="hljs-type">AVPacket</span> pavpacket = *packet_p;
packet_count++;
<span class="hljs-keyword">if</span> (packet_count == <span class="hljs-number">1</span>) {//拿到第一个视频包
playInstance->videoState->video_start_time = av_gettime();
<span class="hljs-type">LOGE</span>(<span class="hljs-string">"视频开始时间 %lld\n"</span>, playInstance->videoState->video_start_time);
}
<span class="hljs-keyword">if</span> (playInstance->disable_video) {
av_free_packet(packet_p);
av_free(msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>);</span>
continue;
}
<span class="hljs-type">ANativeWindow_Buffer</span> windowBuffer;
// 延时同步，控制播放速度。
int64_t pkt_pts = pavpacket.pts;
double show_time = pkt_pts * (playInstance->videoState->video_time_base);
int64_t show_time_micro = show_time * <span class="hljs-number">1000000</span>;
int64_t played_time = av_gettime() - playInstance->videoState->video_start_time;
int64_t delta_time = show_time_micro - played_time;
<span class="hljs-keyword">if</span> (delta_time < -(<span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>)) {
<span class="hljs-type">LOGE</span>(<span class="hljs-string">"视频跳帧\n"</span>);
continue；
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (delta_time > <span class="hljs-number">0</span>) {
av_usleep(delta_time);
}
int frame_finished = <span class="hljs-number">0</span>;
avcodec_decode_video2(playInstance->videoState->pVideoCodecCtx,
playInstance->videoState->pVideoFrame,
&frame_finished, &pavpacket);
<span class="hljs-keyword">if</span> (frame_finished) {
sws_scale(//对解码后的数据进行色彩空间转换，yuv420p 转为rgba8888
playInstance->videoState->sws_ctx,
(uint8_t const *const *) (playInstance->videoState->pVideoFrame)-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span>
(playInstance->videoState->pVideoFrame)->linesize,
<span class="hljs-number">0</span>,
playInstance->videoState->pVideoCodecCtx->height,
playInstance->videoState->pVideoFrameRgba-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span>
playInstance->videoState->pVideoFrameRgba->linesize
);
<span class="hljs-keyword">if</span> (!(playInstance->disable_video) &&
<span class="hljs-type">ANativeWindow_lock</span>(playInstance->window, &windowBuffer, <span class="hljs-type">NULL</span>) < <span class="hljs-number">0</span>) {
<span class="hljs-type">LOGE</span>(<span class="hljs-string">"cannot lock window"</span>);
continue;
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (!playInstance->disable_video) {
uint8_t *dst = (uint8_t *) windowBuffer.bits;
int dstStride = windowBuffer.stride * <span class="hljs-number">4</span>;
uint8_t *src = (playInstance->videoState->pVideoFrameRgba-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0]);</span>
int srcStride = playInstance->videoState->pVideoFrameRgba->linesize[<span class="hljs-number">0</span>];
for (int h = <span class="hljs-number">0</span>; h < playInstance->display_height; h++) {
memcpy(dst + h * dstStride, src + h * srcStride, srcStride);
}
<span class="hljs-type">ANativeWindow_unlockAndPost</span>(playInstance->window);//释放对surface的锁，并且更新对应surface数据进行显示
}
av_free_packet(packet_p);
av_free(msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>);</span>
}
}
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li></ul>
我们主要分析延时同步的那一段代码：
<code class="hljs autohotkey has-numbering">// 延时同步
int64_t pkt_pts = pavpacket.pts<span class="hljs-comment">;</span>
double show_time = pkt_pts * (playInstance->videoState->video_time_base)<span class="hljs-comment">;</span>
int64_t show_time_micro = show_time * <span class="hljs-number">1000000</span><span class="hljs-comment">;</span>
int64_t played_time = av_gettime() - playInstance->videoState->video_start_time<span class="hljs-comment">;</span>
int64_t delt<span class="hljs-built_in">a_time</span> = show_time_micro - played_time<span class="hljs-comment">;</span>
<span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> < -(<span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>)) {
LOGE(<span class="hljs-string">"视频跳帧\n"</span>)<span class="hljs-comment">;</span>
<span class="hljs-keyword">continue</span>；
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> > <span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>) {
av_usleep(delt<span class="hljs-built_in">a_time</span>)<span class="hljs-comment">;</span>
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul>
这段代码主要是音视频同步的，这块采用的是基于第三方时间基准，同步调整音频和视频。在音频处理中也有类似的代码。针对延时做同步。这个基准线是标准时间，通过修改过时间差值来设置跳帧还是等待。当然这里音视频同步实现比较简单，按照正常的使用应该是在一定范围内可以认为是同步的。
由于声音对于人来说比较敏感，一点杂音都能分辨出来，而视频的跳帧相对来说更容易让人接受，这是因为视觉停留的原因。因此，建议采用基于音频时间基准来进行音视频同步。
IOS
IOS相对于android而言在视频渲染上来说可能比android端稍微复杂，因为ios没有像android的surfaceView可以直接进行操作，都是通过OpenGL来绘制画面。因此可能会比较难于理解。而对于音频，IOS可以采用AudioToolbox进行处理。对于ffmpeg相关的东西android和ios是一样的，也主要是一个流程。音视频同步采用的是同一种方案。在此就不在介绍ios的音视频同步问题。
1、视频渲染
前面在android部分已经阐述到使用av_read_frame方法来读取每一帧的信息，这部分ios和android一样，android由于原生支持surfaceView进行绘制渲染，而ios不支持，需要借助opengl来绘制，因此在av_read_frame之后就和android存在区别。主要区别如下代码：
<code class="language-oc hljs objectivec has-numbering">
- (<span class="hljs-built_in">NSArray</span> *) decodeFrames: (<span class="hljs-built_in">CGFloat</span>) minDuration
{
<span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span> &&
_audioStream == -<span class="hljs-number">1</span>)
<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>;
<span class="hljs-keyword">if</span> (_formatCtx == <span class="hljs-literal">nil</span>) {
printf(<span class="hljs-string">"AvFormatContext is nil"</span>);
<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>;
}
<span class="hljs-built_in">NSMutableArray</span> *result = [<span class="hljs-built_in">NSMutableArray</span> array];
AVPacket packet;
<span class="hljs-built_in">CGFloat</span> decodedDuration = <span class="hljs-number">0</span>;
<span class="hljs-built_in">BOOL</span> finished = <span class="hljs-literal">NO</span>;
<span class="hljs-keyword">while</span> (!finished) {
<span class="hljs-keyword">if</span> (av_read_frame(_formatCtx, &packet) < <span class="hljs-number">0</span>) {
_isEOF = <span class="hljs-literal">YES</span>;
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> ==_videoStream) {
<span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>;
<span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) {
<span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>;
<span class="hljs-keyword">int</span> len = avcodec_decode_video2(_videoCodecCtx,
_videoFrame,
&gotframe,
&packet);
<span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) {
LoggerVideo(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode video error, skip packet"</span>);
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (gotframe) {
<span class="hljs-keyword">if</span> (!_disableDeinterlacing &&
_videoFrame->interlaced_frame) {
avpicture_deinterlace((AVPicture*)_videoFrame,
(AVPicture*)_videoFrame,
_videoCodecCtx->pix_fmt,
_videoCodecCtx->width,
_videoCodecCtx->height);
}
KxVideoFrame *frame = [<span class="hljs-keyword">self</span> handleVideoFrame];
<span class="hljs-keyword">if</span> (frame) {
[result addObject:frame];
_position = frame<span class="hljs-variable">.position</span>;
decodedDuration += frame<span class="hljs-variable">.duration</span>;
<span class="hljs-keyword">if</span> (decodedDuration > minDuration)
finished = <span class="hljs-literal">YES</span>;
}
<span class="hljs-keyword">char</span> *buf = (<span class="hljs-keyword">char</span> *)malloc(_videoFrame->width * _videoFrame->height * <span class="hljs-number">3</span> / <span class="hljs-number">2</span>);
AVPicture *pict;
<span class="hljs-keyword">int</span> w, h;
<span class="hljs-keyword">char</span> *y, *u, *v;
pict = (AVPicture *)_videoFrame;<span class="hljs-comment">//这里的frame就是解码出来的AVFrame</span>
w = _videoFrame->width;
h = _videoFrame->height;
y = buf;
u = y + w * h;
v = u + w * h / <span class="hljs-number">4</span>;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h; i++)
memcpy(y + w * i, pict->data[<span class="hljs-number">0</span>] + pict->linesize[<span class="hljs-number">0</span>] * i, w);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++)
memcpy(u + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">1</span>] + pict->linesize[<span class="hljs-number">1</span>] * i, w / <span class="hljs-number">2</span>);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++)
memcpy(v + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">2</span>] + pict->linesize[<span class="hljs-number">2</span>] * i, w / <span class="hljs-number">2</span>);
[myview setVideoSize:_videoFrame->height height:_videoFrame->width];
[myview displayYUV420pData:buf width:_videoFrame->width height:_videoFrame->height];
free(buf);
}
<span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len)
<span class="hljs-keyword">break</span>;
pktSize -= len;
}
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _audioStream) {
<span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>;
<span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) {
<span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>;
<span class="hljs-keyword">int</span> len = avcodec_decode_audio4(_audioCodecCtx,
_audioFrame,
&gotframe,
&packet);
<span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) {
LoggerAudio(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode audio error, skip packet"</span>);
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (gotframe) {
KxAudioFrame * frame = [<span class="hljs-keyword">self</span> handleAudioFrame];
<span class="hljs-keyword">if</span> (frame) {
[result addObject:frame];
<span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span>) {
_position = frame<span class="hljs-variable">.position</span>;
decodedDuration += frame<span class="hljs-variable">.duration</span>;
<span class="hljs-keyword">if</span> (decodedDuration > minDuration)
finished = <span class="hljs-literal">YES</span>;
}
}
}
<span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len)
<span class="hljs-keyword">break</span>;
pktSize -= len;
}
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _artworkStream) {
<span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.size</span>) {
KxArtworkFrame *frame = [[KxArtworkFrame alloc] init];
frame<span class="hljs-variable">.picture</span> = [NSData dataWithBytes:packet<span class="hljs-variable">.data</span> length:packet<span class="hljs-variable">.size</span>];
[result addObject:frame];
}
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _subtitleStream) {
<span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>;
<span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) {
AVSubtitle subtitle;
<span class="hljs-keyword">int</span> gotsubtitle = <span class="hljs-number">0</span>;
<span class="hljs-keyword">int</span> len = avcodec_decode_subtitle2(_subtitleCodecCtx,
&subtitle,
&gotsubtitle,
&packet);
<span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) {
LoggerStream(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode subtitle error, skip packet"</span>);
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (gotsubtitle) {
KxSubtitleFrame *frame = [<span class="hljs-keyword">self</span> handleSubtitle: &subtitle];
<span class="hljs-keyword">if</span> (frame) {
[result addObject:frame];
}
avsubtitle_free(&subtitle);
}
<span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len)
<span class="hljs-keyword">break</span>;
pktSize -= len;
}
}
av_free_packet(&packet);
}
<span class="hljs-keyword">return</span> result;
}
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li><li>185</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li><li>185</li></ul>
这段代码主要描述的是decodeFrames，将视频帧和音频帧单独处理。视频主要代码如下：
<code class="hljs cpp has-numbering"><span class="hljs-keyword">if</span> (packet.stream_index ==_videoStream) {
<span class="hljs-keyword">int</span> pktSize = packet.size;
<span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) {
<span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>;
<span class="hljs-keyword">int</span> len = avcodec_decode_video2(_videoCodecCtx,
_videoFrame,
&gotframe,
&packet);
<span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) {
LoggerVideo(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode video error, skip packet"</span>);
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (gotframe) {
<span class="hljs-keyword">if</span> (!_disableDeinterlacing &&
_videoFrame->interlaced_frame) {
avpicture_deinterlace((AVPicture*)_videoFrame,
(AVPicture*)_videoFrame,
_videoCodecCtx->pix_fmt,
_videoCodecCtx->width,
_videoCodecCtx->height);
}
VideoFrame *frame = [self handleVideoFrame];
<span class="hljs-keyword">if</span> (frame) {
[result addObject:frame];
_position = frame.position;
decodedDuration += frame.duration;
<span class="hljs-keyword">if</span> (decodedDuration > minDuration)
finished = YES;
}
<span class="hljs-keyword">char</span> *buf = (<span class="hljs-keyword">char</span> *)<span class="hljs-built_in">malloc</span>(_videoFrame->width * _videoFrame->height * <span class="hljs-number">3</span> / <span class="hljs-number">2</span>);
AVPicture *pict;
<span class="hljs-keyword">int</span> w, h;
<span class="hljs-keyword">char</span> *y, *u, *v;
pict = (AVPicture *)_videoFrame;<span class="hljs-comment">//这里的frame就是解码出来的AVFrame</span>
w = _videoFrame->width;
h = _videoFrame->height;
y = buf;
u = y + w * h;
v = u + w * h / <span class="hljs-number">4</span>;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h; i++)
<span class="hljs-built_in">memcpy</span>(y + w * i, pict->data[<span class="hljs-number">0</span>] + pict->linesize[<span class="hljs-number">0</span>] * i, w);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++)
<span class="hljs-built_in">memcpy</span>(u + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">1</span>] + pict->linesize[<span class="hljs-number">1</span>] * i, w / <span class="hljs-number">2</span>);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++)
<span class="hljs-built_in">memcpy</span>(v + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">2</span>] + pict->linesize[<span class="hljs-number">2</span>] * i, w / <span class="hljs-number">2</span>);
[myview setVideoSize:_videoFrame->height height:_videoFrame->width];
[myview displayYUV420pData:buf width:_videoFrame->width height:_videoFrame->height];
<span class="hljs-built_in">free</span>(buf);
}
<span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len)
<span class="hljs-keyword">break</span>;
pktSize -= len;
}
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet.stream_index == _audioStream) {
<span class="hljs-keyword">int</span> pktSize = packet.size;
<span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) {
<span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>;
<span class="hljs-keyword">int</span> len = avcodec_decode_audio4(_audioCodecCtx,
_audioFrame,
&gotframe,
&packet);
<span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) {
LoggerAudio(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode audio error, skip packet"</span>);
<span class="hljs-keyword">break</span>;
}
<span class="hljs-keyword">if</span> (gotframe) {
AudioFrame * frame = [self handleAudioFrame];
<span class="hljs-keyword">if</span> (frame) {
[result addObject:frame];
<span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span>) {
_position = frame.position;
decodedDuration += frame.duration;
<span class="hljs-keyword">if</span> (decodedDuration > minDuration)
finished = YES;
}
}
}
<span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len)
<span class="hljs-keyword">break</span>;
pktSize -= len;
}
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li></ul>
这块代码和android的类似，将数据转化为yuv格式，然后拿去渲染，myview是一个View，它主要就是通过OpenGL 将数据渲染成画面。主要代码：
<code class="hljs objectivec has-numbering">- (<span class="hljs-keyword">void</span>)displayYUV420pData:(<span class="hljs-keyword">void</span> *)data width:(<span class="hljs-built_in">NSInteger</span>)w height:(<span class="hljs-built_in">NSInteger</span>)h
{
@synchronized(<span class="hljs-keyword">self</span>)
{
<span class="hljs-keyword">if</span> (w != _videoW || h != _videoH)
{
[<span class="hljs-keyword">self</span> setVideoSize:w height:h];
}
[EAGLContext setCurrentContext:_glContext];
glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXY]);
glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w, h, GL_RED_EXT, GL_UNSIGNED_BYTE, data);
<span class="hljs-comment">//[self debugGlError];</span>
glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXU]);
glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w/<span class="hljs-number">2</span>, h/<span class="hljs-number">2</span>, GL_RED_EXT, GL_UNSIGNED_BYTE, data + w * h);
<span class="hljs-comment">// [self debugGlError];</span>
glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXV]);
glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w/<span class="hljs-number">2</span>, h/<span class="hljs-number">2</span>, GL_RED_EXT, GL_UNSIGNED_BYTE, data + w * h * <span class="hljs-number">5</span> / <span class="hljs-number">4</span>);
<span class="hljs-comment">//[self debugGlError];</span>
[<span class="hljs-keyword">self</span> render];
}
<span class="hljs-preprocessor">#ifdef DEBUG</span>
GLenum err = glGetError();
<span class="hljs-keyword">if</span> (err != GL_NO_ERROR)
{
printf(<span class="hljs-string">"GL_ERROR=======>%d\n"</span>, err);
}
<span class="hljs-keyword">struct</span> timeval nowtime;
gettimeofday(&nowtime, <span class="hljs-literal">NULL</span>);
<span class="hljs-keyword">if</span> (nowtime<span class="hljs-variable">.tv_sec</span> != _time<span class="hljs-variable">.tv_sec</span>)
{
printf(<span class="hljs-string">"视频 %d 帧率: %d\n"</span>, <span class="hljs-keyword">self</span><span class="hljs-variable">.tag</span>, _frameRate);
memcpy(&_time, &nowtime, <span class="hljs-keyword">sizeof</span>(<span class="hljs-keyword">struct</span> timeval));
_frameRate = <span class="hljs-number">1</span>;
}
<span class="hljs-keyword">else</span>
{
_frameRate++;
}
<span class="hljs-preprocessor">#endif</span>
}
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li></ul>
通过OpenGL将数据绘制出来。这就是整个IOS展示画面主要代码。其他的一些东西类似于android。
2、音频渲染
在上面视频渲染的过程中，列出了部分对音频的处理过程。其主要方法为handleAudioFrame，下面代码书handleAudioFrame方法：
<code class="hljs haskell has-numbering">- (<span class="hljs-type">AudioFrame</span> *) handleAudioFrame
{
<span class="hljs-keyword">if</span> (!_audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0])</span>
return nil;
id<<span class="hljs-type">AudioManager</span>> audioManager = [<span class="hljs-type">AudioManager</span> audioManager];
const <span class="hljs-type">NSUInteger</span> numChannels = audioManager.numOutputChannels;
<span class="hljs-type">NSInteger</span> numFrames;
void * audioData;
<span class="hljs-keyword">if</span> (_swrContext) {
const <span class="hljs-type">NSUInteger</span> ratio = <span class="hljs-type">MAX</span>(<span class="hljs-number">1</span>, audioManager.samplingRate / _audioCodecCtx->sample_rate) *
<span class="hljs-type">MAX</span>(<span class="hljs-number">1</span>, audioManager.numOutputChannels / _audioCodecCtx->channels) * <span class="hljs-number">2</span>;
const int bufSize = av_samples_get_buffer_size(<span class="hljs-type">NULL</span>,
audioManager.numOutputChannels,
_audioFrame->nb_samples * ratio,
<span class="hljs-type">AV_SAMPLE_FMT_S16</span>,
<span class="hljs-number">1</span>);
<span class="hljs-keyword">if</span> (!_swrBuffer || _swrBufferSize < bufSize) {
_swrBufferSize = bufSize;
_swrBuffer = realloc(_swrBuffer, _swrBufferSize);
}
<span class="hljs-type">Byte</span> *outbuf[<span class="hljs-number">2</span>] = { _swrBuffer, <span class="hljs-number">0</span> };
numFrames = swr_convert(_swrContext,
outbuf,
_audioFrame->nb_samples * ratio,
(const uint8_t **)_audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span>
_audioFrame->nb_samples);
<span class="hljs-keyword">if</span> (numFrames < <span class="hljs-number">0</span>) {
<span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">0</span>, @<span class="hljs-string">"fail resample audio"</span>);
return nil;
}
//int64_t delay = swr_get_delay(_swrContext, audioManager.samplingRate);
//<span class="hljs-keyword">if</span> (delay > <span class="hljs-number">0</span>)
// <span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">0</span>, @<span class="hljs-string">"resample delay %lld"</span>, delay);
audioData = _swrBuffer;
} <span class="hljs-keyword">else</span> {
<span class="hljs-keyword">if</span> (_audioCodecCtx->sample_fmt != <span class="hljs-type">AV_SAMPLE_FMT_S16</span>) {
<span class="hljs-type">NSAssert</span>(false, @<span class="hljs-string">"bucheck, audio format is invalid"</span>);
return nil;
}
audioData = _audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0];</span>
numFrames = _audioFrame->nb_samples;
}
const <span class="hljs-type">NSUInteger</span> numElements = numFrames * numChannels;
<span class="hljs-type">NSMutableData</span> *<span class="hljs-typedef"><span class="hljs-keyword">data</span> = [<span class="hljs-type">NSMutableData</span> dataWithLength:numElements * sizeof<span class="hljs-container">(<span class="hljs-title">float</span>)</span>];</span>
float scale = <span class="hljs-number">1.0</span> / (float)<span class="hljs-type">INT16_MAX</span> ;
vDSP_vflt16((<span class="hljs-type">SInt16</span> *)audioData, <span class="hljs-number">1</span>, <span class="hljs-typedef"><span class="hljs-keyword">data</span>.mutableBytes, 1, numElements);</span>
vDSP_vsmul(<span class="hljs-typedef"><span class="hljs-keyword">data</span>.mutableBytes, 1, &scale, <span class="hljs-keyword">data</span>.mutableBytes, 1, numElements);</span>
<span class="hljs-type">AudioFrame</span> *frame = [[<span class="hljs-type">AudioFrame</span> alloc] init];
frame.position = av_frame_get_best_effort_timestamp(_audioFrame) * _audioTimeBase;
frame.duration = av_frame_get_pkt_duration(_audioFrame) * _audioTimeBase;
frame.samples = <span class="hljs-typedef"><span class="hljs-keyword">data</span>;</span>
<span class="hljs-keyword">if</span> (frame.duration == <span class="hljs-number">0</span>) {
// sometimes ffmpeg can't determine the duration <span class="hljs-keyword">of</span> audio frame
// especially <span class="hljs-keyword">of</span> wma/wmv format
// so <span class="hljs-keyword">in</span> this <span class="hljs-keyword">case</span> must compute duration
frame.duration = frame.samples.length / (sizeof(float) * numChannels * audioManager.samplingRate);
}
<span class="hljs-preprocessor">#if DEBUG</span>
<span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">2</span>, @<span class="hljs-string">"AFD: %.4f %.4f | %.4f "</span>,
frame.position,
frame.duration,
frame.samples.length / (<span class="hljs-number">8.0</span> * <span class="hljs-number">44100.0</span>));
<span class="hljs-preprocessor">#endif</span>
return frame;
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li></ul>
handleAudioFrame主要方法是将数据组合转换，比如采样率重设，转码等等功能，最后放进一个数组中，最后交给AudioToolbox进行处理。我们接着放下看：
<code class="language-swift hljs objectivec has-numbering">func enableAudio(on: Bool) {
let audioManager = AudioManager<span class="hljs-variable">.audioManager</span>()
<span class="hljs-keyword">if</span> on && decoder<span class="hljs-variable">.validAudio</span> {
<span class="hljs-comment">// 闭包</span>
audioManager<span class="hljs-variable">.outputBlock</span> = {(outData: UnsafeMutablePointer<Float>,numFrames: UInt32, numChannels: UInt32) -> Void in
<span class="hljs-keyword">self</span><span class="hljs-variable">.audioCallbackFillData</span>(outData, numFrames: Int(numFrames), numChannels: Int(numChannels))
}
audioManager<span class="hljs-variable">.play</span>()
}<span class="hljs-keyword">else</span>{
audioManager<span class="hljs-variable">.pause</span>()
audioManager<span class="hljs-variable">.outputBlock</span> = <span class="hljs-literal">nil</span>
}
}
func audioCallbackFillData(outData: UnsafeMutablePointer<Float>, numFrames: Int, numChannels: Int) {
var numFrames = numFrames
var weakOutData = outData
<span class="hljs-keyword">if</span> buffered {
memset(weakOutData, <span class="hljs-number">0</span>, numFrames * numChannels * <span class="hljs-keyword">sizeof</span>(Float))
<span class="hljs-keyword">return</span>
}
autoreleasepool {
<span class="hljs-keyword">while</span> numFrames > <span class="hljs-number">0</span> {
<span class="hljs-keyword">if</span> currentAudioFrame == <span class="hljs-literal">nil</span> {
<span class="hljs-built_in">dispatch_sync</span>(lockQueue) {
<span class="hljs-keyword">if</span> let count = <span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>?<span class="hljs-variable">.count</span> {
<span class="hljs-keyword">if</span> count > <span class="hljs-number">0</span> {
let frame: AudioFrame = <span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>![<span class="hljs-number">0</span>] as! AudioFrame
<span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>!<span class="hljs-variable">.removeObjectAtIndex</span>(<span class="hljs-number">0</span>)
<span class="hljs-keyword">self</span><span class="hljs-variable">.moviePosition</span> = frame<span class="hljs-variable">.position</span>
<span class="hljs-keyword">self</span><span class="hljs-variable">.bufferedDuration</span> -= Float(frame<span class="hljs-variable">.duration</span>)
<span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFramePos</span> = <span class="hljs-number">0</span>
<span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFrame</span> = frame<span class="hljs-variable">.samples</span>
}
}
}
}
<span class="hljs-keyword">if</span> (currentAudioFrame != <span class="hljs-literal">nil</span>) {
let bytes = currentAudioFrame<span class="hljs-variable">.bytes</span> + currentAudioFramePos
let bytesLeft: Int = (currentAudioFrame<span class="hljs-variable">.length</span> - currentAudioFramePos)
let frameSizeOf: Int = numChannels * <span class="hljs-keyword">sizeof</span>(Float)
let bytesToCopy: Int = min(numFrames * frameSizeOf, bytesLeft)
let framesToCopy: Int = bytesToCopy / frameSizeOf
memcpy(weakOutData, bytes, bytesToCopy)
numFrames -= framesToCopy
weakOutData = weakOutData<span class="hljs-variable">.advancedBy</span>(framesToCopy * numChannels)
<span class="hljs-keyword">if</span> bytesToCopy < bytesLeft {
<span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFramePos</span> += bytesToCopy
}
<span class="hljs-keyword">else</span> {
<span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFrame</span> = <span class="hljs-literal">nil</span>
}
}<span class="hljs-keyword">else</span>{
memset(weakOutData, <span class="hljs-number">0</span>, numFrames * numChannels * <span class="hljs-keyword">sizeof</span>(Float))
<span class="hljs-comment">//LoggerStream(1, @"silence audio");</span>
<span class="hljs-keyword">self</span><span class="hljs-variable">.debugAudioStatus</span> = <span class="hljs-number">3</span>
<span class="hljs-keyword">self</span><span class="hljs-variable">.debugAudioStatusTS</span> = <span class="hljs-built_in">NSDate</span>()
<span class="hljs-keyword">break</span>
}
}
}
}
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li></ul>
这是一段Swift代码。在ios采用的是swift+oc+c++混合编译，正好借此熟悉swift于oc和c++的交互。enableAudio主要是创建一个audioManager实例，进行注册回调，和开始播放和暂停服务。audioManager是一个单例。是一个封装AudioToolbox类。下面的代码是激活AudioSession（初始化Audio）和失效AudioSession代码。
<code class="language-oc hljs objectivec has-numbering">- (<span class="hljs-built_in">BOOL</span>) activateAudioSession
{
<span class="hljs-keyword">if</span> (!_activated) {
<span class="hljs-keyword">if</span> (!_initialized) {
<span class="hljs-keyword">if</span> (checkError(AudioSessionInitialize(<span class="hljs-literal">NULL</span>,
kCFRunLoopDefaultMode,
sessionInterruptionListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't initialize audio session"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
_initialized = <span class="hljs-literal">YES</span>;
}
<span class="hljs-keyword">if</span> ([<span class="hljs-keyword">self</span> checkAudioRoute] &&
[<span class="hljs-keyword">self</span> setupAudio]) {
_activated = <span class="hljs-literal">YES</span>;
}
}
<span class="hljs-keyword">return</span> _activated;
}
- (<span class="hljs-keyword">void</span>) deactivateAudioSession
{
<span class="hljs-keyword">if</span> (_activated) {
[<span class="hljs-keyword">self</span> pause];
checkError(AudioUnitUninitialize(_audioUnit),
<span class="hljs-string">"Couldn't uninitialize the audio unit"</span>);
<span class="hljs-comment">/*
fails with error (-10851) ?
checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0,
NULL,
0),
"Couldn't clear the render callback on the audio unit");
*/</span>
checkError(AudioComponentInstanceDispose(_audioUnit),
<span class="hljs-string">"Couldn't dispose the output audio unit"</span>);
checkError(AudioSessionSetActive(<span class="hljs-literal">NO</span>),
<span class="hljs-string">"Couldn't deactivate the audio session"</span>);
checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_AudioRouteChange,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't remove audio session property listener"</span>);
checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_CurrentHardwareOutputVolume,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't remove audio session property listener"</span>);
_activated = <span class="hljs-literal">NO</span>;
}
}
- (<span class="hljs-built_in">BOOL</span>) setupAudio
{
<span class="hljs-comment">// --- Audio Session Setup ---</span>
UInt32 sessionCategory = kAudioSessionCategory_MediaPlayback;
<span class="hljs-comment">//UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord;</span>
<span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_AudioCategory,
<span class="hljs-keyword">sizeof</span>(sessionCategory),
&sessionCategory),
<span class="hljs-string">"Couldn't set audio category"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
<span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't add audio session property listener"</span>))
{
<span class="hljs-comment">// just warning</span>
}
<span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_CurrentHardwareOutputVolume,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't add audio session property listener"</span>))
{
<span class="hljs-comment">// just warning</span>
}
<span class="hljs-comment">// Set the buffer size, this will affect the number of samples that get rendered every time the audio callback is fired</span>
<span class="hljs-comment">// A small number will get you lower latency audio, but will make your processor work harder</span>
<span class="hljs-preprocessor">#if !TARGET_IPHONE_SIMULATOR</span>
Float32 preferredBufferSize = <span class="hljs-number">0.0232</span>;
<span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration,
<span class="hljs-keyword">sizeof</span>(preferredBufferSize),
&preferredBufferSize),
<span class="hljs-string">"Couldn't set the preferred buffer duration"</span>)) {
<span class="hljs-comment">// just warning</span>
}
<span class="hljs-preprocessor">#endif</span>
<span class="hljs-keyword">if</span> (checkError(AudioSessionSetActive(<span class="hljs-literal">YES</span>),
<span class="hljs-string">"Couldn't activate the audio session"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
[<span class="hljs-keyword">self</span> checkSessionProperties];
<span class="hljs-comment">// ----- Audio Unit Setup -----</span>
<span class="hljs-comment">// Describe the output unit.</span>
AudioComponentDescription description = {<span class="hljs-number">0</span>};
description<span class="hljs-variable">.componentType</span> = kAudioUnitType_Output;
description<span class="hljs-variable">.componentSubType</span> = kAudioUnitSubType_RemoteIO;
description<span class="hljs-variable">.componentManufacturer</span> = kAudioUnitManufacturer_Apple;
<span class="hljs-comment">// Get component</span>
AudioComponent component = AudioComponentFindNext(<span class="hljs-literal">NULL</span>, &description);
<span class="hljs-keyword">if</span> (checkError(AudioComponentInstanceNew(component, &_audioUnit),
<span class="hljs-string">"Couldn't create the output audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
UInt32 size;
<span class="hljs-comment">// Check the output stream format</span>
size = <span class="hljs-keyword">sizeof</span>(AudioStreamBasicDescription);
<span class="hljs-keyword">if</span> (checkError(AudioUnitGetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&_outputFormat,
&size),
<span class="hljs-string">"Couldn't get the hardware output stream format"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
_outputFormat<span class="hljs-variable">.mSampleRate</span> = _samplingRate;
<span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&_outputFormat,
size),
<span class="hljs-string">"Couldn't set the hardware output stream format"</span>)) {
<span class="hljs-comment">// just warning</span>
}
_numBytesPerSample = _outputFormat<span class="hljs-variable">.mBitsPerChannel</span> / <span class="hljs-number">8</span>;
_numOutputChannels = _outputFormat<span class="hljs-variable">.mChannelsPerFrame</span>;
LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output bytes per sample: %ld"</span>, _numBytesPerSample);
LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output num channels: %ld"</span>, _numOutputChannels);
<span class="hljs-comment">// Slap a render callback on the unit</span>
AURenderCallbackStruct callbackStruct;
callbackStruct<span class="hljs-variable">.inputProc</span> = renderCallback; <span class="hljs-comment">// 注册回调，这个回调是用来取数据的，也就是</span>
callbackStruct<span class="hljs-variable">.inputProcRefCon</span> = (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>);
<span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&callbackStruct,
<span class="hljs-keyword">sizeof</span>(callbackStruct)),
<span class="hljs-string">"Couldn't set the render callback on the audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
<span class="hljs-keyword">if</span> (checkError(AudioUnitInitialize(_audioUnit),
<span class="hljs-string">"Couldn't initialize the audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;
<span class="hljs-keyword">return</span> <span class="hljs-literal">YES</span>;
}</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li></ul>
正真将数据渲染代码
<code class="hljs mel has-numbering">- (BOOL) renderFrames: (UInt32) numFrames
ioData: (AudioBufferList <span class="hljs-variable">*)</span> ioData
{
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) {
memset(ioData->mBuffers[iBuffer].mData, <span class="hljs-number">0</span>, ioData->mBuffers[iBuffer].mDataByteSize);
}
<span class="hljs-keyword">if</span> (_playing && _outputBlock ) {
<span class="hljs-comment">// Collect data to render from the callbacks</span>
_outputBlock(_outData, numFrames, _numOutputChannels);
<span class="hljs-comment">// Put the rendered data into the output buffer</span>
<span class="hljs-keyword">if</span> (_numBytesPerSample == <span class="hljs-number">4</span>) <span class="hljs-comment">// then we've already got floats</span>
{
<span class="hljs-keyword">float</span> zero = <span class="hljs-number">0.0</span>;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) {
<span class="hljs-keyword">int</span> thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iChannel = <span class="hljs-number">0</span>; iChannel < thisNumChannels; ++iChannel) {
vDSP_vsadd(_outData+iChannel, _numOutputChannels, &zero, (<span class="hljs-keyword">float</span> <span class="hljs-variable">*)</span>ioData->mBuffers[iBuffer].mData, thisNumChannels, numFrames);
}
}
}
<span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (_numBytesPerSample == <span class="hljs-number">2</span>) <span class="hljs-comment">// then we need to convert SInt16 -> Float (and also scale)</span>
{
<span class="hljs-keyword">float</span> <span class="hljs-keyword">scale</span> = (<span class="hljs-keyword">float</span>)INT16_MAX;
vDSP_vsmul(_outData, <span class="hljs-number">1</span>, &<span class="hljs-keyword">scale</span>, _outData, <span class="hljs-number">1</span>, numFrames<span class="hljs-variable">*_numOutputChannels</span>);
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) {
<span class="hljs-keyword">int</span> thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels;
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iChannel = <span class="hljs-number">0</span>; iChannel < thisNumChannels; ++iChannel) {
vDSP_vfix16(_outData+iChannel, _numOutputChannels, (SInt16 <span class="hljs-variable">*)</span>ioData->mBuffers[iBuffer].mData+iChannel, thisNumChannels, numFrames);
}
}
}
}
<span class="hljs-keyword">return</span> noErr;
}
</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul>
总结
本文主要是讲述了ffmpeg实现播放的逻辑，分为android和ios两端，根据两端平台的特性做了相应的处理。在android端采用的是NativeWindow（surface）实现视频播放，OpenSL ES实现音频播放。实现音视频同步的逻辑是基于第三方时间基准线，音频和视频同时调整的方案。在ios端采用的是OpenGL实现视频渲染，AudioToolbox实现音频播放。音视频同步和android采用的是一样。其中两端的ffmpeg逻辑是一致的。在ios端OpenGL实现视频渲染没有重点阐述如何使用OpenGL。这个有兴趣的同学可以自行研究。
备注：整个代码工程等整理之后会发布出来。
最后添加两张播放效果图
-
顶
- 4
-
踩
扫一扫
1505
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
