Android OpenGL ES 学习(十二) - MediaCodec + OpenGL 解析H264视频+滤镜

OpenGL 学习教程
Android OpenGL ES 学习(一) – 基本概念
Android OpenGL ES 学习(二) – 图形渲染管线和GLSL
Android OpenGL ES 学习(三) – 绘制平面图形
Android OpenGL ES 学习(四) – 正交投影
Android OpenGL ES 学习(五) – 渐变色
Android OpenGL ES 学习(六) – 使用 VBO、VAO 和 EBO/IBO 优化程序
Android OpenGL ES 学习(七) – 纹理
Android OpenGL ES 学习(八) –矩阵变换
Android OpenGL ES 学习(九) – 坐标系统和。实现3D效果
Android OpenGL ES 学习(十) – GLSurfaceView 源码解析GL线程以及自定义 EGL
Android OpenGL ES 学习(十一) –渲染YUV视频以及视频抖音特效

代码工程地址： https://github.com/LillteZheng/OpenGLDemo.git
更多音视频，参考：Android 音视频入门/进阶教程

这是OpenGL 最后一篇教程了，待我把C/Jni/Ndk 相关的知识，再深入一遍，再来学习光照等知识。

前面我们学习了OpenGL是如何渲染 YUV 视频的Android OpenGL ES 学习(十一) –渲染YUV视频以及视频抖音特效 ，这一章，我们让OpenGL 与 MediaCodec 结合，实现解析 H264 文件，并实现抖音效果。效果如下：

MediaCodec 为Android 的硬编，在一些快速解码设备，我们都是使用MediaCodec，如果你对 MediaCodec 如何解码不熟悉，可以先阅读
Android 音视频编解码(一) – MediaCodec 初探
Android 音视频编解码(二) – MediaCodec 解码(同步和异步)

实际工作中，我们也会使用 MediaCodec 把其他设备传输过来的码流，通过与 OpenGL 结合，实现解码和滤镜效果，比如投屏，投屏的基础上，加一些滤镜和特效。

OpenGL 与 MediaCodec 结合，需要 OpenGL 提供一个 Surface ，让MediaCodec 把解码出来的 YUV 渲染出来，而这个 Surface 就是 SurfaceTexure

一. 外部纹理 SurfaceTexture

SurfaceTexture 是 Surface 与 OpenGL ES 的结合 ，与传统的纹理(GL_TEXTURE_2D)不同，它有以下特点：

• SurfaceTexture 可以直接 BufferQueue 拿到数据并渲染，在拿到 BufferQueue实例时，会将使用者标志设置成 GRALLOC_USAGE_HW_TEXTURE ，以确保 SurfaceTexture 可以识别缓冲区的数据。
• 与 GL_TEXTURE_2D 不同，需要使用 samplerExternalOES 去识别外部纹理。
• 不能执行与 GL_TEXTURE_2D 相同的操作。

1.1 时间戳和转换

SurfaceTexture实例包括检索时间戳的getTimeStamp()方法和检索变换矩阵的getTransformMatrix()方法。调用updateTexImage()设置时间戳和转换矩阵

• 转换：比如某些情况下，接收端的数据是颠倒的，使用Matrix ，我们可以很容易把画面反转回来。
• 时间戳：这个在相机会用的多，比如相机的每一帧，都需要带一个从捕获时拿到的演示时间戳，通过设置这个属性，我们能保证一致的时间戳。

1.2 数据回调

当你创建了SurfaceTexture ，也会创建一个待消耗的BufferQueue，当生产方(比如 MediaCodec )有新的缓冲数据加入队列，会回调 onFrameAvailable() 方法，表示已经消化了一帧。
当你调用了 updateTexImage() ，会释放当前的缓冲区，并从BufferQueue 拿到最新的缓冲区，这时会调用 EGL 的一些操作，使 GLES 可以将缓冲区作为外部纹理使用，即告知 OpenGL ，当前缓冲区可用，可进行一些操作。

二. 渲染视频

从上面的了解，我们可以得出MediaCodec ， SurfaceTexture 和 OpenGL 结合的关系：

流程如下：

1. 创建 SurfaceTexture，并把OpenGL的纹理 id 给到 SurfaceTexture
2. 创建 MediaCodec，并拿到 SurfaceTexture 的 Surface
3. 当第一次回调 onDrawFrame 时，会调用 updateTexture，刷新 BufferQueue，待 Mediacodec 生产数据时，更新 BufferQueue，会重触发 onDrawFrame ，循环至视频解码结束。

2.1 OpenGL 外部纹理

OpenGL 的外部纹理，使用的是GLES11Ext中的 samplerExternalOES：

uniform samplerExternalOES ourTexture;

   
   

   
   
1

因此，我们的片段着色器可以修改成：

/**
 * 片段着色器
 */
private var FRAGMENT_SHADER = """#version 300 es
    precision mediump float;
    out vec4 FragColor;
    in vec2 vTexture;
    uniform samplerExternalOES ourTexture;
    void main() {
        FragColor = texture(ourTexture,vTexture);
    }
"""

   
   

   
   
1
2
3
4
5
6
7
8
9
10
11
12

纹理的绑定，需要注意的是使用 GLES11Ext ：

GLES30.glGenTextures(1, textures, 0)
GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[0])

//纹理环绕
GLES30.glTexParameteri(
GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GLES30.GL_TEXTURE_WRAP_S,
GLES30.GL_REPEAT
)
GLES30.glTexParameteri(
GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GLES30.GL_TEXTURE_WRAP_T,
GLES30.GL_REPEAT
)

//纹理过滤
GLES30.glTexParameteri(
GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GLES30.GL_TEXTURE_MIN_FILTER,
GLES30.GL_LINEAR
)
GLES30.glTexParameteri(
GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GLES30.GL_TEXTURE_MAG_FILTER,
GLES30.GL_LINEAR
)

//解绑纹理对象
GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0)

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29

然后创建 SurfaceTexture：

 surfaceTexture = SurfaceTexture(textures[0]).apply {
   setDefaultBufferSize(width, height)
   setOnFrameAvailableListener {
   }
}

 
 

 
 
1
2
3
4
5

2.2 与MediaCodec 绑定

Mediacodec 解码H264比较简单，配置解码的属性，使用异步解码即可，不熟悉Mediacodec可以参考：
Android 音视频编解码(一) – MediaCodec 初探
Android 音视频编解码(二) – MediaCodec 解码(同步和异步)

解码代码如下：

/**
 * @author by zhengshaorui 2022/12/26
 * describe：视频解码
 */
class VideoDncoder {
    companion object {
        internal val instance: VideoDncoder by lazy { VideoDncoder() }
        private const val MSG_INIT = 1;
        private const val MSG_QUERY = 2;
        private const val DECODE_NAME = "video/avc"
        private const val TAG = "VideoEncoder"
    }
<span class="token keyword">private</span> <span class="token keyword">var</span> handleThread<span class="token operator">:</span> <span class="token class-name">HandlerThread</span><span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
<span class="token keyword">private</span> <span class="token keyword">var</span> handler<span class="token operator">:</span> <span class="token class-name">Handler</span><span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
<span class="token keyword">private</span> <span class="token keyword">var</span> surface<span class="token operator">:</span> <span class="token class-name">Surface</span><span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
<span class="token keyword">private</span> <span class="token keyword">var</span> decoder<span class="token operator">:</span> <span class="token class-name">MediaCodec</span><span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
<span class="token keyword">private</span> val indexQueue <span class="token operator">=</span> <span class="token class-name">LinkedBlockingDeque</span><span class="token generics"><span class="token punctuation">&lt;</span><span class="token class-name">Int</span><span class="token punctuation">&gt;</span></span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">private</span> val handlerCallback <span class="token operator">=</span> <span class="token class-name">Handler<span class="token punctuation">.</span>Callback</span> <span class="token punctuation">{<!-- --></span> msg <span class="token operator">-&gt;</span>
    when <span class="token punctuation">(</span>msg<span class="token punctuation">.</span>what<span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token constant">MSG_INIT</span> <span class="token operator">-&gt;</span> <span class="token punctuation">{<!-- --></span>
            <span class="token function">configAndStart</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token punctuation">}</span>
        <span class="token constant">MSG_QUERY</span> <span class="token operator">-&gt;</span> <span class="token punctuation">{<!-- --></span>

            <span class="token comment">// handler?.sendEmptyMessageDelayed(MSG_QUERY, 10)</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span>
    <span class="token boolean">false</span>
<span class="token punctuation">}</span>
<span class="token keyword">private</span> <span class="token keyword">var</span> listener<span class="token operator">:</span> <span class="token class-name">IDecoderListener</span><span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
fun <span class="token function">start</span><span class="token punctuation">(</span>surface<span class="token operator">:</span> <span class="token class-name">Surface</span><span class="token punctuation">,</span> iDecoderListener<span class="token operator">:</span> <span class="token class-name">IDecoderListener</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    listener <span class="token operator">=</span> iDecoderListener
    <span class="token keyword">this</span><span class="token punctuation">.</span>surface <span class="token operator">=</span> surface
    <span class="token keyword">if</span> <span class="token punctuation">(</span>handleThread <span class="token operator">==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        handleThread <span class="token operator">=</span> <span class="token class-name">HandlerThread</span><span class="token punctuation">(</span><span class="token string">"VideoEncoder"</span><span class="token punctuation">)</span><span class="token punctuation">.</span>apply <span class="token punctuation">{<!-- --></span>
            <span class="token function">start</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
            handler <span class="token operator">=</span> <span class="token class-name">Handler</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>looper<span class="token punctuation">,</span> handlerCallback<span class="token punctuation">)</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span>
    handler<span class="token operator">?</span><span class="token punctuation">.</span>let <span class="token punctuation">{<!-- --></span>
        it<span class="token punctuation">.</span><span class="token function">removeMessages</span><span class="token punctuation">(</span><span class="token constant">MSG_INIT</span><span class="token punctuation">)</span>
        it<span class="token punctuation">.</span><span class="token function">sendEmptyMessage</span><span class="token punctuation">(</span><span class="token constant">MSG_INIT</span><span class="token punctuation">)</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>

<span class="token comment">/**
 * 喂数据
 */</span>
fun <span class="token function">feedData</span><span class="token punctuation">(</span>buffer<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> offset<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">,</span> length<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    val index <span class="token operator">=</span> indexQueue<span class="token punctuation">.</span><span class="token function">take</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>index <span class="token operator">!=</span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        decoder<span class="token operator">?</span><span class="token punctuation">.</span>let <span class="token punctuation">{<!-- --></span>
            it<span class="token punctuation">.</span><span class="token function">getInputBuffer</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span><span class="token operator">?</span><span class="token punctuation">.</span>apply <span class="token punctuation">{<!-- --></span>
                <span class="token function">clear</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
                val time <span class="token operator">=</span> <span class="token class-name">System</span><span class="token punctuation">.</span><span class="token function">nanoTime</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">1000000</span>
                <span class="token function">put</span><span class="token punctuation">(</span>buffer<span class="token punctuation">,</span> offset<span class="token punctuation">,</span> length<span class="token punctuation">)</span>


                it<span class="token punctuation">.</span><span class="token function">queueInputBuffer</span><span class="token punctuation">(</span>index<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> length<span class="token punctuation">,</span> time<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
            <span class="token punctuation">}</span>

        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>


<span class="token keyword">private</span> fun <span class="token function">configAndStart</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    <span class="token keyword">var</span> width <span class="token operator">=</span> <span class="token function">getRealWidth</span><span class="token punctuation">(</span><span class="token class-name">MainApplication</span><span class="token punctuation">.</span>context<span class="token punctuation">)</span>
    <span class="token keyword">var</span> height <span class="token operator">=</span> <span class="token function">getRealHeight</span><span class="token punctuation">(</span><span class="token class-name">MainApplication</span><span class="token punctuation">.</span>context<span class="token punctuation">)</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">null</span> <span class="token operator">==</span> surface <span class="token operator">||</span> <span class="token operator">!</span>surface<span class="token operator">!</span><span class="token operator">!</span><span class="token punctuation">.</span>isValid <span class="token operator">||</span> width <span class="token operator">&lt;</span> <span class="token number">1</span> <span class="token operator">||</span> height <span class="token operator">&lt;</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token keyword">throw</span>  <span class="token class-name">IllegalArgumentException</span><span class="token punctuation">(</span><span class="token string">"Some argument is invalid"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">d</span><span class="token punctuation">(</span><span class="token constant">TAG</span><span class="token punctuation">,</span> <span class="token string">"configAndStart() called: $width,$height"</span><span class="token punctuation">)</span>
    decoder <span class="token operator">=</span> <span class="token class-name">MediaCodec</span><span class="token punctuation">.</span><span class="token function">createDecoderByType</span><span class="token punctuation">(</span><span class="token constant">DECODE_NAME</span><span class="token punctuation">)</span>
    val format <span class="token operator">=</span> <span class="token class-name">MediaFormat</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    format<span class="token punctuation">.</span><span class="token function">setString</span><span class="token punctuation">(</span><span class="token class-name">MediaFormat</span><span class="token punctuation">.</span><span class="token constant">KEY_MIME</span><span class="token punctuation">,</span> <span class="token constant">DECODE_NAME</span><span class="token punctuation">)</span>
    format<span class="token punctuation">.</span><span class="token function">setInteger</span><span class="token punctuation">(</span><span class="token class-name">MediaFormat</span><span class="token punctuation">.</span><span class="token constant">KEY_WIDTH</span><span class="token punctuation">,</span> width<span class="token punctuation">)</span>
    format<span class="token punctuation">.</span><span class="token function">setInteger</span><span class="token punctuation">(</span><span class="token class-name">MediaFormat</span><span class="token punctuation">.</span><span class="token constant">KEY_HEIGHT</span><span class="token punctuation">,</span> height<span class="token punctuation">)</span>
    decoder<span class="token operator">?</span><span class="token punctuation">.</span>let <span class="token punctuation">{<!-- --></span>
        it<span class="token punctuation">.</span><span class="token function">reset</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
        it<span class="token punctuation">.</span><span class="token function">configure</span><span class="token punctuation">(</span>format<span class="token punctuation">,</span> surface<span class="token punctuation">,</span> <span class="token keyword">null</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
        it<span class="token punctuation">.</span><span class="token function">setCallback</span><span class="token punctuation">(</span>decodeCallback<span class="token punctuation">)</span>
        it<span class="token punctuation">.</span><span class="token function">start</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">d</span><span class="token punctuation">(</span><span class="token constant">TAG</span><span class="token punctuation">,</span> <span class="token string">"解码器启动成功"</span><span class="token punctuation">)</span>
        listener<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">onReady</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
        handler<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">sendEmptyMessage</span><span class="token punctuation">(</span><span class="token constant">MSG_QUERY</span><span class="token punctuation">)</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>

<span class="token keyword">public</span> <span class="token keyword">interface</span> <span class="token class-name">IDecoderListener</span> <span class="token punctuation">{<!-- --></span>
    fun <span class="token function">onReady</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>

<span class="token keyword">private</span> val decodeCallback <span class="token operator">=</span> object <span class="token operator">:</span> <span class="token class-name">MediaCodec<span class="token punctuation">.</span>Callback</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    override fun <span class="token function">onInputBufferAvailable</span><span class="token punctuation">(</span>codec<span class="token operator">:</span> <span class="token class-name">MediaCodec</span><span class="token punctuation">,</span> index<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        indexQueue<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span>
    <span class="token punctuation">}</span>

    override fun <span class="token function">onOutputBufferAvailable</span><span class="token punctuation">(</span>
        codec<span class="token operator">:</span> <span class="token class-name">MediaCodec</span><span class="token punctuation">,</span>
        index<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">,</span>
        info<span class="token operator">:</span> <span class="token class-name">MediaCodec<span class="token punctuation">.</span>BufferInfo</span>
    <span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>

        decoder<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">releaseOutputBuffer</span><span class="token punctuation">(</span>index<span class="token punctuation">,</span> <span class="token boolean">true</span><span class="token punctuation">)</span>
    <span class="token punctuation">}</span>

    override fun <span class="token function">onError</span><span class="token punctuation">(</span>codec<span class="token operator">:</span> <span class="token class-name">MediaCodec</span><span class="token punctuation">,</span> e<span class="token operator">:</span> <span class="token class-name">MediaCodec<span class="token punctuation">.</span>CodecException</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">e</span><span class="token punctuation">(</span><span class="token constant">TAG</span><span class="token punctuation">,</span> <span class="token string">"onError() called with: codec = $codec, e = $e"</span><span class="token punctuation">)</span>
    <span class="token punctuation">}</span>

    override fun <span class="token function">onOutputFormatChanged</span><span class="token punctuation">(</span>codec<span class="token operator">:</span> <span class="token class-name">MediaCodec</span><span class="token punctuation">,</span> format<span class="token operator">:</span> <span class="token class-name">MediaFormat</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token keyword">var</span> width <span class="token operator">=</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token class-name">MediaFormat</span><span class="token punctuation">.</span><span class="token constant">KEY_WIDTH</span><span class="token punctuation">)</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span>format<span class="token punctuation">.</span><span class="token function">containsKey</span><span class="token punctuation">(</span><span class="token string">"crop-left"</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> format<span class="token punctuation">.</span><span class="token function">containsKey</span><span class="token punctuation">(</span><span class="token string">"crop-right"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
            width <span class="token operator">=</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token string">"crop-right"</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span> <span class="token operator">-</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token string">"crop-left"</span><span class="token punctuation">)</span>
        <span class="token punctuation">}</span>
        <span class="token keyword">var</span> height <span class="token operator">=</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token class-name">MediaFormat</span><span class="token punctuation">.</span><span class="token constant">KEY_HEIGHT</span><span class="token punctuation">)</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span>format<span class="token punctuation">.</span><span class="token function">containsKey</span><span class="token punctuation">(</span><span class="token string">"crop-top"</span><span class="token punctuation">)</span> <span class="token operator">&amp;&amp;</span> format<span class="token punctuation">.</span><span class="token function">containsKey</span><span class="token punctuation">(</span><span class="token string">"crop-bottom"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
            height <span class="token operator">=</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token string">"crop-bottom"</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span> <span class="token operator">-</span> format<span class="token punctuation">.</span><span class="token function">getInteger</span><span class="token punctuation">(</span><span class="token string">"crop-top"</span><span class="token punctuation">)</span>
        <span class="token punctuation">}</span>
        <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">d</span><span class="token punctuation">(</span><span class="token constant">TAG</span><span class="token punctuation">,</span> <span class="token string">"视频解码后的宽高：$width,$height"</span><span class="token punctuation">)</span>

    <span class="token punctuation">}</span>

<span class="token punctuation">}</span>

fun <span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    handleThread<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">quitSafely</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    handleThread <span class="token operator">=</span> <span class="token keyword">null</span>
    handler <span class="token operator">=</span> <span class="token keyword">null</span>
    surface<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token keyword">try</span> <span class="token punctuation">{<!-- --></span>
        decoder<span class="token operator">?</span><span class="token punctuation">.</span>let <span class="token punctuation">{<!-- --></span>
            it<span class="token punctuation">.</span><span class="token function">stop</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
            it<span class="token punctuation">.</span><span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>e<span class="token operator">:</span> <span class="token class-name">Exception</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>

}

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75
• 76
• 77
• 78
• 79
• 80
• 81
• 82
• 83
• 84
• 85
• 86
• 87
• 88
• 89
• 90
• 91
• 92
• 93
• 94
• 95
• 96
• 97
• 98
• 99
• 100
• 101
• 102
• 103
• 104
• 105
• 106
• 107
• 108
• 109
• 110
• 111
• 112
• 113
• 114
• 115
• 116
• 117
• 118
• 119
• 120
• 121
• 122
• 123
• 124
• 125
• 126
• 127
• 128
• 129
• 130
• 131
• 132
• 133
• 134
• 135
• 136
• 137
• 138
• 139
• 140
• 141
• 142
• 143

2.3 解析H264文件

接下来就是解析H264文件了，需要注意的是，喂给解码器的数据，要以一帧的结尾，不然会出现数据错乱，花屏的问题，如果你对H264不熟悉，可参考 Android 音视频编解码(三) – 视频编码和H264格式原理讲解

因此，我们读取H264每一帧的数据，然后一帧一帧喂给解码器，H264解析的简单代码如下：

/**
 * @author by zhengshaorui 2022/12/28
 * describe：H264 帧解析类
 */
class H264ParseThread(val inputStream: InputStream, val listener: IFrameListener) : Thread() {
    companion object {
        private const val TAG = "H264Parse"
    <span class="token comment">//一般H264帧大小不超过200k,如果解码失败可以尝试增大这个值</span>
    <span class="token keyword">private</span> <span class="token keyword">const</span> val <span class="token constant">FRAME_MAX_LEN</span> <span class="token operator">=</span> <span class="token number">300</span> <span class="token operator">*</span> <span class="token number">1024</span>
    <span class="token keyword">private</span> <span class="token keyword">const</span> val <span class="token constant">P_FRAME</span> <span class="token operator">=</span> <span class="token number">0x01</span>
    <span class="token keyword">private</span> <span class="token keyword">const</span> val <span class="token constant">I_FRAME</span> <span class="token operator">=</span> <span class="token number">0x05</span>
    <span class="token keyword">private</span> <span class="token keyword">const</span> val <span class="token constant">SPS</span> <span class="token operator">=</span> <span class="token number">0x07</span>
    <span class="token keyword">private</span> <span class="token keyword">const</span> val <span class="token constant">PPS</span> <span class="token operator">=</span> <span class="token number">0x08</span>
<span class="token punctuation">}</span>

<span class="token keyword">private</span> <span class="token keyword">var</span> isFinish <span class="token operator">=</span> <span class="token boolean">false</span>

<span class="token keyword">public</span> <span class="token keyword">interface</span> <span class="token class-name">IFrameListener</span> <span class="token punctuation">{<!-- --></span>
    fun <span class="token function">onLog</span><span class="token punctuation">(</span>msg<span class="token operator">:</span> <span class="token class-name">String</span><span class="token punctuation">)</span>
    fun <span class="token function">onFrame</span><span class="token punctuation">(</span>byteArray<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> offset<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">,</span> count<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>

override fun <span class="token function">run</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    <span class="token keyword">super</span><span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token punctuation">)</span>

    <span class="token keyword">try</span> <span class="token punctuation">{<!-- --></span>
        isFinish <span class="token operator">=</span> <span class="token boolean">false</span>
        val header <span class="token operator">=</span> <span class="token class-name">ByteArray</span><span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">)</span>
        val formatLength <span class="token operator">=</span> <span class="token function">getHeaderFormatLength</span><span class="token punctuation">(</span>header<span class="token punctuation">,</span> inputStream<span class="token punctuation">)</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span>formatLength <span class="token operator">&lt;</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
            listener<span class="token punctuation">.</span><span class="token function">onLog</span><span class="token punctuation">(</span><span class="token string">"不符合H264文件规范: $formatLength"</span><span class="token punctuation">)</span>
            <span class="token keyword">return</span>
        <span class="token punctuation">}</span>
        <span class="token comment">//帧数组</span>
        val frame <span class="token operator">=</span> <span class="token class-name">ByteArray</span><span class="token punctuation">(</span><span class="token constant">FRAME_MAX_LEN</span><span class="token punctuation">)</span>
        <span class="token comment">//每次读取的数据</span>
        val readData <span class="token operator">=</span> <span class="token class-name">ByteArray</span><span class="token punctuation">(</span><span class="token number">2</span> <span class="token operator">*</span> <span class="token number">1024</span><span class="token punctuation">)</span>
        <span class="token comment">//把头部信息给到 frame</span>
        <span class="token class-name">System</span><span class="token punctuation">.</span><span class="token function">arraycopy</span><span class="token punctuation">(</span>header<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> frame<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> header<span class="token punctuation">.</span>size<span class="token punctuation">)</span>

        <span class="token comment">//开始肯定是 sps，所以，帧的起始位置为0,由于前面读取了头部，偏移量为4</span>
        <span class="token keyword">var</span> frameLen <span class="token operator">=</span> <span class="token number">4</span>

        <span class="token keyword">while</span> <span class="token punctuation">(</span><span class="token operator">!</span>isFinish<span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
            val readLen <span class="token operator">=</span> inputStream<span class="token punctuation">.</span><span class="token function">read</span><span class="token punctuation">(</span>readData<span class="token punctuation">)</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span>readLen <span class="token operator">&lt;</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
                <span class="token comment">//文件末尾</span>
                listener<span class="token punctuation">.</span><span class="token function">onLog</span><span class="token punctuation">(</span><span class="token string">"文件末尾，退出"</span><span class="token punctuation">)</span>
                isFinish <span class="token operator">=</span> <span class="token boolean">true</span>
                <span class="token keyword">return</span>
            <span class="token punctuation">}</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span>frameLen <span class="token operator">+</span> readLen <span class="token operator">&gt;</span> <span class="token constant">FRAME_MAX_LEN</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
                <span class="token comment">//文件末尾</span>
                listener<span class="token punctuation">.</span><span class="token function">onLog</span><span class="token punctuation">(</span><span class="token string">"文件末尾，大于预留数组，退出"</span><span class="token punctuation">)</span>
                isFinish <span class="token operator">=</span> <span class="token boolean">true</span>
                <span class="token keyword">return</span>
            <span class="token punctuation">}</span>
            <span class="token comment">//先把数据拷贝到帧数组</span>
            <span class="token class-name">System</span><span class="token punctuation">.</span><span class="token function">arraycopy</span><span class="token punctuation">(</span>readData<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> frame<span class="token punctuation">,</span> frameLen<span class="token punctuation">,</span> readLen<span class="token punctuation">)</span>
            <span class="token comment">//修改当前帧的大小</span>
            frameLen <span class="token operator">+=</span> readLen
            <span class="token comment">//寻找第一帧</span>
            <span class="token keyword">var</span> firstHeadIndex <span class="token operator">=</span> <span class="token function">findHeader</span><span class="token punctuation">(</span>frame<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> frameLen<span class="token punctuation">)</span>
            <span class="token keyword">while</span> <span class="token punctuation">(</span>firstHeadIndex <span class="token operator">&gt;=</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
                <span class="token comment">//找第二帧,从第一帧之后的间隔开始找</span>
                val secondFrameIndex <span class="token operator">=</span>
                    <span class="token function">findHeader</span><span class="token punctuation">(</span>frame<span class="token punctuation">,</span> firstHeadIndex <span class="token operator">+</span> <span class="token number">100</span><span class="token punctuation">,</span> frameLen<span class="token punctuation">)</span>
                <span class="token keyword">if</span> <span class="token punctuation">(</span>secondFrameIndex <span class="token operator">&gt;</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
                    <span class="token comment">//找到第二帧</span>
                    listener<span class="token punctuation">.</span><span class="token function">onFrame</span><span class="token punctuation">(</span>frame<span class="token punctuation">,</span> firstHeadIndex<span class="token punctuation">,</span> secondFrameIndex <span class="token operator">-</span> firstHeadIndex<span class="token punctuation">)</span>
                    <span class="token comment">//把第二帧的数组数据，拷贝到前面，方便继续寻找下一帧</span>
                    val temp <span class="token operator">=</span> frame<span class="token punctuation">.</span><span class="token function">copyOfRange</span><span class="token punctuation">(</span>secondFrameIndex<span class="token punctuation">,</span> frameLen<span class="token punctuation">)</span>
                    <span class="token class-name">System</span><span class="token punctuation">.</span><span class="token function">arraycopy</span><span class="token punctuation">(</span>temp<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> frame<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> temp<span class="token punctuation">.</span>size<span class="token punctuation">)</span>
                    <span class="token comment">//帧下表指向第二帧的数据</span>
                    frameLen <span class="token operator">=</span> temp<span class="token punctuation">.</span>size

                    <span class="token comment">//继续寻找下一帧</span>
                    firstHeadIndex <span class="token operator">=</span> <span class="token function">findHeader</span><span class="token punctuation">(</span>frame<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> frameLen<span class="token punctuation">)</span>
                <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{<!-- --></span>
                    <span class="token comment">//没有找到，继续循环去找</span>
                    firstHeadIndex <span class="token operator">=</span> <span class="token operator">-</span><span class="token number">1</span>
                <span class="token punctuation">}</span>


            <span class="token punctuation">}</span>


        <span class="token punctuation">}</span>

    <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>e<span class="token operator">:</span> <span class="token class-name">Exception</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        listener<span class="token punctuation">.</span><span class="token function">onLog</span><span class="token punctuation">(</span><span class="token string">"read file fail: $e"</span><span class="token punctuation">)</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>

fun <span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
    isFinish <span class="token operator">=</span> <span class="token boolean">true</span>
<span class="token punctuation">}</span>

<span class="token keyword">private</span> fun <span class="token function">findHeader</span><span class="token punctuation">(</span>data<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> offset<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">,</span> count<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token class-name">Int</span> <span class="token punctuation">{<!-- --></span>
    <span class="token keyword">for</span> <span class="token punctuation">(</span>i in offset until count<span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">isFrameHeader</span><span class="token punctuation">(</span>data<span class="token punctuation">,</span> i<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
            <span class="token keyword">return</span> i
        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span>
    <span class="token keyword">return</span> <span class="token operator">-</span><span class="token number">1</span>

<span class="token punctuation">}</span>

<span class="token keyword">private</span> fun <span class="token function">isFrameHeader</span><span class="token punctuation">(</span>data<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> index<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token class-name">Boolean</span> <span class="token punctuation">{<!-- --></span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>data<span class="token punctuation">.</span>size <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token keyword">return</span> <span class="token boolean">false</span>
    <span class="token punctuation">}</span>
    val d1 <span class="token operator">=</span> data<span class="token punctuation">[</span>index<span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span>
    val d2 <span class="token operator">=</span> data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span>
    val isNaluHeader <span class="token operator">=</span> d1 <span class="token operator">&amp;&amp;</span> d2
    <span class="token keyword">if</span> <span class="token punctuation">(</span>isNaluHeader <span class="token operator">&amp;&amp;</span> data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">1</span> <span class="token operator">&amp;&amp;</span> <span class="token function">isFrameHeadType</span><span class="token punctuation">(</span>data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">{<!-- --></span>
        <span class="token keyword">return</span> <span class="token boolean">true</span>
    <span class="token punctuation">}</span><span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>isNaluHeader <span class="token operator">&amp;&amp;</span> data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">1</span> <span class="token operator">&amp;&amp;</span> <span class="token function">isFrameHeadType</span><span class="token punctuation">(</span>data<span class="token punctuation">[</span>index <span class="token operator">+</span> <span class="token number">4</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">{<!-- --></span>
        <span class="token keyword">return</span> <span class="token boolean">true</span>
    <span class="token punctuation">}</span>

    <span class="token keyword">return</span> <span class="token boolean">false</span>
<span class="token punctuation">}</span>

<span class="token comment">/**
 * 解析的时候，找到I和P去解析即可
 * 为啥使用and这个会导致播放卡顿？有大佬可以解释一下吗
 */</span>
<span class="token keyword">private</span> fun <span class="token function">isSpecialFrame</span><span class="token punctuation">(</span><span class="token keyword">byte</span><span class="token operator">:</span> <span class="token class-name">Byte</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token class-name">Boolean</span> <span class="token punctuation">{<!-- --></span>
    val type <span class="token operator">=</span> <span class="token keyword">byte</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span> and <span class="token number">0x11</span>
    <span class="token keyword">return</span>  type <span class="token operator">==</span> <span class="token constant">P_FRAME</span> <span class="token operator">||</span> type <span class="token operator">==</span> <span class="token constant">I_FRAME</span> <span class="token operator">||</span> type <span class="token operator">==</span> <span class="token constant">SPS</span> <span class="token operator">||</span> type <span class="token operator">==</span> <span class="token constant">PPS</span>
<span class="token punctuation">}</span>

<span class="token comment">/**
 * 65    --   I帧/IDR帧
 * 41/61 --   p帧
 * 67    --   sps
 * 68    --   pps
 *
 */</span>
fun <span class="token function">isFrameHeadType</span><span class="token punctuation">(</span>head<span class="token operator">:</span> <span class="token class-name">Byte</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token class-name">Boolean</span> <span class="token punctuation">{<!-- --></span>
    <span class="token comment">// val type = byte.toInt() and 0x11</span>
    <span class="token keyword">return</span> head <span class="token operator">==</span> <span class="token number">0x65</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">||</span> head <span class="token operator">==</span> <span class="token number">0x61</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
            <span class="token operator">||</span> head <span class="token operator">==</span> <span class="token number">0x41</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">||</span> head <span class="token operator">==</span> <span class="token number">0x67</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
            <span class="token operator">||</span> head <span class="token operator">==</span> <span class="token number">0x68</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">||</span> head <span class="token operator">==</span> <span class="token number">0x06</span><span class="token punctuation">.</span><span class="token function">toByte</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>

<span class="token keyword">private</span> fun <span class="token function">getHeaderFormatLength</span><span class="token punctuation">(</span>header<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> inputStream<span class="token operator">:</span> <span class="token class-name">InputStream</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token class-name">Int</span> <span class="token punctuation">{<!-- --></span>
    <span class="token comment">//先读取头部4个字节，判断h264 是哪种格式</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>header<span class="token punctuation">.</span>size <span class="token operator">&lt;</span> <span class="token number">4</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token keyword">return</span> <span class="token operator">-</span><span class="token number">1</span>
    <span class="token punctuation">}</span>
    inputStream<span class="token punctuation">.</span><span class="token function">read</span><span class="token punctuation">(</span>header<span class="token punctuation">)</span>
    val h1 <span class="token operator">=</span> header<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    val h2 <span class="token operator">=</span> header<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    val h3 <span class="token operator">=</span> header<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    val h4 <span class="token operator">=</span> header<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">toInt</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token comment">//码流格式类型00 00 01 或者 00 00 00 01</span>
    <span class="token keyword">return</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>h1 <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> h2 <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> h3 <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token number">3</span>
    <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>h1 <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> h2 <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> h3 <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> h4 <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        <span class="token number">4</span>
    <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{<!-- --></span>
        <span class="token comment">//不符合H264文件规范</span>
        <span class="token operator">-</span><span class="token number">1</span>
    <span class="token punctuation">}</span>

<span class="token punctuation">}</span>

}

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75
• 76
• 77
• 78
• 79
• 80
• 81
• 82
• 83
• 84
• 85
• 86
• 87
• 88
• 89
• 90
• 91
• 92
• 93
• 94
• 95
• 96
• 97
• 98
• 99
• 100
• 101
• 102
• 103
• 104
• 105
• 106
• 107
• 108
• 109
• 110
• 111
• 112
• 113
• 114
• 115
• 116
• 117
• 118
• 119
• 120
• 121
• 122
• 123
• 124
• 125
• 126
• 127
• 128
• 129
• 130
• 131
• 132
• 133
• 134
• 135
• 136
• 137
• 138
• 139
• 140
• 141
• 142
• 143
• 144
• 145
• 146
• 147
• 148
• 149
• 150
• 151
• 152
• 153
• 154
• 155
• 156
• 157
• 158
• 159
• 160
• 161
• 162
• 163
• 164
• 165
• 166
• 167
• 168
• 169
• 170

2.3 渲染

综上，拿到H264数据后，直接丢给解码器：

 decoder = VideoDncoder().apply {
   surfaceTexture = SurfaceTexture(textures[0]).apply {
        setDefaultBufferSize(width, height)
        setOnFrameAvailableListener {
        }
    }
    start(Surface(surfaceTexture), object : VideoDncoder.IDecoderListener {
        override fun onReady() {
            readFile()
        }
<span class="token punctuation">}</span><span class="token punctuation">)</span>

}

<span class="token comment">/**
 * 读取文件
 */</span>

private fun readFile() {
val stream = context.resources.assets.open(“video.h264”)
h264ParseThread = H264ParseThread(stream, object : H264ParseThread.IFrameListener {
override fun onLog(msg: String) {
Log.d(TAG, “H264ParseThread msg: $msg”)
}

    override fun <span class="token function">onFrame</span><span class="token punctuation">(</span>byteArray<span class="token operator">:</span> <span class="token class-name">ByteArray</span><span class="token punctuation">,</span> offset<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">,</span> count<span class="token operator">:</span> <span class="token class-name">Int</span><span class="token punctuation">)</span> <span class="token punctuation">{<!-- --></span>
        decoder<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">feedData</span><span class="token punctuation">(</span>byteArray<span class="token punctuation">,</span> offset<span class="token punctuation">,</span> count<span class="token punctuation">)</span>
        <span class="token class-name">Thread</span><span class="token punctuation">.</span><span class="token function">sleep</span><span class="token punctuation">(</span><span class="token number">55</span><span class="token punctuation">)</span>

    <span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
h264ParseThread<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">start</span><span class="token punctuation">(</span><span class="token punctuation">)</span>

<span class="token punctuation">}</span>

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33

onDrawFrame 那里，更新缓冲区，updateTexImage：

    override fun onDrawFrame(gl: GL10?) {
        //步骤1：使用glClearColor设置的颜色，刷新Surface
        GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT)
        surfaceTexture?.updateTexImage()
        GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[0])
        GLES30.glBindVertexArray(vao[0])
        GLES30.glDrawElements(GLES30.GL_TRIANGLE_STRIP, 6, GLES30.GL_UNSIGNED_INT, 0)
    }

 
 

 
 
1
2
3
4
5
6
7
8

记得释放：

    private fun release() {
        h264ParseThread?.release()
        h264ParseThread = null
        decoder?.release()
        decoder = null
        surfaceTexture?.release()
        surfaceTexture = null
    }

 
 

 
 
1
2
3
4
5
6
7
8

这样，我们就完成 Mediacodec 和 OpenGL 的结合，实现了H264的解析。

四. 滤镜

从Android OpenGL ES 学习(十一) –渲染YUV视频以及视频抖音特效 知道，滤镜是对 rgb 的基础上，添加一些效果。
所以，在片段着色器拿到 rgb 的数据后，我们也能实现一些滤镜效果，如灰色：

#version 300 es
precision mediump float;
 out vec4 FragColor;
 in vec2 vTexture;
 uniform samplerExternalOES ourTexture;
 void main() {
    vec4 temColor = texture(ourTexture,vTexture);
	float gray = temColor.r * 0.2126 + temColor.g * 0.7152 + temColor.b * 0.0722;
	FragColor = vec4(gray,gray,gray,1.0);
 }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11

其他效果，参考工程。

至此，OpenGL 教程，就暂时告一段了。

参考：
https://source.android.google.cn/docs/core/graphics/arch-st