相信大家都对直播不会陌生,直播的技术也越来越成熟了,目前有这样的一个技术,当弹幕飘到主播的脸上的时候,弹幕会自动消失,出了人脸范围内,就继续显示出来。这个原理非常的简单,其实就是人脸识别,将人脸识别范围内的弹幕全都隐藏。说起来容易做起来难,本文将分以下几点讲述如何实现RTMP视频流的人脸识别。
- 方案选择
- ffmpeg的接入
- ffmpeg的数据解析
- OpenGL的数据绘制
- 人脸跟踪以及人脸框的绘制
一、方案的选择
笔者一开始想直接使用别人封装好的播放器,输入地址就能播放。接入后发现,确实接入和使用都很简单,也能够显示,但是有一个很致命的问题,就是没有提供获取裸数据的接口,因而没办法进行人脸识别,后面我就转用了ffmpeg。当然如果只是想在设备上播放RTMP流,bilibili的ijkplayer的框架是完全没有问题的,接入和使用都很简单下面是他们的地址。
- https://github.com/Bilibili/ijkplayer
解析方案已经选择完毕,接下来就是绘制和人脸识别,绘制我采用OpenGL。原因是之前有自己封装过一个自定义surfaceView,直接拿来用就可以了。人脸识别引擎我选择虹软的引擎,原因有二,一是使用起来比较简单,虹软的demo写的不错,很多东西可以直接抄过来;二是免费,其他公司的引擎我也用过,都是有试用期限,我不喜欢有期限的东西,而且虹软的引擎效果也不错,当然是我的首选。
二、ffmpeg的接入
1.目录结构
在src/main目录下新建cpp以及jniLibs目录,并将ffmpeg库放入,如下图所示。
2.CMakeLists
首先我们在src/main/cpp目录下新建两个文件,CMakeLists.txt,rtmpplayer-lib。CMake用于库间文件的管理与构建,rtmpplayer-lib是放我们解析数据流的jni代码用的。
CMakeLists.txt
# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html
# Sets the minimum version of CMake required to build the native library.
cmake_minimum_required(VERSION 3.4.1)
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.
add_library( # Sets the name of the library.
rtmpplayer-lib
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
rtmpplayer-lib.cpp)
include_directories(ffmpeg)
# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.
find_library( # Sets the name of the path variable.
log-lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log)
# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.
target_link_libraries( # Specifies the target librar