显示效果
背景
在应用的开发过程中，我们避免不了要使用Opengl共享context的技术。比如我们需要在预览的同时还要将预览的画面传递给MediaCodec进行编码。有的朋友会说Camera2可以设置多个surface，这样就可以把预览的surface和MediaCodec的surface同时传递给Camera2。这种只是实现比较简单的录制功能。现在多数有录制功能的应用都支持滤镜功能，可以把你拍得更漂亮。由于加入了滤镜的功能，所以我们要通过Opengl技术来实现。
当我们加入opengl的滤镜功能后就有一个需要解决的问题，如何让预览和MediaCo dec同时应用滤镜。这时共享 context技术就派上用场了，通过共享context使创建在不同线程的texture id可以相互共享。通常共享的texture id绑定的是一个frame buffer，滤镜效果都绘制在这个frame buffer上，然后将frame buffer分别绘制在预览和MediaCodec。这里不对共享context的实现进行详细的介绍，这里重点关注的是有没有其他的方案来实现这个功能。
关于Camera2的思考（SurfaceTextureRenderer）
大家都知道Camera2是支持设置多个surface的，那么他是如何把camera数据绘制到这些surface上的呢？实现方式就在SurfaceTextureRenderer中，通过查看这个文件我们了解到他构造了一个opengl环境用于绘制camera数据。在构建opengl环境的过程中创建了EGLDisplay、EGLContext、EGLSurface。这里关于EGLSurface的创建有些特殊，我们可以看到代码中创建了多个EGLSurface。每个EGLSurface都与一个surface进行绑定。
private void configureEGLOutputSurfaces(Collection<EGLSurfaceHolder> surfaces) {
if (surfaces == null || surfaces.size() == 0) {
throw new IllegalStateException("No Surfaces were provided to draw to");
}
int[] surfaceAttribs = {
EGL14.EGL_NONE
};
for (EGLSurfaceHolder holder : surfaces) {
holder.eglSurface = EGL14.eglCreateWindowSurface(mEGLDisplay, mConfigs,
holder.surface, surfaceAttribs, /*offset*/ 0);
checkEglError("eglCreateWindowSurface");
}
}
下面的代码是绘制camera数据到各个EGLSurface上，具体绘制到哪个EGLSurface上是通过 makeCurrent方法控制的。看到这里大家大概就明白了，通过EGL14.eglMakeCurrent方法就可以实现绘制内容到不同的surface上。当然在切换surface的时候要留意绘制的viewport大小也会发送变化，所以mvp matrix也要进行相应的更新。
public void drawIntoSurfaces(CaptureCollector targetCollector) {
...
for (EGLSurfaceHolder holder : mSurfaces) {
if (LegacyCameraDevice.containsSurfaceId(holder.surface, targetSurfaceIds)) {
try{
LegacyCameraDevice.setSurfaceDimens(holder.surface, holder.width,
holder.height);
makeCurrent(holder.eglSurface);
LegacyCameraDevice.setNextTimestamp(holder.surface, captureHolder.second);
drawFrame(mSurfaceTexture, holder.width, holder.height,
(mFacing == CameraCharacteristics.LENS_FACING_FRONT) ?
FLIP_TYPE_HORIZONTAL : FLIP_TYPE_NONE);
swapBuffers(holder.eglSurface);
} catch (LegacyExceptionUtils.BufferQueueAbandonedException e) {
Log.w(TAG, "Surface abandoned, dropping frame. ", e);
request.setOutputAbandoned();
}
}
}
...
}
｝
private void makeCurrent(EGLSurface surface)
throws LegacyExceptionUtils.BufferQueueAbandonedException {
EGL14.eglMakeCurrent(mEGLDisplay, surface, surface, mEGLContext);
checkEglDrawError("makeCurrent");
}
通过添加两个SurfaceView验证方案
这里我们创建两个surfaceview用于验证构造多个eglsurface的方案。
surfaceView1.holder.addCallback(object:SurfaceHolder.Callback{
override fun surfaceCreated(holder: SurfaceHolder?) {
}
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, width: Int, height: Int) {
if (eglSurfaceHolder1 == null) {
eglSurfaceHolder1 = EGLCore.EGLSurfaceHolder(holder!!.surface, width.toFloat(), height.toFloat())
renderScope.addSurfaceHolder(eglSurfaceHolder1)
}
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
renderScope.removeSurfaceHolder(eglSurfaceHolder1)
eglSurfaceHolder1 = null
}
})
surfaceView2.holder.addCallback(object:SurfaceHolder.Callback{
override fun surfaceCreated(holder: SurfaceHolder?) {
}
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, width: Int, height: Int) {
if (eglSurfaceHolder2 == null) {
eglSurfaceHolder2 = EGLCore.EGLSurfaceHolder(holder!!.surface, width.toFloat(), height.toFloat())
renderScope.addSurfaceHolder(eglSurfaceHolder2)
}
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
renderScope.removeSurfaceHolder(eglSurfaceHolder2)
eglSurfaceHolder2 = null
}
})
这里把两个surface都添加到RenderScope中，RenderScope创建opengl线程环境，opengl描画都是发生在这个线程的。
class RenderScope(private val render: Render) : BackgroundScope by HandlerThreadScope() {
private var eglCore: EGLCore? = null
init {
runInBackground {
eglCore = EGLCore()
render.onCreate()
}
}
fun addSurfaceHolder(eglSurfaceHolder: EGLCore.EGLSurfaceHolder?) = runInBackground {
eglCore?.addSurfaceHolder(eglSurfaceHolder)
}
fun removeSurfaceHolder(eglSurfaceHolder: EGLCore.EGLSurfaceHolder?) = runInBackground {
eglCore?.removeSurfaceHolder(eglSurfaceHolder)
}
fun removeSurfaceHolder(surface: Surface?) = runInBackground {
eglCore?.removeSurfaceHolder(surface)
}
fun release() = runInBackground {
render.onDestroy()
eglCore?.releaseEGLContext()
eglCore = null
quit()
}
fun requestRender() = runInBackground {
eglCore?.render { render.onDrawFrame(it) }
}
interface Render {
fun onCreate()
fun onDestroy()
fun onDrawFrame(eglSurfaceHolder: EGLCore.EGLSurfaceHolder)
}
}
EGLCore这个类是用来构造opengl环境的，构造的过程这里不细说了。我们重点关注下addSurfaceHolder这个方法。它是用于添加surface的，通过这个方法添加surface后，渲染过程中就可以把内容绘制到这个surface上。render方法用于渲染使用，看他的实现主要就是通过makecurrent来切换eglsurface，然后调用block 进行描画。block调用的次数与添加的surface数量相同。
class EGLCore {
private var mEGLDisplay = EGL14.EGL_NO_DISPLAY
private var mEGLContext = EGL14.EGL_NO_CONTEXT
private var mConfigs: EGLConfig? = null
private val mEGLSurfaces = SparseArray<EGLSurfaceHolder>()
init {
log("initEGLContext")
releaseEGLContext()
mEGLDisplay = EGL14.eglGetDisplay(EGL14.EGL_DEFAULT_DISPLAY)
check(!(mEGLDisplay === EGL14.EGL_NO_DISPLAY)) { "No EGL14 display" }
val version = IntArray(2)
check(EGL14.eglInitialize(mEGLDisplay, version, /*offset*/0, version, /*offset*/1)) { "Cannot initialize EGL14" }
val attributeList = intArrayOf(
EGL14.EGL_RED_SIZE, EGL_COLOR_BIT_LENGTH,
EGL14.EGL_GREEN_SIZE, EGL_COLOR_BIT_LENGTH,
EGL14.EGL_BLUE_SIZE, EGL_COLOR_BIT_LENGTH,
EGL14.EGL_RENDERABLE_TYPE, EGL14.EGL_OPENGL_ES2_BIT,
EGL_RECORDABLE_ANDROID, 1,
EGL14.EGL_SURFACE_TYPE, EGL14.EGL_PBUFFER_BIT or EGL14.EGL_WINDOW_BIT,
EGL14.EGL_NONE
)
val configs = arrayOfNulls<EGLConfig>(1)
val numConfigs = IntArray(1)
EGL14.eglChooseConfig(
mEGLDisplay,
attributeList, /*offset*/0,
configs, /*offset*/0, configs.size,
numConfigs, /*offset*/0
)
checkEglError("eglCreateContext RGB888+recordable ES2")
mConfigs = configs[0]
val contextAttributeList = intArrayOf(
EGL14.EGL_CONTEXT_CLIENT_VERSION, GLES_VERSION,
EGL14.EGL_NONE
)
mEGLContext = EGL14.eglCreateContext(
mEGLDisplay,
configs[0],
EGL14.EGL_NO_CONTEXT,
contextAttributeList, /*offset*/0
)
checkEglError("eglCreateContext")
check(!(mEGLContext === EGL14.EGL_NO_CONTEXT)) { "No EGLContext could be made" }
EGL14.eglMakeCurrent(mEGLDisplay, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_SURFACE, mEGLContext)
}
fun addSurfaceHolder(eglSurfaceHolder: EGLSurfaceHolder?) {
if (mEGLDisplay == EGL14.EGL_NO_DISPLAY) {
log("addSurfaceHolder mEGLDisplay == EGL14.EGL_NO_DISPLAY")
return
}
eglSurfaceHolder ?: return
check(eglSurfaceHolder.surface.isValid) { "addSurface surface is not valid!!" }
val surfaceAttribute = intArrayOf(
EGL14.EGL_NONE
)
eglSurfaceHolder.eglSurface = EGL14.eglCreateWindowSurface(
mEGLDisplay,
mConfigs,
eglSurfaceHolder.surface,
surfaceAttribute, /*offset*/0
)
checkEglError("eglCreateWindowSurface")
mEGLSurfaces.put(eglSurfaceHolder.surface.hashCode(), eglSurfaceHolder)
}
fun removeSurfaceHolder(eglSurfaceHolder: EGLSurfaceHolder?) {
if (mEGLDisplay == EGL14.EGL_NO_DISPLAY) {
log("removeSurfaceHolder mEGLDisplay == EGL14.EGL_NO_DISPLAY")
return
}
eglSurfaceHolder ?: return
mEGLSurfaces[eglSurfaceHolder.surface.hashCode()]?.let {
EGL14.eglDestroySurface(mEGLDisplay, it.eglSurface)
it.eglSurface = null
mEGLSurfaces.remove(it.surface.hashCode())
}
}
fun removeSurfaceHolder(surface: Surface?) {
if (mEGLDisplay == EGL14.EGL_NO_DISPLAY) {
log("removeSurfaceHolder mEGLDisplay == EGL14.EGL_NO_DISPLAY")
return
}
surface ?: return
mEGLSurfaces[surface.hashCode()]?.let {
EGL14.eglDestroySurface(mEGLDisplay, it.eglSurface)
it.eglSurface = null
mEGLSurfaces.remove(surface.hashCode())
}
}
fun render(block: (EGLSurfaceHolder) -> Unit) {
if (mEGLDisplay == EGL14.EGL_NO_DISPLAY) {
log("render mEGLDisplay == EGL14.EGL_NO_DISPLAY")
return
}
mEGLSurfaces.forEach { _, holder ->
if (!holder.available) {
return@forEach
}
EGL14.eglMakeCurrent(mEGLDisplay, holder.eglSurface, holder.eglSurface, mEGLContext)
checkEglError("makeCurrent")
block.invoke(holder)
val result = EGL14.eglSwapBuffers(mEGLDisplay, holder.eglSurface)
when (val error = EGL14.eglGetError()) {
EGL14.EGL_SUCCESS -> result
EGL14.EGL_BAD_NATIVE_WINDOW, EGL14.EGL_BAD_SURFACE -> throw IllegalStateException(
"swapBuffers: EGL error: 0x" + Integer.toHexString(error)
)
else -> throw IllegalStateException(
"swapBuffers: EGL error: 0x" + Integer.toHexString(error)
)
}
}
}
fun releaseEGLContext() {
if (mEGLDisplay !== EGL14.EGL_NO_DISPLAY) {
log("releaseEGLContext")
EGL14.eglMakeCurrent(mEGLDisplay, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_CONTEXT)
mEGLSurfaces.forEach { _, holder ->
if (holder.eglSurface != null) {
EGL14.eglDestroySurface(mEGLDisplay, holder.eglSurface)
holder.eglSurface = null
}
}
mEGLSurfaces.clear()
EGL14.eglDestroyContext(mEGLDisplay, mEGLContext)
EGL14.eglReleaseThread()
EGL14.eglTerminate(mEGLDisplay)
}
mConfigs = null
mEGLDisplay = EGL14.EGL_NO_DISPLAY
mEGLContext = EGL14.EGL_NO_CONTEXT
}
private fun checkEglError(msg: String) {
val error = EGL14.eglGetError()
check(error == EGL14.EGL_SUCCESS) { msg + ": EGL error: 0x" + Integer.toHexString(error) }
}
private fun log(msg: String) {
Log.d("EGLCore", msg)
}
class EGLSurfaceHolder(
val surface: Surface,
val width: Float,
val height: Float
) {
var eglSurface: EGLSurface? = null
var available = true
val mvpMatrix = MVPMatrix().updateViewport(width, height)
val viewPortRectF = RectF(0f, 0f, width, height)
}
companion object {
private const val GLES_VERSION = 2
private const val EGL_COLOR_BIT_LENGTH = 8
const val EGL_RECORDABLE_ANDROID = 0x3142 // from EGL/eglext.h
}
}
Git
https://github.com/mjlong123123/MultiSurface
我的公众号已经开通，公众号会同步发布。
欢迎关注我的公众号
4199
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
