[-综合篇-] 相机、OpenGL、视频、Flutter和SurfaceView

2401_85388621

于 2024-07-04 22:19:00 发布

阅读量456

点赞数 3

文章标签：数码相机音视频 flutter

本文链接：https://blog.csdn.net/2401_85388621/article/details/140191371

版权

本篇会从一下几点的极简操作，来让你对SurfaceView有个感性的认知:

[1].Camera的预览和SurfaceView的使用
[2].Camera2的预览和SurfaceView的使用
[3].OpenGL中的GLSurfaceView
[4].Camera2和OpenGL的结合
[5].视频播放和和OpenGL的结合
[6].Flutter与SurfaceView的联系

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

1.Camera使用SurfaceView开启预览

SurfaceView依赖SurfaceHolder类，所以两者形影不离。Camera的setPreviewDisplay方法入参是一个SurfaceHolder

SurfaceHolder并不是立马就创建出来的，需要一个回调监听。以便对它创建、改变、销毁时的感知并进行相关操作。
该监听的接口为SurfaceHolder.Callback,为了方便，可直接实现之。当然你也可以新建一个类

详细操作见:Android多媒体之Camera的相关操作

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

public class CameraSurfaceView extends SurfaceView implements SurfaceHolder.Callback {
private Camera camera;

public CameraSurfaceView(Context context) {
this(context,null);
}

public CameraSurfaceView(Context context, AttributeSet attrs) {
this(context, attrs,0);
}

public CameraSurfaceView(Context context, AttributeSet attrs, int defStyleAttr) {
super(context, attrs, defStyleAttr);
getHolder().addCallback(this);//为SurfaceView的SurfaceHolder添加回调
}

//-----------------覆写SurfaceHolder.Callback方法----------------------
@Override
public void surfaceCreated(SurfaceHolder holder) {
camera = Camera.open();
camera.setDisplayOrientation(90);
try {
camera.setPreviewDisplay(holder);//Camera+SurfaceHolder
camera.startPreview();
} catch (IOException e) {
e.printStackTrace();
}
}

@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

}

@Override
public void surfaceDestroyed(SurfaceHolder holder) {
camera.release();//释放资源
}

}

2.Camera2中SurfaceView使用

Camera2并不是值Camera2类，而是camera2包下的相机系统,虽然使用起来挺复杂

但简单必有简单的局限，复杂必有复杂的价值,它的显示核心也需要一个SurfaceHolder

详细操作见:Android多媒体之Camera2的相关操作

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

public class Camera2SurfaceView extends SurfaceView implements SurfaceHolder.Callback {
private Handler mainHandler;
private String mCameraID;
private CameraManager mCameraManager;
private CameraDevice mCameraDevice;//相机设备
private CameraCaptureSession mCameraCaptureSession;
private Handler childHandler;

private CameraDevice.StateCallback mStateCallback;

private Semaphore mCameraOpenCloseLock = new Semaphore(1);//以防止在关闭相机之前应用程序退出

public Camera2SurfaceView(Context context) {
this(context,null);
}

public Camera2SurfaceView(Context context, AttributeSet attrs) {
this(context, attrs,0);
}

public Camera2SurfaceView(Context context, AttributeSet attrs, int defStyleAttr) {
super(context, attrs, defStyleAttr);
getHolder().addCallback(this);//为SurfaceView的SurfaceHolder添加回调
}

//-----------------覆写SurfaceHolder.Callback方法----------------------
@Override
public void surfaceCreated(SurfaceHolder holder) {
initHandler();//初始化线程处理器
initCamera();//初始化相机
try {
if (ActivityCompat.checkSelfPermission(getContext(), Manifest.permission.CAMERA) !=
PackageManager.PERMISSION_GRANTED) {
return;
}
mCameraManager.openCamera(mCameraID, mStateCallback, mainHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}

@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

}

@Override
public void surfaceDestroyed(SurfaceHolder holder) {
mCameraDevice.close();//释放资源;//释放资源
}

private void initCamera() {
mCameraID = “” + CameraCharacteristics.LENS_FACING_FRONT;//后摄像头
//获取摄像头管理器
mCameraManager = (CameraManager) getContext().getSystemService(Context.CAMERA_SERVICE);

mStateCallback = new CameraDevice.StateCallback() {
@Override
public void onOpened(@NonNull CameraDevice camera) {
mCameraOpenCloseLock.release();
mCameraDevice = camera;
startPreview();
}
@Override
public void onDisconnected(@NonNull CameraDevice camera) {
mCameraOpenCloseLock.release();
mCameraDevice.close();
}
@Override
public void onError(@NonNull CameraDevice camera, int error) {
mCameraOpenCloseLock.release();
mCameraDevice.close();
}
};
}

private void initHandler() {
HandlerThread handlerThread = new HandlerThread(“Camera2”);
handlerThread.start();
mainHandler = new Handler(getMainLooper());//主线程Handler
childHandler = new Handler(handlerThread.getLooper());//子线程Handler
}

/**

开启预览
*/
private void startPreview() {
try {
// 创建预览需要的CaptureRequest.Builder
final CaptureRequest.Builder reqBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
// 将SurfaceView的surface作为CaptureRequest.Builder的目标
reqBuilder.addTarget(getHolder().getSurface());

// 创建CameraCaptureSession，该对象负责管理处理预览请求和拍照请求
CameraCaptureSession.StateCallback stateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) {
if (null == mCameraDevice) return;
mCameraCaptureSession = cameraCaptureSession; // 当摄像头已经准备好时，开始显示预览
try {// 显示预览
mCameraCaptureSession.setRepeatingRequest(reqBuilder.build(), null, childHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
@Override
public void onConfigureFailed(@NonNull CameraCaptureSession cameraCaptureSession) {

}
};
mCameraDevice.createCaptureSession(Collections.singletonList(getHolder().getSurface()),
stateCallback, childHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
}

3.OpenGL中GLSurfaceView使用

GLSurfaceView作为SurfaceView的子类，打开了一扇叫作OpenGL的大门。

它实现了SurfaceHolder.Callback2接口,需要传入一个GLSurfaceView.Render接口

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

public class TriangleGLView extends GLSurfaceView implements GLSurfaceView.Renderer {
private Triangle mTriangle;

public TriangleGLView(Context context) {
this(context, null);
}

public TriangleGLView(Context context, AttributeSet attrs) {
super(context, attrs);
setEGLContextClientVersion(2);//设置OpenGL ES 2.0 context
setRenderer(this);//设置渲染器
}

@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
mTriangle=new Triangle();
GLES20.glClearColor(1.0f, 0.0f, 0.0f, 1.0f);//rgba
}

@Override
public void onSurfaceChanged(GL10 gl, int width, int height) {
GLES20.glViewport(0, 0, width, height);//GL视口
}

@Override
public void onDrawFrame(GL10 gl) {
//清除颜色缓存和深度缓存
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT| GLES20.GL_DEPTH_BUFFER_BIT);
mTriangle.draw();
}
}

OpenGL的绘制可谓让人望而却步,下面是最简单的三角形绘制,

public class Triangle {
private FloatBuffer vertexBuffer;//顶点缓冲
private final String vertexShaderCode =//顶点着色代码
“attribute vec4 vPosition;” +
“void main() {” +
" gl_Position = vPosition;" +
“}”;
private final String fragmentShaderCode =//片元着色代码
“precision mediump float;” +
“uniform vec4 vColor;” +
“void main() {” +
" gl_FragColor = vColor;" +
“}”;
private final int mProgram;
private int mPositionHandle;//位置句柄
private int mColorHandle;//颜色句柄
private final int vertexCount = sCoo.length / COORDS_PER_VERTEX;//顶点个数
private final int vertexStride = COORDS_PER_VERTEX * 4; // 34=12
// 数组中每个顶点的坐标数
static final int COORDS_PER_VERTEX = 3;
static float sCoo[] = { //以逆时针顺序
0.0f, 0.0f, 0.0f, // 顶部
-1.0f, -1.0f, 0.0f, // 左下
1.0f, -1.0f, 0.0f // 右下
};
// 颜色，rgba
float color[] = {0.63671875f, 0.76953125f, 0.22265625f, 1.0f};
public Triangle() {
//初始化顶点字节缓冲区
ByteBuffer bb = ByteBuffer.allocateDirect(sCoo.length * 4);//每个浮点数:坐标个数 4字节
bb.order(ByteOrder.nativeOrder());//使用本机硬件设备的字节顺序
vertexBuffer = bb.asFloatBuffer();// 从字节缓冲区创建浮点缓冲区
vertexBuffer.put(sCoo);// 将坐标添加到FloatBuffer
vertexBuffer.position(0);//设置缓冲区以读取第一个坐标
int vertexShader = loadShader(
GLES20.GL_VERTEX_SHADER,//顶点着色
vertexShaderCode);
int fragmentShader = loadShader
(GLES20.GL_FRAGMENT_SHADER,//片元着色
fragmentShaderCode);
mProgram = GLES20.glCreateProgram();//创建空的OpenGL ES 程序
GLES20.glAttachShader(mProgram, vertexShader);//加入顶点着色器
GLES20.glAttachShader(mProgram, fragmentShader);//加入片元着色器
GLES20.glLinkProgram(mProgram);//创建可执行的OpenGL ES项目
}

private int loadShader(int type, String shaderCode) {
int shader = GLES20.glCreateShader(type);//创建着色器
GLES20.glShaderSource(shader, shaderCode);//添加着色器源代码
GLES20.glCompileShader(shader);//编译
return shader;
}

public void draw() {
// 将程序添加到OpenGL ES环境中
GLES20.glUseProgram(mProgram);
//获取顶点着色器的vPosition成员的句柄
mPositionHandle = GLES20.glGetAttribLocation(mProgram, “vPosition”);
//启用三角形顶点的句柄
GLES20.glEnableVertexAttribArray(mPositionHandle);
//准备三角坐标数据
GLES20.glVertexAttribPointer(
mPositionHandle, COORDS_PER_VERTEX,
GLES20.GL_FLOAT, false,
vertexStride, vertexBuffer);
// 获取片元着色器的vColor成员的句柄
mColorHandle = GLES20.glGetUniformLocation(mProgram, “vColor”);
//为三角形设置颜色
GLES20.glUniform4fv(mColorHandle, 1, color, 0);
//绘制三角形
GLES20.glDrawArrays(GLES20.GL_TRIANGLES, 0, vertexCount);
//禁用顶点数组
GLES20.glDisableVertexAttribArray(mPositionHandle);
}
}

4.OpenGL在相机中的使用

现在捋一下,相机需要一个SurfaceHolder,而GLSurfaceView是一个SurfaceView,郎情妾意。但好事多磨,并没有想象中的这么简单…

在CameraGLView主类中创建SurfaceTexture对象,并将纹理绑定其上
而通过SurfaceTexture作为入参可以创建SurfaceHolder,一条路就通了。

public class CameraGLView extends GLSurfaceView implements GLSurfaceView.Renderer {

private CameraDrawer cameraDrawer;
public SurfaceTexture surfaceTexture;
private int[] textureId = new int[1];

//----------------------------相机操作------------------------------
private Handler mainHandler;
private Handler childHandler;
private String mCameraID;
private CameraManager mCameraManager;
private CameraDevice mCameraDevice;//相机设备
private CameraCaptureSession mCameraCaptureSession;

private CameraDevice.StateCallback mStateCallback;
private Size mVideoSize;
private Semaphore mCameraOpenCloseLock = new Semaphore(1);//以防止在关闭相机之前应用程序退出
private Surface surface;

public CameraGLView(Context context) {
this(context,null);
}

public CameraGLView(Context context, AttributeSet attrs) {
super(context, attrs);
setEGLContextClientVersion(3);//设置OpenGL ES 3.0 context
setRenderer(this);//设置渲染器
}

private void initCamera() {

mCameraID = “” + CameraCharacteristics.LENS_FACING_FRONT;//后摄像头
//获取摄像头管理器
mCameraManager = (CameraManager) getContext().getSystemService(Context.CAMERA_SERVICE);
mVideoSize=getCameraOutputSizes(mCameraManager,mCameraID,SurfaceTexture.class).get(0);

/**

根据输出类获取指定相机的输出尺寸列表，降序排序
/
public List getCameraOutputSizes(CameraManager cameraManager, String cameraId, Class clz){
try {
CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId);
StreamConfigurationMap configs = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
List sizes = Arrays.asList(configs.getOutputSizes(clz));
Collections.sort(sizes, (o1, o2) -> o1.getWidth() * o1.getHeight() - o2.getWidth() * o2.getHeight());
Collections.reverse(sizes);
return sizes;
} catch (CameraAccessException e) {
e.printStackTrace();
}
return null;
}
/*
开启预览
*/
private void startPreview() {
surfaceTexture.setDefaultBufferSize(mVideoSize.getWidth(), mVideoSize.getHeight());
surfaceTexture.setOnFrameAvailableListener(surfaceTexture -> requestRender());
surface = new Surface(surfaceTexture);

try {
// 创建预览需要的CaptureRequest.Builder
final CaptureRequest.Builder reqBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
// 将SurfaceView的surface作为CaptureRequest.Builder的目标
reqBuilder.addTarget(surface);
// 创建CameraCaptureSession，该对象负责管理处理预览请求和拍照请求
CameraCaptureSession.StateCallback stateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) {
if (null == mCameraDevice) return;
mCameraCaptureSession = cameraCaptureSession; // 当摄像头已经准备好时，开始显示预览
try {// 显示预览
mCameraCaptureSession.setRepeatingRequest(reqBuilder.build(), null, childHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
@Override
public void onConfigureFailed(@NonNull CameraCaptureSession cameraCaptureSession) {

}
};
mCameraDevice.createCaptureSession(Collections.singletonList(surface), stateCallback, childHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}

@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
cameraDrawer=new CameraDrawer(getContext());
//创建纹理对象
GLES30.glGenTextures(textureId.length, textureId, 0);
//将纹理对象绑定到srufaceTexture
surfaceTexture = new SurfaceTexture(textureId[0]); //创建并连接程序

initHandler();//初始化线程处理器
initCamera();//初始化相机
try {
if (ActivityCompat.checkSelfPermission(getContext(), Manifest.permission.CAMERA) !=
PackageManager.PERMISSION_GRANTED) {
return;
}
mCameraManager.openCamera(mCameraID, mStateCallback, mainHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}

@Override
public void onSurfaceChanged(GL10 gl, int width, int height) {
glViewport(0,0,width,height);
}

@Override
public void onDrawFrame(GL10 gl) {
surfaceTexture.updateTexImage();
GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT);
cameraDrawer.draw(textureId[0]);
}
}

通过CameraDrawer类绘制纹理，这就跟画三角形非常类似,通过着色器(shader)进行着色

fragment片元:camera.fsh

#version 300 es
#extension GL_OES_EGL_image_external_essl3 : require
precision mediump float;

in vec2 vTexCoord;
out vec4 outColor;
uniform samplerExternalOES sTexture;

void main(){
outColor = texture(sTexture, vTexCoord);
}

vertex顶元:camera.vsh

#version 300 es
in vec4 aPosition;
in vec2 aTexCoord;

out vec2 vTexCoord;

void main(){
gl_Position = aPosition;
vTexCoord = aTexCoord;
}

绘画器:CameraDrawer

public class CameraDrawer {
private static final String VERTEX_ATTRIB_POSITION = “a_Position”;
private static final int VERTEX_ATTRIB_POSITION_SIZE = 3;
private static final String VERTEX_ATTRIB_TEXTURE_POSITION = “a_texCoord”;
private static final int VERTEX_ATTRIB_TEXTURE_POSITION_SIZE = 2;
private static final String UNIFORM_TEXTURE = “s_texture”;

private float[] vertex ={
-1f,1f,0.0f,//左上
-1f,-1f,0.0f,//左下
1f,-1f,0.0f,//右下
1f,1f,0.0f//右上
};

//纹理坐标，（s,t），t坐标方向和顶点y坐标反着
public float[] textureCoord = {
0.0f,1.0f,
1.0f,1.0f,
1.0f,0.0f,
0.0f,0.0f
};

private FloatBuffer vertexBuffer;
private FloatBuffer textureCoordBuffer;
private int program;

private Context context;

public CameraDrawer(Context context) {
this.context = context;

initVertexAttrib(); //初始化顶点数据

program = GLUtil.loadAndInitProgram( this.context,“camera.vsh”,“camera.fsh”);
GLES30.glClearColor(1.0f, 1.0f, 1.0f, 0.0f);
}

private void initVertexAttrib() {
textureCoordBuffer = GLUtil.getFloatBuffer(textureCoord);
vertexBuffer = GLUtil.getFloatBuffer(vertex);
}

public void draw(int textureId){
GLES30.glUseProgram(program);
//初始化句柄
int vertexLoc = GLES30.glGetAttribLocation(program, VERTEX_ATTRIB_POSITION);
int textureLoc = GLES30.glGetAttribLocation(program, VERTEX_ATTRIB_TEXTURE_POSITION);

GLES30.glEnableVertexAttribArray(vertexLoc);
GLES30.glEnableVertexAttribArray(textureLoc);

GLES30.glVertexAttribPointer(vertexLoc,
VERTEX_ATTRIB_POSITION_SIZE,
GLES30.GL_FLOAT,
false,
0,
vertexBuffer);

GLES30.glVertexAttribPointer(textureLoc,
VERTEX_ATTRIB_TEXTURE_POSITION_SIZE,
GLES30.GL_FLOAT,
false,
0,
textureCoordBuffer);

//纹理绑定
GLES30.glActiveTexture( GLES30.GL_TEXTURE0);
GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textureId);
GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MIN_FILTER, GLES30.GL_NEAREST);
GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_LINEAR);
GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_S, GLES30.GL_CLAMP_TO_EDGE);
GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_T, GLES30.GL_CLAMP_TO_EDGE);
int uTextureLoc = GLES30.glGetUniformLocation(program, UNIFORM_TEXTURE);
GLES30.glUniform1i(uTextureLoc,0);
//绘制
GLES30.glDrawArrays( GLES30.GL_TRIANGLE_FAN,0,vertex.length / 3);
//禁用顶点
GLES30.glDisableVertexAttribArray(vertexLoc);
GLES30.glDisableVertexAttribArray(textureLoc);
}
}

也许你并不了解OpenGL,看到结果会大呼:TM,这么麻烦，才实先预览?拜拜，学不动，告辞。

对，很麻烦，之后还会更麻烦。但你不会，别人会。你怕麻烦，别人去钻研，这就是人与人的差距。
我最不能理解的是怕麻烦的人到处询问学习方法。只要你不怕麻烦，遇到问题肯去钻，去看源码，去debug，还有什么能阻挡你。世事有难易乎,为之则难者易，不为则易者难。

OpenGL打开了一扇大门，根据shader可以进行非常多的操作，滤镜，贴图，着色，变换…甚至可以说给我一个shader的用武之处，我能给你创造一个世界

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

5.OpenGL在视频播放中的使用

如果你稍微了解一下视频播放，会知道MediaPlayer可以和Surface狼狈为奸

于是乎，同理，可以将视频播放和OpenGL结合,然后通过shader来逆天改命
这里思路几乎一致GLVideoView中进行SurfaceTexture和纹理绑定，并生成Surface给MediaPlayer
关于MediaPlayer的视频播放，详见:Android多媒体之视频播放器(基于MediaPlayer)

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

public class GLVideoView extends GLSurfaceView implements GLSurfaceView.Renderer,
SurfaceTexture.OnFrameAvailableListener, MediaPlayer.OnVideoSizeChangedListener {
private float[] sTMatrix = new float[16];
private final float[] projectionMatrix=new float[16];

尾声

最后，我再重复一次，如果你想成为一个优秀的 Android 开发人员，请集中精力，对基础和重要的事情做深度研究。

对于很多初中级Android工程师而言，想要提升技能，往往是自己摸索成长，不成体系的学习效果低效漫长且无助。整理的这些架构技术希望对Android开发的朋友们有所参考以及少走弯路，本文的重点是你有没有收获与成长，其余的都不重要，希望读者们能谨记这一点。

这里，笔者分享一份从架构哲学的层面来剖析的视频及资料分享给大家梳理了多年的架构经验，筹备近6个月最新录制的，相信这份视频能给你带来不一样的启发、收获。

Android进阶学习资料库

一共十个专题，包括了Android进阶所有学习资料，Android进阶视频，Flutter，java基础，kotlin，NDK模块，计算机网络，数据结构与算法，微信小程序，面试题解析，framework源码！

ableListener, MediaPlayer.OnVideoSizeChangedListener {
private float[] sTMatrix = new float[16];
private final float[] projectionMatrix=new float[16];