通过外部USB摄像头使用适用于Android的Mapbox Vision SDK

MapBox 查看原文档https://docs.mapbox.com/help/tutorials/android-vision-sdk-usb-camera/

SDK通过来使用设备的内部摄像机Camera 2 API，但是也可以通过Vision SDK使用外部摄像机或任何其他视频源（例如文件或互联网流）。使用外部摄像头可以提高移动应用程序的可用性，因为它不需要用户将手机安装到汽车挡风玻璃上。要使用外部源，您将需要实现一个custom VideoSource。本教程将展示如何VideoSource为连接的USB相机实现自定义。这是一个示例示例设置，该示例设置在带有Logitech C920 USB摄像头的Samsung S10 +上运行Vision SDK：

入门在开始本教程之前，请阅读Vision SDK for Android文档中的“ 安装并配置”步骤。这将引导您逐步了解如何安装Vision SDK和配置应用程序。Android上的USB摄像头某些Android设备可能会通过默认/ API 支持某些USB相机，但支持和设备覆盖范围有限。本教程使用该库从USB摄像机获取视频流。库本身使用修改后的版本，并使用USB摄像头。CameraCamera2UVCCameralibusblibuvc摄像头和设备定位应按照要求将USB摄像头安装在挡风玻璃下。设备定位要求取决于您使用的Vision功能：如果您使用不需要相机校准或传感器数据（例如分段和检测）的视觉功能，则对电话位置没有严格的要求。如果您将USB相机与需要相机校准和传感器数据（包括陀螺仪，加速计和重力）的其他功能（例如Vision AR或Vision Safety）一起使用，则需要安装手机，以便来自传感器的数据该设备是可靠的。这意味着电话方向应固定并尽可能与相机的方向对齐。

添加UVCCamera库将以下Maven存储库添加到repositories顶级部分

build.gradlerepositories {
  maven { url 'http://raw.github.com/saki4510t/libcommon/master/repository/' }}

下来，添加依赖项。您可以UVCCamera自己构建该库，也可以使用本教程中预构建的libuvccamera-release.aar。假设您已将库放入libs项目的目录中，则将依赖项添加到项目级别

build.gradle：build.gradledependencies {
  implementation fileTree(dir: 'libs', include: ['*.aar'])
  implementation("com.serenegiant:common:$uvccamera_common") {
    exclude module: 'support-v4'
  }
}

android.permission.CAMERA授予访问USB摄像头的权限，并将其添加到

AndroidManifest.xml：
AndroidManifest.xml<uses-permission android:name="android.permission.CAMERA" />

创建一个USB VideoSource要将Vision与USB相机连接，请实施自定义VideoSource并将其传递给VisionManager.create。
准备处理帧时，VisionManager它将调用attach此Video 的方法，以便您可以保存此侦听器并开始向其馈送帧。在attach回调中，创建一个USBMonitor对象并调用其register方法以触发USB相机启动过程。注意您应该切换到后台线程来调用所有与USB相机相关的方法，以避免主线程延迟。

//VideoSource implementation that connects to USB camera and feeds frames to VisionManager.
private VideoSource usbVideoSource = new VideoSource() {
// VisionManager will attach videoSourceListener after VisionManager.create is called.
// Here we open USB camera connection and continue connection via onDeviceConnectListener callbacks.
// NOTE : method is called from the same thread, that VisionManager.create is called.
@Overridepublic void attach(@NonNull VideoSourceListener videoSourceListener) {
if (!backgroundHandlerThread.isAlive())
 {backgroundHandlerThread.start();}
 backgroundHandler = new Handler(backgroundHandlerThread.getLooper());backgroundHandler.post(() ->{// Init and register USBMonitor.synchronized (UsbVideoSourceActivity.this) {usbVideoSourceListener = videoSourceListener;usbMonitor = new USBMonitor(UsbVideoSourceActivity.this, onDeviceConnectListener);usbMonitor.register();}});} // VisionManager will detach listener after VisionManager.destroy is called.// Here we close USB camera connection.// NOTE : method is called from the same thread, that VisionManager.destroy is called.@Overridepublic void detach() {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {if (usbMonitor != null) {usbMonitor.unregister();usbMonitor.destroy();}if (uvcCamera != null) {uvcCamera.stopPreview();releaseCamera();}usbVideoSourceListener = null;}}); backgroundHandlerThread.quitSafely();}};

复制USB相机的生命周期要USBMonitor在上一步中创建，您需要一个实例OnDeviceConnectListener。该对象将处理发送USB连接状态更新的回调。首次请求在onAttach回调中连接到摄像机的权限（此调用将触发描述请求的系统对话框）。确认对话框后，将进行onConnect回调，并在此启动连接本身。断开摄像头连接后，可以使用onDisconnect回调函数释放摄像头并重新分配使用的资源。private USBMonitor.OnDeviceConnectListener onDeviceConnectListener = new USBMonitor.OnDeviceConnectListener() {@Overridepublic void onAttach(UsbDevice device) {synchronized (UsbVideoSourceActivity.this) {usbMonitor.requestPermission(device);}} @Overridepublic void onConnect(UsbDevice device,USBMonitor.UsbControlBlock ctrlBlock,boolean createNew) {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {releaseCamera();initializeCamera(ctrlBlock);}});} @Overridepublic void onDetach(UsbDevice device) {} @Overridepublic void onCancel(UsbDevice device) {} @Overridepublic void onDisconnect(UsbDevice device, USBMonitor.UsbControlBlock ctrlBlock) {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {releaseCamera();}}); }};
复制创建并发布 UVCCamera创建摄像机实例并对其进行配置的函数包含以下步骤：创建一个UVCCamera对象，然后通过设置预览大小参数UVCCamera.setPreviewSize。创建外部OpenGL ES纹理并设置预览以使用包装在SurfaceTexture对象中的该纹理。由于不需要将来自摄像机的原始视频流显示在屏幕上，因此适合外部表面。取而代之的是，帧直接进入Vision SDK，然后在处理VisionView后显示分割结果。设置一个框架回调camera.setFrameCallback，它将检索单个框架并将其通过馈送到Vision SDK usbVideoSourceListener。相机初始化的代码是：```
private void initializeCamera(USBMonitor.UsbControlBlock ctrlBlock) {uvcCamera = new UVCCamera();uvcCamera.open(ctrlBlock);uvcCamera.setPreviewSize(CAMERA_FRAME_SIZE.getImageWidth(),CAMERA_FRAME_SIZE.getImageHeight(),UVCCamera.FRAME_FORMAT_YUYV); SurfaceTexture surfaceTexture = new SurfaceTexture(createExternalGlTexture());surfaceTexture.setDefaultBufferSize(CAMERA_FRAME_SIZE.getImageWidth(),CAMERA_FRAME_SIZE.getImageHeight());// Start preview to external GL texture// NOTE : this is necessary for callback passed to [UVCCamera.setFrameCallback]// to be triggered afterwardsuvcCamera.setPreviewTexture(surfaceTexture);uvcCamera.startPreview(); // Set callback that will feed frames from the USB camera to Vision SDKuvcCamera.setFrameCallback((frame) -> usbVideoSourceListener.onNewFrame(new VideoSourceListener.FrameHolder.ByteBufferHolder(frame),ImageFormat.RGBA,CAMERA_FRAME_SIZE),UVCCamera.PIXEL_FORMAT_RGBX);}复制最后，创建外部OpenGL ES纹理。注意您应该从带有GL上下文的线程中调用此函数。

private int createExternalGlTexture() {int[] textures = new int[1];GLES20.glGenTextures(1, textures, 0);int texId = textures[0];GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texId);GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER,GLES20.GL_LINEAR);GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER,GLES20.GL_LINEAR);GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S,GLES20.GL_CLAMP_TO_EDGE);GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T,GLES20.GL_CLAMP_TO_EDGE);return texId;}复制要释放UVCCamera和关联的资源，请调用其close和destroy方法：爪哇科特林private void releaseCamera() {if (uvcCamera != null) {uvcCamera.close();uvcCamera.destroy();uvcCamera = null;}}复制使用自定义实现的USB创建视觉 VideoSource现在您已经实现了一个新的VideoSource，可以使用它来初始化VisionManager：爪哇科特林@Overrideprotected void onStart() {super.onStart();startVisionManager();} private void startVisionManager() {if (allPermissionsGranted() && !visionManagerWasInit) {VisionManager.create(usbVideoSource);VisionManager.setModelPerformance(ModelPerformance.On(ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH))visionView.setVisionManager(VisionManager.INSTANCE);VisionManager.start(); visionManagerWasInit = true;}}复制完成的产品您已使用外部USB摄像头启动了Vision SDK。UsbActivity.java 在GitHub上查看爪哇科特林package com.mapbox.vision.examples; import android.graphics.SurfaceTexture;import android.hardware.usb.UsbDevice;import android.opengl.GLES11Ext;import android.opengl.GLES20;import android.os.Handler;import android.os.HandlerThread; import androidx.annotation.NonNull; import com.mapbox.vision.VisionManager;import com.mapbox.vision.mobile.core.models.frame.ImageFormat;import com.mapbox.vision.mobile.core.models.frame.ImageSize;import com.mapbox.vision.performance.ModelPerformance;import com.mapbox.vision.performance.ModelPerformanceMode;import com.mapbox.vision.performance.ModelPerformanceRate;import com.mapbox.vision.video.videosource.VideoSource;import com.mapbox.vision.video.videosource.VideoSourceListener;import com.mapbox.vision.view.VisionView;import com.serenegiant.usb.USBMonitor;import com.serenegiant.usb.UVCCamera; /*** Example shows how Vision SDK can work with external USB camera.* UVCCamera library is used to connect to the USB camera itself,* frames from camera are then fed to Vision SDK./public class UsbVideoSourceActivity extends BaseActivity { private static final ImageSize CAMERA_FRAME_SIZE = new ImageSize(1280, 720); private VisionView visionView; private HandlerThread backgroundHandlerThread = new HandlerThread(“VideoDecode”);private Handler backgroundHandler; private boolean visionManagerWasInit = false; /** Vision SDK will attach listener to get frames and camera parameters from the USB camera./private VideoSourceListener usbVideoSourceListener; /** VideoSource implementation that connects to USB camera and feeds frames to VisionManager./private VideoSource usbVideoSource = new VideoSource() {/** VisionManager will attach [videoSourceListener] after [VisionManager.create] is called.* Here we open USB camera connection, and proceed connection via [onDeviceConnectListener] callbacks.** NOTE : method is called from the same thread, that [VisionManager.create] is called./@Overridepublic void attach(@NonNull VideoSourceListener videoSourceListener) {if (!backgroundHandlerThread.isAlive()) {backgroundHandlerThread.start();}backgroundHandler = new Handler(backgroundHandlerThread.getLooper());backgroundHandler.post(() ->{// Init and register USBMonitor.synchronized (UsbVideoSourceActivity.this) {usbVideoSourceListener = videoSourceListener;usbMonitor = new USBMonitor(UsbVideoSourceActivity.this, onDeviceConnectListener);usbMonitor.register();}});} /** VisionManager will detach listener after [VisionManager.destroy] is called.* Here we close USB camera connection.** NOTE : method is called from the same thread, that [VisionManager.destroy] is called./@Overridepublic void detach() {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {if (usbMonitor != null) {usbMonitor.unregister();usbMonitor.destroy();}if (uvcCamera != null) {uvcCamera.stopPreview();releaseCamera();}usbVideoSourceListener = null;}}); backgroundHandlerThread.quitSafely();}}; private USBMonitor usbMonitor;private UVCCamera uvcCamera; @Overrideprotected void initViews() {setContentView(R.layout.activity_main);visionView = findViewById(R.id.vision_view);} @Overrideprotected void onPermissionsGranted() {startVisionManager();} @Overrideprotected void onStart() {super.onStart();startVisionManager();} @Overrideprotected void onStop() {super.onStop();stopVisionManager();} @Overrideprotected void onResume() {super.onResume();visionView.onResume();} @Overrideprotected void onPause() {super.onPause();visionView.onPause();} private void startVisionManager() {if (allPermissionsGranted() && !visionManagerWasInit) {VisionManager.create(usbVideoSource);VisionManager.setModelPerformance(new ModelPerformance.On(ModelPerformanceMode.FIXED, ModelPerformanceRate.HIGH.INSTANCE));visionView.setVisionManager(VisionManager.INSTANCE);VisionManager.start(); visionManagerWasInit = true;}} private void stopVisionManager() {if (visionManagerWasInit) {VisionManager.stop();VisionManager.destroy(); visionManagerWasInit = false;}} private USBMonitor.OnDeviceConnectListener onDeviceConnectListener = new USBMonitor.OnDeviceConnectListener() {@Overridepublic void onAttach(UsbDevice device) {synchronized (UsbVideoSourceActivity.this) {usbMonitor.requestPermission(device);}} @Overridepublic void onConnect(UsbDevice device,USBMonitor.UsbControlBlock ctrlBlock,boolean createNew) {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {releaseCamera();initializeCamera(ctrlBlock);}});} @Overridepublic void onDetach(UsbDevice device) {} @Overridepublic void onCancel(UsbDevice device) {} @Overridepublic void onDisconnect(UsbDevice device, USBMonitor.UsbControlBlock ctrlBlock) {backgroundHandler.post(() -> {synchronized (UsbVideoSourceActivity.this) {releaseCamera();}});}}; private void releaseCamera() {if (uvcCamera != null) {uvcCamera.close();uvcCamera.destroy();uvcCamera = null;}} private void initializeCamera(USBMonitor.UsbControlBlock ctrlBlock) {uvcCamera = new UVCCamera();uvcCamera.open(ctrlBlock);uvcCamera.setPreviewSize(CAMERA_FRAME_SIZE.getImageWidth(),CAMERA_FRAME_SIZE.getImageHeight(),UVCCamera.FRAME_FORMAT_YUYV); SurfaceTexture surfaceTexture = new SurfaceTexture(createExternalGlTexture());surfaceTexture.setDefaultBufferSize(CAMERA_FRAME_SIZE.getImageWidth(),CAMERA_FRAME_SIZE.getImageHeight());// Start preview to external GL texture// NOTE : this is necessary for callback passed to [UVCCamera.setFrameCallback]// to be triggered afterwardsuvcCamera.setPreviewTexture(surfaceTexture);uvcCamera.startPreview(); // Set callback that will feed frames from the USB camera to Vision SDKuvcCamera.setFrameCallback((frame) -> usbVideoSourceListener.onNewFrame(new VideoSourceListener.FrameHolder.ByteBufferHolder(frame),ImageFormat.RGBA,CAMERA_FRAME_SIZE),UVCCamera.PIXEL_FORMAT_RGBX);} /** Create external OpenGL texture for [uvcCamera].*/private int createExternalGlTexture() {int[] textures = new int[1];GLES20.glGenTextures(1, textures, 0);int texId = textures[0];GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texId);GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER,GLES20.GL_LINEAR);GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER,GLES20.GL_LINEAR);GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S,GLES20.GL_CLAMP_TO_EDGE);GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T,GLES20.GL_CLAMP_TO_EDGE);return texId;}}复制下一步