安卓 camera 调用流程_[Camera]RK_Camera_HAL

platform: RK3399
OS: Android 7.1
kernel：4.4
参考：
1. KrisFei https://blog.csdn.net/kris_fei/article/details/52451409
2. Kiazhu https://blog.csdn.net/kiazhu/article/details/84652749
3. https://source.android.google.cn/devices/camera?hl=en
4.https://www.cnblogs.com/blogs-of-lxl/p/10981303.html

概述

​ Android Camera Hal 是android framework和kernel联系的重要通道。该部分一般由厂商实现并封装自己的算法或者业务逻辑，进一步抽象出内核设备驱动的各项功能，并通过google实现的统一接口供android framework进行调用。

​ 下面是旧版的Camera框架图(适用于android 8.0以下,android 8.0以上请使用新的camera hal组件及框架)

应用框架

应用代码位于应用框架级别，它利用 android.hardware.Camera API 与相机硬件进行交互。在内部，此代码会调用相应的 JNI 粘合类，以访问与相机互动的原生代码。

JNI

与 android.hardware.Camera 关联的 JNI 代码位于 frameworks/base/core/jni/android_hardware_Camera.cpp 中。此代码会调用较低级别的原生代码以获取对实体相机的访问权限，并返回用于在框架级别创建 android.hardware.Camera 对象的数据。

原生框架

在 frameworks/av/camera/Camera.cpp 中定义的原生框架可提供相当于 android.hardware.Camera 类的原生类。此类会调用 IPC binder 代理，以获取对相机服务的访问权限。

Binder IPC 代理

IPC binder 代理用于促进跨越进程边界的通信。调用相机服务的 3 个相机 binder 类位于 frameworks/av/camera 目录中。 ICameraService 是相机服务的接口；ICamera 是已打开的特定相机设备的接口；ICameraClient 是返回到应用框架的设备接口。

相机服务

位于 frameworks/av/services/camera/libcameraservice/CameraService.cpp 下的相机服务是与 HAL 进行互动的实际代码。

HAL

硬件抽象层定义了由相机服务调用、且您必须实现以确保相机硬件正常运行的标准接口。

内核驱动程序

相机的驱动程序可与实际相机硬件以及您的 HAL 实现进行互动。相机和驱动程序必须支持 YV12 和 NV21 图片格式，以便在显示和视频录制时支持预览相机图片。

Android Camera HAL简介

HAL1简介

功能调用简图

Camera hal1中的camera_module_t 接口

camera_module_t HAL_MODULE_INFO_SYM = {
    common: {
         tag: HARDWARE_MODULE_TAG,
         version_major: ((CONFIG_CAMERAHAL_VERSION&0xffff00)>>16),
         version_minor: CONFIG_CAMERAHAL_VERSION&0xff,
         id: CAMERA_HARDWARE_MODULE_ID,
         name: CAMERA_MODULE_NAME,
         author: "RockChip",
         methods: &camera_module_methods,
         dso: NULL, /* remove compilation warnings */
         reserved: {0}, /* remove compilation warnings */
    },
    get_number_of_cameras: camera_get_number_of_cameras,
    get_camera_info: camera_get_camera_info,
    set_callbacks:NULL,
    get_vendor_tag_ops:NULL,
    }

HAL3简介

​ Camera API2/HAL3架构下使用了全新的CameraMetadata结构取代了之前的SetParameter/Paramters等操作，实现了Java到native到HAL3的参数传递。引入了管道的概念将安卓设备和摄像头之间联系起来，系统向摄像头发送 Capture 请求，而摄像头会返回 CameraMetadata，这一切建立在一个叫作 CameraCaptureSession 的会话中。

功能调用简图

以最常见的android.control Section为例,下图描述了Camera Metadata对不同section以及相应section下不同tag的布局图

RK Camera HAL整体框架

​ canera server通过与CameraHal_Module的标准接口建立和Hal层的连接后，CmaeraHal会通过SensorListener返回消息给上层。如果上层要预览或者拍照，要通过binder机制向Hal层发送命令，Hal层MessageQueue.cpp的消息队列获取到上层的命令后，会通过消息通知器通知CameraHal，CameraHal有一个CommandThread接收命令，收到命令后，最终会下达命令给对应的Adapter去执行对于的动作。

​ RK 的Camera Hal还实现了人脸检测等功能以及相关的辅助函数，就没有在上面的主体框架中画出来。

RK Camera HAL 调用流程

开机启动

CameraHal_Module.cpp --> camera_get_number_of_cameras

-->profiles = camera_board_profiles::getInstance();  // 解析/etc/cam_board.xml 
-->camera_board_profiles::LoadSensor(profiles);   //注册摄像头驱动
     -->OpenAndRegistOneSensor(profiles->mDevieVector[profiles->mXmlDevInfo[i].index]); //最多支持双摄
           -->RegisterSensorDevice(pCamInfo); //注册i2c设备
                -->camsys_fd = open(pSensorInfo->mCamsysDevPath, O_RDWR); //打开dev/camsys_marvin1 设备节点(camsys_marvin1注册见“RK平台摄像头驱动”文章)
               -->ioctl(camsys_fd, CAMSYS_VERCHK, &(pCamInfo->mCamsysVersion)); //检查驱动版本
               --> ioctl(camsys_fd, CAMSYS_REGISTER_DEVIO, &extdev);  //注册摄像头
               -->rk_sensor_pwrseq(camsys_fd, pCamInfo, 1); // 启动摄像头
               --> ioctl(camsys_fd, CAMSYS_I2CWR, &i2cinfo); //配置I2C信息
               --> ioctl(camsys_fd, CAMSYS_QUREYIOMMU, &iommu_enabled); //查询IOMMU使能
               --> ioctl(camsys_fd, CAMSYS_I2CRD, &i2cinfo); //查询设备ID
               -->rk_sensor_pwrseq(camsys_fd, pCamInfo, 0);  //power off
   --> if(pSensorInfo->mFacing == RK_CAM_FACING_FRONT){     //配置摄像头方向
                    camInfoTmp[cam_cnt&0x01].facing_info.facing = CAMERA_FACING_FRONT;      
   --> open(cam_path, O_RDONLY); //打开videoX
   -->ioctl(fd, VIDIOC_QUERYCAP, &capability) //查询capability
   -->rk_cam_total_info* pNewCamInfo = new rk_cam_total_info(); //所有sensor信息都放在里面
   -->ioctl(fd, VIDIOC_ENUM_FMT, &fmtdesc） 
   --> ioctl(fd, VIDIOC_ENUM_FRAMESIZES, &fsize) 
   -->rk_DV_info *pDVResolution = new rk_DV_info(); 
   --> ioctl(fd, VIDIOC_ENUM_FRAMEINTERVALS, &fival)
   --->camera_board_profiles::ProduceNewXml(profiles); //生成media_profiles.xml

App打开Camera

CameraHal_Module.cpp -->camera_device_open

//标准实现
-->camera_device = (rk_camera_device_t*)malloc(sizeof(*camera_device));
--> camera_ops = (camera_device_ops_t*)malloc(sizeof(*camera_ops));
.... //初始化camera_device camera_ops

//RK自己实现
//CameraHal类负责与cameraservice联系，实现
//cameraservice要求实现的接口。此类只负责公共资源的申请，以及任务的分发。
 -->android::CameraHal(cameraid)
    -->commandThreadCommandQ("commandCmdQ") //创建commandCmdQ的message queue
    ->    new IonMemManager 
    mPreviewBuf = new PreviewBufferProvider(mCamMemManager); //previewBuffer
    mVideoBuf = new BufferProvider(mCamMemManager);  //video
    mRawBuf = new BufferProvider(mCamMemManager);   //raw
    mJpegBuf = new BufferProvider(mCamMemManager);  //jpeg
    mUvcBuf = new BufferProvider(mCamMemManager);   //uvc
//根据不同类型的camera new不同类型的adapter
--->mCameraAdapter new CameraIspAdapter(cameraId)
-->mDisplayAdapter = new DisplayAdapter();  //new Display adapter
    -->displayThreadCommandQ    //创建名为displayCmdQ的message queue.
            new DisplayThread    //处理显示相关事物
-->mEventNotifier = new AppMsgNotifier(mCameraAdapter);
    -->encProcessThreadCommandQ //创建名字为pictureEncThreadQ的message queue.
            eventThreadCommandQ    //创建名字为eventThreadQ的message queue.
            create_vpu_memory_pool_allocator  //创建vpu 内存池分配器
            new CameraAppMsgThread    
            new EncProcessThread
            new CameraAppFaceDetThread   //人脸检测线程
            new CameraAppCallbackThread   //回调线程
-->mCameraAdapter->initialize() 
    --->cameraCreate(mCamId)  //创建一个camera
    --->initDefaultParameters(mCamId);   //初始化各个参数默认值
--->mCommandThread = new CommandThread(this);  //command Thread loop
---->mSensorListener = new SensorListener(); //注册Listerner
        ---->new SensorLooperThread
                ---->mLooper->pollOnce    //循环查询是否有event上报，有就会运行sensor_events_listener()获取Orientation。

支持的ops

camera_ops->set_preview_window = camera_set_preview_window;
        camera_ops->set_callbacks = camera_set_callbacks;
        camera_ops->enable_msg_type = camera_enable_msg_type;
        camera_ops->disable_msg_type = camera_disable_msg_type;
        camera_ops->msg_type_enabled = camera_msg_type_enabled;
        camera_ops->start_preview = camera_start_preview;
        camera_ops->stop_preview = camera_stop_preview;
        camera_ops->preview_enabled = camera_preview_enabled;
        camera_ops->store_meta_data_in_buffers = camera_store_meta_data_in_buffers;
        camera_ops->start_recording = camera_start_recording;
        camera_ops->stop_recording = camera_stop_recording;
        camera_ops->recording_enabled = camera_recording_enabled;
        camera_ops->release_recording_frame = camera_release_recording_frame;
        camera_ops->auto_focus = camera_auto_focus;
        camera_ops->cancel_auto_focus = camera_cancel_auto_focus;
        camera_ops->take_picture = camera_take_picture;
        camera_ops->cancel_picture = camera_cancel_picture;
        camera_ops->set_parameters = camera_set_parameters;
        camera_ops->get_parameters = camera_get_parameters;
        camera_ops->put_parameters = camera_put_parameters;
        camera_ops->send_command = camera_send_command;
        camera_ops->release = camera_release;
        camera_ops->dump = camera_dump;

command Thread loop支持的cmd

CMD_PREVIEW_START
CMD_PREVIEW_STOP
CMD_SET_PREVIEW_WINDOW
CMD_SET_PARAMETERS
CMD_PREVIEW_CAPTURE_CANCEL
 CMD_CONTINUOS_PICTURE
 CMD_AF_START
 CMD_AF_CANCEL
 CMD_START_FACE_DETECTION
 CMD_EXIT

设置参数

CameraHal::setParameters    //参数是char*
    setParameters ->     //参数是CameraParameters
        commandThreadCommandQ.put    //cmd是 CMD_SET_PARAMETERS
            commandThread ->    
                mCameraAdapter->setParameters    
                    CameraSOCAdapter::setParameters ->      //假设位soc camera
                        mRefEventNotifier->setPreviewDataCbRes  
                        cameraConfig
                              ioctl(mCamFd, VIDIOC_S_CTRL, &control); //设置 白平衡
                             //设置zoom,color effect,scene,anti banding,white balance
                            //lock ,exposurelock,focus,flash mode,exposure
                              ioctl(mCamFd, VIDIOC_S_EXT_CTRLS, &extCtrInfos);

设置显示窗口

CameraHal::setPreviewWindow ->
    commandThreadCommandQ.put ->    //cmd: CMD_SET_PREVIEW_WINDOW
        commandThread ->
            mDisplayAdapter->setPreviewWindow
            // mDisplayAdapter->startDisplay(app_previw_w, app_preview_h);  //启动预览

Start preview

CameraHal::startPreview
---> commandThreadCommandQ.put -> //cmd: CMD_PREVIEW_START
     -->commandThread
          mParameters.getPreviewSize  //获取预览参数
          mCameraAdapter->getCurPreviewState //获取当前预览状态
        //选择picture size, 如果和preview 分辨率不一样，那需要stop preview再start preview
         selectPreferedDrvSize(&prefered_w,&prefered_h,false); 
          //如果在预览中
          mDisplayAdapter->pauseDisplay
              displayThreadCommandQ.put(&msg)  //CMD_DISPLAY_PAUSE
                     displayThread
                              cameraDisplayBufferDestory();
          mEventNotifier->stopReceiveFrame
                  flushPicture();          //EncProcessThread::CMD_ENCPROCESS_PAUSE
                  pausePreviewCBFrameProcess();    //CameraAppMsgThread::CMD_EVENT_PAUSE
                  stopFaceDection();   //CameraAppFaceDetThread::CMD_FACEDET_PAUSE
          mCameraAdapter->stopPreview
                   cameraStream(false)
                          ioctl(mCamFd, cmd, &type)  
                  mCameraPreviewThread->requestExitAndWait(); //quit preview thread
                  mCameraPreviewThread.clear();
                  cameraStop() 
                   mPreviewBufferProvider->freeBuffer()
          mCameraAdapter->startPreview
                      mPreviewBufProvider->createBuffer   //PREVIEWBUFFER
                               mCamBuffer->createPreviewBuffer
                                      createIonBuffer
                      cameraSetSize(w, h, mCamDriverPreviewFmt,is_capture)
                               ioctl(mCamFd, VIDIOC_S_FMT, &format)
                      cameraStart()
                                ioctl(mCamFd, VIDIOC_REQBUFS, &creqbuf)
                                ioctl(mCamFd, VIDIOC_QUERYBUF, &buffer) 
                                ioctl(mCamFd, VIDIOC_QBUF, &buffer)
                                mmap           
                                cameraStream(true)
                      mCameraPreviewThread = new CameraPreviewThread(this) //进入preview thread loop
           mEventNotifier->startReceiveFrame
           mEventNotifier->startFaceDection  //如果配置了人脸检测功能就开始人脸检测
           //否则同上，只是不用stopPreview

           mDisplayAdapter->startDisplay
                      displayThreadCommandQ.put(&msg) //CMD_DISPLAY_START
                           displayThread              
                                  cameraDisplayBufferDestory()
                      //准备display buffer放frame,然后thread进入休眠等待frame的到来            
                                  cameraDisplayBufferCreate()
          //ioctl stream on后frame就会从kernel上来，接着调用previewthread
          CameraAdapter::previewThread
                      getFrame
                               ioctl(mCamFd, VIDIOC_DQBUF, &cfilledbuffer1) 
                      //判断以什么形式显示：display,video,picture,datacb
                      //判断调用那个notify 这里是CMD_PREVIEWBUF_DISPING
                      mRefDisplayAdapter->notifyNewFrame
                              displayThreadCommandQ.put(&msg)  //CMD_DISPLAY_FRAME
                                    displayThread
                                       mANativeWindow->dequeue_buffer
                                       mANativeWindow->lock_buffer
                                        mANativeWindow->enqueue_buffer
                                         mFrameProvider->returnFrame
                                                    adapterReturnFrame  //把使用好的buffer还给系统
                                                           ioctl(mCamFd, VIDIOC_QBUF, &vb)

Stop preview

camera_stop_preview
         gCameraHals[rk_dev->cameraid]->stopPreview()
                   CameraHal::stopPreview
                              commandThreadCommandQ.put(&msg) //CMD_PREVIEW_STOP
                                      commandThread:
                                                 mDisplayAdapter->pauseDisplay()
                                                 mEventNotifier->stopReceiveFrame()
                                                 mCameraAdapter->stopPreview()
                                                       cameraStream(false)
                                                                  ioctl(mCamFd, cmd, &type)  
                                                       mCameraPreviewThread->requestExitAndWait(); //quit preview thread
                                                      mCameraPreviewThread.clear();
                                                      cameraStop() 
                                                      mPreviewBufferProvider->freeBuffer()

Take Picture

camera_take_picture
        gCameraHals[rk_dev->cameraid]->takePicture()
               CameraHal::commandThread   //CMD_CONTINUOS_PICTURE
                fillPicturInfo(picinfo)
                mEventNotifier->takePicture(picinfo)
                        AppMsgNotifier::takePicture     //设置running state为STA_RECEIVE_PIC_FRAME，等着previewThread去get frame
          //previewThread接到frame之后
         CameraAdapter::previewThread
                  mRefEventNotifier->isNeedSendToPicture
                  mRefEventNotifier->notifyNewPicFrame
                        AppMsgNotifier::notifyNewPicFrame
         //cmd: EncProcessThread::CMD_ENCPROCESS_SNAPSHOT
                               encProcessThreadCommandQ.put()  
           encProcessThread
                    captureEncProcessPicture
                                copyAndSendCompressedImage
                                     callback_compressed_image
                                                  CameraAppCallbackThread:: CMD_MSG_COMPRESSED_IMAGE
                                                          mDataCb
                                                          frame->release(frame)
                    mFrameProvider->returnFrame

Record Video

CameraAdapter::previewThread 
    getFrame
    mRefEventNotifier->notifyNewVideoFrame
        eventThreadCommandQ.put  //cmdCameraAppMsgThread::CMD_EVENT_VIDEO_ENCING
            AppMsgNotifier::eventThread 
                processVideoCb 
                    AppMsgNotifier::processVideoCb 
                        callback_video_frame 
                            callbackThreadCommandQ.put //cmd:CameraAppCallbackThread::CMD_MSG_VIDEO_FRAME
                                AppMsgNotifier::callbackThread ->
                                    mDataCbTimestamp

Preview datacallback

CameraAdapter::previewThread ->
    mRefEventNotifier->notifyNewPreviewCbFrame    ->    //前提是设置了callback回调函数
        AppMsgNotifier::notifyNewPreviewCbFrame ->
            eventThreadCommandQ.put ->    //cmd: CameraAppMsgThread::CMD_EVENT_PREVIEW_DATA_CB
                AppMsgNotifier::eventThread ->
                    processPreviewDataCb ->
                        AppMsgNotifier::processPreviewDataCb ->    //格式转换在这里完成
                            cameraFormatConvert
                            callback_preview_frame ->
                                callbackThreadCommandQ.put    ->    //cmd: CameraAppCallbackThread::CMD_MSG_PREVIEW_FRAME
                                    AppMsgNotifier::callbackThread ->
                                        mDataCb    //调用上层传递下来的callback.
                                        frame->release    //release buffer.
                    mFrameProvider->returnFrame

个人博客：https://www.letcos.top/