Camera-MTK OpenCamera时序以及耗时

OpenCamera时序分析

1.应用层

08:53:56.087 应用层调用openCamera接口

2.CameraService层

08:53:56.099 10ms左右CameraService(进程号:742 线程号:25763)进行相关打印，从应用层到CameraServer之间经过FrameWork层(同一进程)以及ICameraService与CameraService之间的跨进程AIDL通信。此时CameraManagera(进程号:29471 线程号:29508)的相关部分处于阻塞状态，等待CameraService处理完成。

3.CameraProvider(Camera HAL Interface-CameraProviderImpl SP端)

08:53:56.116 17ms左右CameraProvider的device部分进行相关的log打印，从CameraService(CameraProviderManager)到CameraProvider(ICameraProviderImpl)之间有着跨进程的HIDL通信，此时CameraProviderManager(进程号:742 线程号:25763)处于阻塞状态，等待ICameraProviderImpl相关处理完成。

 

4.Camera HAL Interface层

08:53:56.151 耗时35ms Camera HAL Interface(很多人把这个层级称为CameraProvider层，可能因为它跟CameraServer是通过ICameraProviderImpl进行交互的)(进程号:773 线程号:5314)完成相关操作，之前被阻塞的CameraServer的CameraProviderManager拿到相应的信息，开始执行，但是此时并没有openCamera，才刚刚powerOn拉高camera。

而且Camera HAL Interface相关的时序以及逻辑是在手机启动之后就已经起来的，与相机有没有打开没有关系，且里面会分成Provider、Commmon、Device、Session等好多模块对相机的open、preview、capture动作进行传递。

5.Camera HAL层

08:53:56.151 开始执行Camera HAL层(进程号:773 线程号:29521)相关的时序及逻辑

08:53:56.220 直到应用层/FrameWork层CameraManagera、ICameraService相关的进程结束阻塞，Camera HAL里面还有很多在执行。

 

6.Log时序总结

openCamera是从08:53:56.087-08:53:56.220结束

CameraProviderManager - ICameraProviderImpl HIDL跨进程通信耗时35ms

ICameraService - CameraService AIDL跨进程通信中主要的阻塞、等待或者消费方式以及接口需要更进一步研究与明确耗费时间。

OpenCamera调用接口分析

1.应用层

 

 

 

2.Camera FrameWork

可以在这边打Log，相关进程会hang在这边connectDevice的调用下

3.CameraService 

CameraSevice里面与openCamera相关的重要节点都在connectHelper里面了，比如：

handleEvictionsLocked：这里面会处理所有的camera id冲突场景，且会给不同调用camera id的packages进行打分操作。

 下面的ClientDescriptor相关的还没搞懂。

 

makeClient：Binder的Bn端，会初始化CameraDeviceClient、Camera2ClientBase、CameraDeviceClientBase以及调用其initialize方法。

 

 

 这边InitializeImpl主要最主要的是run了FrameProcess这个线程，这个线程就在后面的过程中用来接收预览帧数据。

 这边关键节点是startCameraOps、然后又调用了CameraDeviceBase的initialize方法

 可以看到这边定义了很多纯虚函数，需要找到其具体的实现接口

相关方法的接口具体实现都在Camera3Device里面

 CameraProviderManager里面的deviceInfo3在刚开机时候CameraServer与CameraProvider之间的交互过程中已经获取了。

 

4.Camera HardWare Interface(CameraProvider)

这边其实和展锐平台相关的HardWare Interface相关节点都是一样的

主要就是几个关键的结构体的传递，建立了Camera HardWare Interface和Camera HAL之间的联系。

1.hw_device_t里面有hw_module_t，可以通过hw_module_t去调用open方法。

2.hw_module_t这边在里面封装了hw_module_methods_t的函数指针，指向了open的实例化函数接口。

3.camera3_device_t在MTK camera HAL架构中貌似没有用到，在展讯的平台架构中使用场景还是挺多的。

 

 4.camera_module_t 这个结构体算是链接Camera HardWare Interface和Camera HAL层最为紧密的纽带了。

 5.Camera HAL

MTK HAL层有三个entry接口，具体咋么实现什么情况对应哪个需要再去研究。

vendor/mediatek/proprietary/hardware/mtkcam3/main/mtkhal/hidl/depend/entry.cpp

可以看到对Camera HardWare Interface里面camera_module_t的所有接口说明都在这边进行了实例化。接下来对一些比较重要的接口进行详细说明和解释

extern "C" NSCam::legacy_dev3::LegacyCameraModule* getLegacyCameraModule() {
  static NSCam::CameraDeviceManagerImpl singleton("legacy");
  static bool init = singleton.initialize();
  if (!init) {
    MY_LOGE("CameraDeviceManagerImpl::initialize fail %p", &singleton);
    return nullptr;
  }
  static NSCam::legacy_dev3::LegacyCameraModule* module =
      createLegacyCameraModule(&singleton);
  if (module == nullptr) {
    MY_LOGE("LegacyCameraModule is null: %p", module);
    return nullptr;
  }
  return module;
}


//  Implementation of hw_module_methods_t

static int open_device(hw_module_t const* module,
                       const char* name,
                       hw_device_t** device) {
  // return  getLegacyCameraModule()->open(device, module, name);
#warning "set device_version to 0 to avoid build error"
  // device_version = 0 would query info every time
  return getLegacyCameraModule()->open(device, module, name, 0);
}

static hw_module_methods_t* get_module_methods() {
  static hw_module_methods_t _methods = {.open = open_device};
  return &_methods;
}

vendor/mediatek/proprietary/hardware /mtkcam3/main/mtkhal/devicemgr/depend/CameraDeviceManagerImpl.cpp

class CameraDeviceManagerImpl : public CameraDeviceManagerBase {
  //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  //  Implementations.
  //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 public:      Instantiation.
  explicit CameraDeviceManagerImpl(char const* type);
  virtual ~CameraDeviceManagerImpl();

 protected:      Operations.
  virtual auto onEnumerateDevicesLocked() -> ::android::status_t;

  virtual auto onGetMaxNumOfMultiOpenCameras() const -> uint32_t;

  virtual auto onValidateOpenLocked(
      const std::shared_ptr<IVirtualDevice>& pVirtualDevice) const
      -> ::android::status_t;

  virtual auto onAttachOpenDeviceLocked(
      const std::shared_ptr<IVirtualDevice>& pVirtualDevice) -> void;

  virtual auto onDetachOpenDeviceLocked(
      const std::shared_ptr<IVirtualDevice>& pVirtualDevice) -> void;

  virtual auto onEnableTorchLocked(const std::string& deviceName, bool enable)
      -> ::android::status_t;
};

vendor/mediatek/proprietary/hardware/mtkcam3/main/mtkhal/devicemgr/base/CameraDeviceManagerBase.cpp

vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/main/mtkhal/devicemgr/ICameraDeviceManager.h

class ICameraDeviceManager {
  //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  //  Definitions.
  //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 public:  
  class IVirtualDevice : virtual public ::android::RefBase {
   public:
    /**
     * The device instance names must be of the form
     *      "device@<major>.<minor>/<type>/<id>" where
     * <major>/<minor> is the HIDL version of the interface.
     * <id> is a small incrementing integer for "internal" device types,
     * with 0 being the main back-facing camera and 1 being the main
     * front-facing camera, if they exist.
     */

    struct Info {
      std::string mInstanceName;   //  instance device:
                                   //  "device@<major>.<minor>/<type>/<id>"
      int32_t mInstanceId;         //  instance id
      int32_t mVirtualInstanceId;  //  virtual instance id
      uint32_t mMajorVersion;      //  major version
      uint32_t mMinorVersion;      //  minor version
      bool mHasFlashUnit;          //  has flash unit
      bool mIsHidden;              //  Is camera hidden
      int32_t mFacing;             //  facing
//      mcam::DeviceType mType;    //  NormalDevice/PostProcDevice/...
    };

   public:
    virtual auto getDeviceInterface(std::shared_ptr<IVirtualDevice>& rpDevice)
        const -> ::android::status_t = 0;
    virtual auto getDeviceInfo() const -> Info const& = 0;
    virtual auto getInstanceName() const -> char const* {
      return getDeviceInfo().mInstanceName.c_str();
    }

    ...

}

vendor/mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/CameraDevice3Impl.h

        ICameraDeviceManager.h里面并没有相关接口的实例化，真正的接口实例化是在CameraDevice3Impl.cpp里面实现的。

class CameraDevice3Impl
    : public ICameraDeviceManager::IVirtualDevice
{
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//  Definitions.
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// public:     

//     struct  MyDebuggee : public IDebuggee
//     {
//         static const std::string        mName;
//         std::shared_ptr<IDebuggeeCookie>mCookie = nullptr;
//         android::wp<CameraDevice3Impl>  mContext = nullptr;

//                         MyDebuggee(CameraDevice3Impl* p) : mContext(p) {}
//         virtual auto    debuggeeName() const -> std::string { return mName; }
//         virtual auto    debug(
//                             android::Printer& printer,
//                             const std::vector<std::string>& options
//                         ) -> void;
//     };

//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//  Data Members.
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
protected:  

    //  setup during constructor
    int32_t                                 mLogLevel = 0;                  //  log level.
    ICameraDeviceManager*                   mDeviceManager = nullptr;       //  device manager.
    std::shared_ptr<Info>                   mStaticDeviceInfo = nullptr;    //  device info
    // std::shared_ptr<MyDebuggee>             mDebuggee = nullptr;
    ::android::sp<IMetadataProvider>        mMetadataProvider = nullptr;
    ::android::sp<IMetadataConverter>       mMetadataConverter = nullptr;
    ::android::sp<ICameraDevice3Session>    mSession = nullptr;
    std::map<uint32_t, ::android::sp<IMetadataProvider>> mPhysicalMetadataProviders;
    mutable ::android::Mutex                mGetResourceLock;

//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//  Operations.
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public:         Instantiation.
    virtual         ~CameraDevice3Impl();
                    CameraDevice3Impl(
                        ICameraDeviceManager* deviceManager,
                        IMetadataProvider* metadataProvider,
                        std::map<uint32_t, ::android::sp<IMetadataProvider>>const& physicalMetadataProviders,
                        char const* deviceType,
                        int32_t instanceId,
                        int32_t virtualInstanceId);

    virtual auto    initialize(
                        ICameraDevice3Session* session
                        ) -> bool;

public:         Operations.
    auto            getLogLevel() const             { return mLogLevel; }
    auto const&     getStaticDeviceInfo() const     { return mStaticDeviceInfo; }
    auto const&     getMetadataConverter() const    { return mMetadataConverter; }

//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//  Interfaces.
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public:         ICameraDeviceManager::IVirtualDevice Interfaces.

    virtual auto    getDeviceInterface(
                        ::android::sp<CameraDevice3Impl>& rpDevice
                        ) const -> ::android::status_t override;

    virtual auto    getDeviceInfo() const -> Info const& override;

public:         ICameraDevice Interfaces implement.

    virtual auto    getResourceCost(v3::CameraResourceCost& mtkCost) -> int;

    virtual auto    setTorchMode(v3::TorchMode mode) -> ::android::status_t;

    virtual auto    open() -> ::android::sp<ICameraDevice3Session>;

    virtual auto    dumpState(android::Printer& printer, const std::vector<std::string>& options) -> void;

    virtual auto    getCameraCharacteristics() -> const camera_metadata_t*;

    virtual auto    getPhysicalCameraCharacteristics(
                        int physicalId,
                        camera_metadata* physicalMetadata) -> int;

    virtual auto    isStreamCombinationSupported(
                        const v3::StreamConfiguration& streams,
                        bool& isSupported) -> int;

public:         Operations

    virtual auto    getDeviceSession() -> const ::android::sp<ICameraDevice3Session>;

};

到这边HAL openCamera相关的动作只算走了一半，仅仅是entry.cpp里面实例化CameraDeviceManagerImpl实例化的过程。

这边已经开始对sensor进行了相关的init操作

真正的camera device open操作会在LegacyCameraModule里面，上面的CameraDevice3Impl也有相关的open操作，主要是CameraDeviceSession的调用。

vendor/mediatek/proprietary/hardware/mtkcam3/main/hal/entry/legacy/device/LegacyCameraDevice3.cpp

vendor/mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/CameraDevice3Impl.cpp

相关的mSession的实例化的操作在initialize这边，调用的是CameraDevice3Session的open操作

 接下来要追踪的便是CameraDeviceImpl.initial里面的session是谁传过来的，是哪边创建的、创建的什么tag。

vendor/mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/CameraDevice3Factory.cpp

extern "C"
NSCam::ICameraDeviceManager::IVirtualDevice*
createVirtualCameraDevice(CreateVirtualCameraDeviceParams* params)
{
    MY_LOGD("[hidldc] new createVirtualCameraDevice");
    if ( ! params || ! params->pDeviceManager || ! params->pMetadataProvider ) {
        MY_LOGE("Bad params");
        return nullptr;
    }

    auto pDevice = new CameraDevice3Impl(
        params->pDeviceManager,
        params->pMetadataProvider,
        params->physicalMetadataProviders,
        params->deviceType,
        params->instanceId,
        params->virtualInstanceId
    );

    if ( ! pDevice ) {
        MY_LOGE("Fail to new CameraDevice3Impl");
        return nullptr;
    }

    NSCam::ICameraDevice3Session::CreationInfo const info = {
        .mDeviceManager     = params->pDeviceManager,
        .mStaticDeviceInfo  = pDevice->getStaticDeviceInfo(),
        .mMetadataProvider  = params->pMetadataProvider,
        .mMetadataConverter = pDevice->getMetadataConverter(),
        .mPhysicalMetadataProviders = params->physicalMetadataProviders
    };

    bool bInitSuccess = pDevice->initialize(createCameraDevice3Session(info));
    if ( ! bInitSuccess ) {
        delete pDevice;
        pDevice = nullptr;
    }

    return pDevice;
}

createCameraDevice3Session的接口的实现是在CameraDevice3SessionFactory.cpp里面实现的

extern "C"
NSCam::ICameraDevice3Session*
createCameraDevice3Session(
    NSCam::ICameraDevice3Session::CreationInfo const& info
)
{
    //  Based on the information of instanceId + pMetadataProvider,
    //  decide which device session to create.
    return new NSCam::v3::CameraDevice3SessionImpl(info);
}

然后的主要实现就在CameraDevice3SessionImpl

vendor/mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/CameraDevice3SessionImpl.cpp

         这边不在做仔细的剖析了，里面涉及很多接口的调用，直接贴上最关键的节点PipelineMode->open，这样就与MTK 的pipeline mode进行了对接。

        其实在实际解决问题时候，很多人不会去check Camera HAL的前期框架，关注点更会放在MTK Pipeline的框架上，整个流程及框架肯定是越通透对以后的性能、时序分析更加有帮助，能多研究点就多看点呗。

总结

        MTK Camera HAL性能到这边远没有结束，总的MTK Camera HAL层openCamera架构可以简单的以线程号分成三个大的部分，现在仅仅是对PipelineMode之前的部分进行了流程调用分析，其中的耗时以及等待机制还需要打更多的log去追，以后的优化更不是简简单单能搞定的。