Camera HAL3的整体架构和流程（二）

前言

上一篇文章大概介绍了一下Camera的整体架构，先回顾一下：

上图是Google官方的Camera架构，Google封装了HAL层接口：ICameraProvider, ICameraDevice, ICameraDeviceSession。

这几个接口的作用如下：

• ICameraProvider：用于枚举单个设备并管理其状态。
• ICameraDevice：相机设备接口。
• ICameraDeviceSession：活跃的相机设备会话接口。

MTK HAL3结构主要实现了这几个接口， CameraProviderImpl实现了ICameraProvider接口，CameraDevice3Impl实现了ICameraDevice接口，CameraDevice3SessionImpl实现了ICameraDeviceSession接口。

Google 将相机子系统塑造为一个管道（pipeline），MTK实现了一个PipelineModelImpl来管理pipeline。

本文详细地描述了从framework到HAL层open camera的整个过程。

open Camera流程

总结MTK平台open Camera的整个流程，忽略callback过程，以下是整个过程中设计到的文件路径：

文件名	所在路径
CameraManager.java	frameworks/base/core/java/android/hardware/camera2/
CameraService.cpp	frameworks/av/services/camera/libcameraservice/
CameraDeviceClient.cpp	frameworks/av/services/camera/libcameraservice/api2/
Camera2ClientBase.cpp	frameworks/av/services/camera/libcameraservice/common/
Camera3Device.cpp	frameworks/av/services/camera/libcameraservice/device3/
CameraProviderManager.cpp	frameworks/av/services/camera/libcameraservice/common/
CameraDevice3Impl.cpp	mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/
CameraDevice3SessionImpl.cpp	mediatek/proprietary/hardware/mtkcam3/main/hal/device/3.x/device/
PipelineModelImpl.cpp	mediatek/proprietary/hardware/mtkcam3/pipeline/model/
HalDeviceAdapter.cpp	mediatek/proprietary/hardware/mtkcam3/pipeline/model/adapter/
HalSensorList.openList.cpp	mediatek/proprietary/hardware/mtkcam/drv/src/sensor/common/v1/
HalSensor.cpp	mediatek/proprietary/hardware/mtkcam/drv/src/sensor/common/v1/
imgsensor_drv.cpp	mediatek/proprietary/hardware/mtkcam/drv/src/sensor/common/v1/
IHalSensor.h	/mediatek/proprietary/hardware/mtkcam/include/mtkcam/drv/

MTK 代码路径说明

下图是整个open camera的流程，可以结合下面的具体步骤一起看。

由Google的Camera框架可知open Camera从CameraManager.java开始（CameraDevice.java负责发出capture命令，这里只讲open）

CameraManager.java

openCamera()==> openCameraForUid() ==>openCameraDeviceUserAsync()

直接看openCameraDeviceUserAsync()函数，限于篇幅，代码部分省略。
需要注意的是第19行ICameraService cameraService = CameraManagerGlobal.get().getCameraService();这里通过Binder获取了CameraService实例。再看第25行cameraUser = cameraService.connectDevice。然后deviceImpl.setRemoteDevice(cameraUser); device = deviceImpl; 最后return device。
由connectDevice往下，到CameraService::connectDevice()，从这里开始进入CameraService！

private CameraDevice openCameraDeviceUserAsync(String cameraId, CameraDevice.StateCallback callback, Executor executor, final int uid)throws CameraAccessException {
        CameraCharacteristics characteristics = getCameraCharacteristics(cameraId);
        CameraDevice device = null;
        
        synchronized (mLock) {
            ICameraDeviceUser cameraUser = null;
            android.hardware.camera2.impl.CameraDeviceImpl deviceImpl =
                    new android.hardware.camera2.impl.CameraDeviceImpl(
                        cameraId,
                        callback,
                        executor,
                        characteristics,
                        mContext.getApplicationInfo().targetSdkVersion);

            ICameraDeviceCallbacks callbacks = deviceImpl.getCallbacks();
            try {
                if (supportsCamera2ApiLocked(cameraId)) {
                    // Use cameraservice's cameradeviceclient implementation for HAL3.2+ devices
                    ICameraService cameraService = CameraManagerGlobal.get().getCameraService();//19行
                    if (cameraService == null) {
                        throw new ServiceSpecificException(
                            ICameraService.ERROR_DISCONNECTED,
                            "Camera service is currently unavailable");
                    }
                    cameraUser = cameraService.connectDevice(callbacks, cameraId,
                            mContext.getOpPackageName(), uid);
                } else {
                    // Use legacy camera implementation for HAL1 devices
                    int id;
                    try {
                        id = Integer.parseInt(cameraId);
                    } catch (NumberFormatException e) {
                        throw new IllegalArgumentException("Expected cameraId to be numeric, but it was: "
                                + cameraId);
                    }

                    Log.i(TAG, "Using legacy camera HAL.");
                    cameraUser = CameraDeviceUserShim.connectBinderShim(callbacks, id,
                            getDisplaySize());
                }
            } 
            ...
                
            // TODO: factor out callback to be non-nested, then move setter to constructor
            // For now, calling setRemoteDevice will fire initial
            // onOpened/onUnconfigured callbacks.
            // This function call may post onDisconnected and throw CAMERA_DISCONNECTED if
            // cameraUser dies during setup.
            deviceImpl.setRemoteDevice(cameraUser);
            device = deviceImpl;
        }

        return device;
    }

CameraService.cpp

看connectDevice()函数，需要注意第14行ret = connectHelper(…,client),其最后一个参数client作为输出。

Status CameraService::connectDevice(
        const sp<hardware::camera2::ICameraDeviceCallbacks>& cameraCb,
        const String16& cameraId,
        const String16& clientPackageName,
        int clientUid,
        /*out*/
        sp<hardware::camera2::ICameraDeviceUser>* device) {

    ATRACE_CALL();
    Status ret = Status::ok();
    String8 id = String8(cameraId);
    sp<CameraDeviceClient> client = nullptr;

    ret = connectHelper<hardware::camera2::ICameraDeviceCallbacks,CameraDeviceClient>(cameraCb, id,
            /*api1CameraId*/-1,
            CAMERA_HAL_API_VERSION_UNSPECIFIED, clientPackageName,
            clientUid, USE_CALLING_PID, API_2, /*shimUpdateOnly*/ false, /*out*/client);
    *device = client;
    return ret;
}

直接看connectHelper()，需要注意的是第10行ret = makeClient(…, &tmp)，tmp作为输出, 得到client后第22行调用client->initialize().那么这个client是谁的实例呢，沿着makeClient函数跟下去。

Status CameraService::connectHelper(const sp<CALLBACK>& cameraCb, const String8& cameraId,
        int api1CameraId, int halVersion, const String16& clientPackageName, int clientUid,
        int clientPid, apiLevel effectiveApiLevel, bool shimUpdateOnly,
        /*out*/sp<CLIENT>& device) {
    ...
    sp<CLIENT> client = nullptr;
    {
    ...
        sp<BasicClient> tmp = nullptr;
        if(!(ret = makeClient(this, cameraCb, clientPackageName,
                cameraId, api1CameraId, facing,
                clientPid, clientUid, getpid(),
                halVersion, deviceVersion, effectiveApiLevel,
                /*out*/&tmp)).isOk()) {
            return ret;
        }
        client = static_cast<CLIENT*>(tmp.get());

        LOG_ALWAYS_FATAL_IF(client.get() == nullptr, "%s: CameraService in invalid state",
                __FUNCTION__);
                
        err = client->initialize(mCameraProviderManager, mMonitorTags);
        if (err != OK) {
            ...
        }
        ...
    } // lock is destroyed, allow further connect calls
    // Important: release the mutex here so the client can call back into the service from its
    // destructor (can be at the end of the call)
    device = client;
    return ret;
}

makeClient函数也在CameraService.cpp中，需要注意的是第22行 *client = new CameraDeviceClient，因为本文是HAL3+API2，所以走CameraDeviceClient构造函数.

Status CameraService::makeClient(const sp<CameraService>& cameraService,
        const sp<IInterface>& cameraCb, const String16& packageName, const String8& cameraId,
        int api1CameraId, int facing, int clientPid, uid_t clientUid, int servicePid,
        int halVersion, int deviceVersion, apiLevel effectiveApiLevel,
        /*out*/sp<BasicClient>* client) {
		...
          case CAMERA_DEVICE_API_VERSION_3_0:
          case CAMERA_DEVICE_API_VERSION_3_1:
          case CAMERA_DEVICE_API_VERSION_3_2:
          case CAMERA_DEVICE_API_VERSION_3_3:
          case CAMERA_DEVICE_API_VERSION_3_4:
          case CAMERA_DEVICE_API_VERSION_3_5:
            if (effectiveApiLevel == API_1) { // Camera1 API route
                sp<ICameraClient> tmp = static_cast<ICameraClient*>(cameraCb.get());
                *client = new Camera2Client(cameraService, tmp, packageName,
                        cameraId, api1CameraId,
                        facing, clientPid, clientUid,
                        servicePid);
            } else { // Camera2 API route
                sp<hardware::camera2::ICameraDeviceCallbacks> tmp =
                        static_cast<hardware::camera2::ICameraDeviceCallbacks*>(cameraCb.get());
                *client = new CameraDeviceClient(cameraService, tmp, packageName, cameraId,
                        facing, clientPid, clientUid, servicePid);
            }
            break;
          default:
            // Should not be reachable
            ALOGE("Unknown camera device HAL version: %d", deviceVersion);
            return STATUS_ERROR_FMT(ERROR_INVALID_OPERATION,
                    "Camera device \"%s\" has unknown HAL version %d",
                    cameraId.string(), deviceVersion);
        }
    } 
    ...
    return Status::ok();
}

CameraDeviceClient.cpp

调用CameraDeviceClient的构造函数，其继承于Camera2ClientBase。继续到Camera2ClientBase.cpp

CameraDeviceClient::CameraDeviceClient(...) : Camera2ClientBase(...),
    mInputStream(),
    mStreamingRequestId(REQUEST_ID_NONE),
    mRequestIdCounter(0) {

    ATRACE_CALL();
    ALOGI("CameraDeviceClient %s: Opened", cameraId.string());
}

Camera2ClientBase.cpp

调用Camera2ClientBase构造函数，可以看到其继承了mDevice(new Camera3Device)，Camera3Device正是我们需要的！

Camera2ClientBase<TClientBase>::Camera2ClientBase(...):
        ...
        mDevice(new Camera3Device(cameraId)),
        mDeviceActive(false), mApi1CameraId(api1CameraId)
{
    ALOGI("Camera %s: Opened. Client: %s (PID %d, UID %d)", cameraId.string(),
            String8(clientPackageName).string(), clientPid, clientUid);

    mInitialClientPid = clientPid;
    LOG_ALWAYS_FATAL_IF(mDevice == 0, "Device should never be NULL here.");
}

Camera3Device.cpp

最后调用了Camera3Device的构造方法，到这makeClient最后输出的client找到了！它就是Camera3Device的实例！所以client->initialize等同于Camera3Device::initialize。

看initialize(sp <CameraProviderManager> manager, …)函数，需要注意的是第15行res = manager->openSession(mId.string(), this,/*out*/ &session);其中输出是sp <ICameraDeviceSession> session。这里调用了CameraProviderManager::openSession

status_t Camera3Device::initialize(sp<CameraProviderManager> manager, const String8& monitorTags) {
    ATRACE_CALL();
    Mutex::Autolock il(mInterfaceLock);
    Mutex::Autolock l(mLock);

    ALOGV("%s: Initializing HIDL device for camera %s", __FUNCTION__, mId.string());
    if (mStatus != STATUS_UNINITIALIZED) {
        CLOGE("Already initialized!");
        return INVALID_OPERATION;
    }
    if (manager == nullptr) return INVALID_OPERATION;

    sp<ICameraDeviceSession> session;
    ATRACE_BEGIN("CameraHal::openSession");
    status_t res = manager->openSession(mId.string(), this,
            /*out*/ &session);
    ATRACE_END();
  ...
}

CameraProviderManager.cpp

看openSession()函数，需要注意第14行，ret = interface->open()，其中interface=deviceInfo3->startDeviceInterface，实际上是调用CameraProviderManager::ProviderInfo::DeviceInfo3::startDeviceInterface，这个函数最终会调用ICameraProvider接口的getCameraDeviceInterface_V3_x来获取ICameraDevice的实例。
从这就开始进入HAL层了！

status_t CameraProviderManager::openSession(const std::string &id,
        const sp<device::V3_2::ICameraDeviceCallback>& callback,
        /*out*/
        sp<device::V3_2::ICameraDeviceSession> *session) {
	...
    auto *deviceInfo3 = static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo);
	...
    auto interface = deviceInfo3->startDeviceInterface<
            CameraProviderManager::ProviderInfo::DeviceInfo3::InterfaceT>();
    if (interface == nullptr) {
        return DEAD_OBJECT;
    }

    ret = interface->open(callback, [&status, &session]
            (Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });
    ...
    return mapToStatusT(status);
}

CameraDevice3Impl.cpp

前面说过CameraDevice3Impl.cpp是MTK对Google HAL接口ICameraDevice的实现，所以上一步的interface->open, 实际上是CameraDevice3Impl::open

看open()函数，需要注意第11行status = mSession->open，其中mSession在CameraDevice3Impl.h中定义::android::sp <ICameraDevice3Session> mSession = nullptr;，也就是调用了CameraDevice3Session::open

Return<void>
CameraDevice3Impl::
open(const ::android::sp<V3_2::ICameraDeviceCallback>& callback, open_cb _hidl_cb)
{
    int systraceLevel = ::property_get_int32("vendor.debug.mtkcam.systrace.level", MTKCAM_SYSTRACE_LEVEL_DEFAULT);
    MY_LOGI("open camera3 device (%s) systraceLevel(%d) instanceId(%d) vid(%d)",
                        mStaticDeviceInfo->mInstanceName.c_str(),
                        systraceLevel,
                        mStaticDeviceInfo->mInstanceId,
                        mStaticDeviceInfo->mVirtualInstanceId);
    ::android::status_t status = mSession->open(V3_5::ICameraDeviceCallback::castFrom(callback));
    if  ( ::android::OK != status ) {
        _hidl_cb(mapToHidlCameraStatus(status), nullptr);
    }
    else {
        _hidl_cb(mapToHidlCameraStatus(status), mSession);
    }

    return Void();
}

CameraDevice3SessionImpl.cpp

看open()函数，需要注意的是第9行err = onOpenLocked(callback); onOpenLocked定义在CameraDevice3SessionImpl.cpp

auto ThisNamespace::open(
    const ::android::sp<V3_5::ICameraDeviceCallback>& callback
) -> ::android::status_t
{
    ...
            //------------------------------------------------------------------
            int err = NO_INIT;
            status = tryRunCommandLocked(getWaitCommandTimeout(), "onOpenLocked", [&, this](){
                err = onOpenLocked(callback);
            });
            if ( status == OK ) {
                status = err;
       ...
    return status;
}

继续看onOpenLocked()函数，需要注意的是第22行err = pPipelineModel->open();也就是调用了PipelineModelImpl::open, 从这里开始进入pipeline！

auto ThisNamespace::onOpenLocked(
    const ::android::sp<V3_4::ICameraDeviceCallback>& callback
) -> ::android::status_t
{
    ...
    //--------------------------------------------------------------------------
    {
        Mutex::Autolock _l(mPipelineModelLock);
        auto pPipelineModelMgr = IPipelineModelManager::get();
        if  ( CC_UNLIKELY(pPipelineModelMgr == nullptr) ) {
            MY_LOGE("IPipelineModelManager::get() is null object!");
            return NO_INIT;
        }
        //
        auto pPipelineModel = pPipelineModelMgr->getPipelineModel( getInstanceId() );
        if ( CC_UNLIKELY(pPipelineModel == nullptr) ) {
            MY_LOGE("IPipelineModelManager::getPipelineModel(%d) is null object!", getInstanceId());
            return NO_INIT;
        }
        //
        ::android::status_t err = OK;
        err = pPipelineModel->open(getInstanceName().c_str(), this);
        if  ( CC_UNLIKELY(OK != err) ) {
            MY_LOGE( "fail to IPipelinemodel->open() status:%d(%s)", -err, ::strerror(-err) );
            return NO_INIT;
        }
        mPipelineModel = pPipelineModel;
        mConfigTimestamp = (uint64_t)::systemTime();
        MY_LOGD("timestamp(%" PRIu64 ")", mConfigTimestamp);
    }
    //--------------------------------------------------------------------------
    return OK;
}

PipelineModelImpl.cpp

看open()函数,需要注意的是第18行和19行，分别是mHalDeviceAdapter->open()，mHalDeviceAdapter->powerOn()。接下来看HalDeviceAdapter.cpp

auto PipelineModelImpl::open(
    std::string const& userName,
    android::wp<IPipelineModelCallback> const& callback
) -> int
{
    MY_LOGD1("+");
    //
    {
        std::lock_guard<std::timed_mutex> _l(mLock);

        mUserName = userName;
        mCallback = callback;

        mvOpenFutures.push_back(
            std::async(std::launch::async,
                [this]() {
                    return CC_LIKELY( mHalDeviceAdapter!=nullptr )
                        && CC_LIKELY( mHalDeviceAdapter->open() )
                        && CC_LIKELY( mHalDeviceAdapter->powerOn() );
                }
            )
        );
    }
    //
    MY_LOGD1("-");
    return OK;
}

HalDeviceAdapter.cpp

先看open()函数，需要注意的是第4行auto pHalSensorList = MAKE_HalSensorList(); 其中MAKE_HalSensorList()在IHalSensor.h中定义。第17行auto pSensor = pHalSensorList->createSensor ，实际是调用了HalSensorList::createSensor()，这在HalSensorList.openList.cpp中定义

virtual auto    open() -> bool override
                    {
                        CAM_TRACE_NAME("Sensor creation");
                        auto pHalSensorList = MAKE_HalSensorList();
                        if  ( CC_UNLIKELY(pHalSensorList == nullptr) ) {
                            MY_LOGE("Bad HalSensorList");
                            return false;
                        }

                        if ( CC_UNLIKELY(mvPhySensorId.size() == 0) ) {
                            MY_LOGE("No physical sensor");
                            return false;
                        }
                        //
                        bool ret = true;
                        for (size_t i = 0; i < mvPhySensorId.size(); i++) {
                            auto pSensor = pHalSensorList->createSensor(mName.c_str(), mvPhySensorId[i]);
                            mvHalSensor.push_back(pSensor);
                            if ( CC_UNLIKELY(pSensor == nullptr) ) {
                                ret = false;
                                MY_LOGE("Bad HalSensor - mvPhySensorId[%zu]=%d", i, mvPhySensorId[i]);
                            }
                        }

                        if  (CC_UNLIKELY(!ret)) {
                            MY_LOGE("Fail on open(). [TODO] let's clean up resources");
                        }
                        mIsOpen = ret;
                        return ret;
                    }

再看powerOn()函数，需要注意的是第18行mvHalSensor[i]->powerOn，它调用了HalSensor::powerOn。

virtual auto    powerOn() -> bool override
                    {
                        ...
                        // create thread to power on sensors
                        std::future<bool> future_initSensor =
                            std::async(std::launch::async,
                                [ this ]() {
                                    CAM_TRACE_NAME("Sensors powerOn");
                                    ::prctl(PR_SET_NAME, (unsigned long)"LogicalDevPoweron", 0, 0, 0);
                                    //
                                    for (size_t i = 0; i < mvPhySensorId.size(); i++)
                                    {
                                        MUINT const sensorIndex = mvPhySensorId[i];
                                        if (CC_UNLIKELY( mvHalSensor[i] == nullptr ))
                                        {
                                            return false;
                                        }
                                        if (CC_UNLIKELY( !mvHalSensor[i]->powerOn(mName.c_str(), 1, &sensorIndex) ))
                                        {
                                            #if (MTKCAM_HAVE_AEE_FEATURE == 1)
                                            aee_system_exception(
                                                LOG_TAG,
                                                NULL,
                                                DB_OPT_DEFAULT,
                                                android::String8::format(
                                                    "[%zu] sensorIndex:%d powerOn failed"
                                                    "\nCRDISPATCH_KEY:MtkCam/Cam1Device:Sensor power on failed"
                                                    , i, sensorIndex).c_str()
                                            );
                                            #endif
                                            return false;
                                        }
                                    }
                                    //
                                    return true;
                                }
                            );
                        // init 3A and poweron 3A
                        ...
                        return  ( success_init3A && success_sensorPowerOn );
                    }

HalSensorList.openList.cpp

由刚刚的pHalSensorList->createSensor函数可知，调用了HalSensorList::createSensor()==>openSensor，直接看openSensor函数，其中有一行pHalSensor = new HalSensor();

HalSensor.cpp

由上面步骤可知，mHalDeviceAdapter->open()，最终调用了HalSensor的构造函数，mHalDeviceAdapter->powerOn()最终调用了HalSensor::powerOn()函数。
先看HalSensor的构造函数，它继承于IHalSensor，这定义在IHalSensor.h中。IHalSensor.h
里面包括了所有sensor的信息（包括sensor id，name，address等等）。

imgsensor_drv.cpp

上个步骤中HalSensor::powerOn会循环调用三次SenDrv->open，也就是ImgSensorDrv::open，最终调用到ImgSensorDrv::featureControl

MINT32
ImgSensorDrv::open(IMGSENSOR_SENSOR_IDX sensorIdx)
{
    MUINT32                           featureParaLen = sizeof(MUINT32);
    MUINT32                           featurePara;

    return featureControl(sensorIdx, SENSOR_FEATURE_OPEN, (MUINT8 *)&featurePara, &featureParaLen);
}

看featureControl()函数

MINT32
ImgSensorDrv::featureControl(
    IMGSENSOR_SENSOR_IDX sensorIdx,
    ACDK_SENSOR_FEATURE_ENUM FeatureId,
    MUINT8 *pFeaturePara,
    MUINT32 *pFeatureParaLen
)
{
    ACDK_SENSOR_FEATURECONTROL_STRUCT featureCtrl;

    if (m_fdSensor == -1) {
        LOG_ERR("[sendCommand]m_fdSensor fail, sendCommand must be called after init()!");
        return SENSOR_UNKNOWN_ERROR;
    }

    if (pFeaturePara == NULL || pFeatureParaLen == NULL) {
        return SENSOR_INVALID_PARA;
    }

    featureCtrl.InvokeCamera = sensorIdx;
    featureCtrl.FeatureId = FeatureId;
    featureCtrl.pFeaturePara = pFeaturePara;
    featureCtrl.pFeatureParaLen = pFeatureParaLen;

    if (ioctl(m_fdSensor, KDIMGSENSORIOC_X_FEATURECONCTROL , &featureCtrl) < 0) {
        LOG_ERR("[featureControl] Err-ctrlCode (%s)", strerror(errno));
        return -errno;
    }
	...
    return SENSOR_NO_ERROR;
}//halSensorFeatureControl

最终调用ioctl函数进入kernel层！

总结

至此整个open Camera的流程已经通了，虽然中间忽略了非常多细节(比如很多callback，创建Session等等等等)，但是这篇文章只讨论open流程，要抓住事情的主要矛盾！要是同时跟很多条线路，大脑内存会溢出的！

整个open流程涉及了14个文件，从framework到HAL的文件都有，如何从一个层到另外一个层的过程很容易弄错，可以在代码中加一些自己的log来防止弄混淆走错线，也可以关注我的微信公众号，里面会有Camera相关的学习记录，代码小技巧和技能分享。