Android Camera原理之camera service与camera provider session会话与capture request轮转

上层调用CameraManager.openCamera的时候，会触发底层的一系列反应，之前我们分享过camera framework到camera service之间的调用，但是光看这一块还不够深入，接下来我们讨论一下camera service与camera provider之间在openCamera调用的时候做了什么事情。

status_t Camera3Device::initialize(sp<CameraProviderManager> manager, const String8& monitorTags) {
    sp<ICameraDeviceSession> session;
    status_t res = manager->openSession(mId.string(), this,
            /*out*/ &session);
    //......
    res = manager->getCameraCharacteristics(mId.string(), &mDeviceInfo);
    //......
    std::shared_ptr<RequestMetadataQueue> queue;
    auto requestQueueRet = session->getCaptureRequestMetadataQueue(
        [&queue](const auto& descriptor) {
            queue = std::make_shared<RequestMetadataQueue>(descriptor);
            if (!queue->isValid() || queue->availableToWrite() <= 0) {
                ALOGE("HAL returns empty request metadata fmq, not use it");
                queue = nullptr;
                // don't use the queue onwards.
            }
        });
    //......
    std::unique_ptr<ResultMetadataQueue>& resQueue = mResultMetadataQueue;
    auto resultQueueRet = session->getCaptureResultMetadataQueue(
        [&resQueue](const auto& descriptor) {
            resQueue = std::make_unique<ResultMetadataQueue>(descriptor);
            if (!resQueue->isValid() || resQueue->availableToWrite() <= 0) {
                ALOGE("HAL returns empty result metadata fmq, not use it");
                resQueue = nullptr;
                // Don't use the resQueue onwards.
            }
        });
    //......
 
    mInterface = new HalInterface(session, queue);
    std::string providerType;
    mVendorTagId = manager->getProviderTagIdLocked(mId.string());
    mTagMonitor.initialize(mVendorTagId);
    if (!monitorTags.isEmpty()) {
        mTagMonitor.parseTagsToMonitor(String8(monitorTags));
    }
 
    return initializeCommonLocked();
}

上面camera service中执行openCamera 中的核心步骤，可以看出，第一步执行的就是 manager->openSession(mId.string(), this, /out/ &session);

本文就是通过剖析openSession的执行流程来 还原 camera service 与camera provider 的执行过程。

CameraProviderManager--openSession流程.jpg

 

为了让大家看得更加清楚，列出各个文件的位置：
CameraProviderManager : frameworks/av/services/camera/libcameraservice/common/CameraServiceProvider.cpp

CameraDevice : hardware/interfaces/camera/device/3.2/default/CameraDevice.cpp

CameraModule : hardware/interfaces/camera/common/1.0/default/CameraModule.cpp

CameraHAL : hardware/libhardware/modules/camera/3_0/CameraHAL.cpp

Camera : hardware/libhardware/modules/camera/3_0/Camera.cpp

CameraDeviceSession : hardware/interfaces/camera/device/3.2/default/CameraDeviceSession.cpp

中间涉及到一些指针函数的映射，如果看不明白，可以参考：《Android Camera原理之底层数据结构总结》，具体的调用流程就不说了，按照上面的时序图走，都能看明白的。

一些回调关系还是值得说下的。我们看下CameraProviderManager::openSession调用的地方：

status_t CameraProviderManager::openSession(const std::string &id,
        const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback,
        /*out*/
        sp<hardware::camera::device::V3_2::ICameraDeviceSession> *session) {
 
    std::lock_guard<std::mutex> lock(mInterfaceMutex);
 
    auto deviceInfo = findDeviceInfoLocked(id,
            /*minVersion*/ {3,0}, /*maxVersion*/ {4,0});
    if (deviceInfo == nullptr) return NAME_NOT_FOUND;
 
    auto *deviceInfo3 = static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo);
 
    Status status;
    hardware::Return<void> ret;
    ret = deviceInfo3->mInterface->open(callback, [&status, &session]
            (Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });
    if (!ret.isOk()) {
        ALOGE("%s: Transaction error opening a session for camera device %s: %s",
                __FUNCTION__, id.c_str(), ret.description().c_str());
        return DEAD_OBJECT;
    }
    return mapToStatusT(status);
}

我们看下IPC调用的地方：

ret = deviceInfo3->mInterface->open(callback, [&status, &session]
            (Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });

传入两个参数，一个是 const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback， 另一个是open_cb _hidl_cb

callback 提供了 camera HAL层到 camera service的回调。

open_cb _hidl_cb 是硬件抽象层提供了一种IPC间回传数据的方式。就本段代码而言，需要传回两个数据，一个status：表示当前openSession是否成功；另一个是session：表示camera session会话创建成功之后返回的session数据。

CameraDevice::open(...)函数

{
        session = createSession(
                device, info.static_camera_characteristics, callback);
        if (session == nullptr) {
            ALOGE("%s: camera device session allocation failed", __FUNCTION__);
            mLock.unlock();
            _hidl_cb(Status::INTERNAL_ERROR, nullptr);
            return Void();
        }
        if (session->isInitFailed()) {
            ALOGE("%s: camera device session init failed", __FUNCTION__);
            session = nullptr;
            mLock.unlock();
            _hidl_cb(Status::INTERNAL_ERROR, nullptr);
            return Void();
        }
        mSession = session;
 
        IF_ALOGV() {
            session->getInterface()->interfaceChain([](
                ::android::hardware::hidl_vec<::android::hardware::hidl_string> interfaceChain) {
                    ALOGV("Session interface chain:");
                    for (auto iface : interfaceChain) {
                        ALOGV("  %s", iface.c_str());
                    }
                });
        }
        mLock.unlock();
}
_hidl_cb(status, session->getInterface());

最后执行的代码 _hidl_cb(status, session→getInterface()); 当前session创建成功之后，回调到 camera service 中。

const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback 设置到什么地方？这个问题非常重要的，camera 上层很依赖底层的回调，所以我们要搞清楚底层的回调被设置到什么地方，然后在搞清楚在合适的时机触发这些回调。

执行CameraDeviceSession构造函数的时候，传入了这个callback。

CameraDeviceSession::CameraDeviceSession(
    camera3_device_t* device,
    const camera_metadata_t* deviceInfo,
    const sp<ICameraDeviceCallback>& callback) :
        camera3_callback_ops({&sProcessCaptureResult, &sNotify}),
        mDevice(device),
        mDeviceVersion(device->common.version),
        mIsAELockAvailable(false),
        mDerivePostRawSensKey(false),
        mNumPartialResults(1),
        mResultBatcher(callback) {
    mDeviceInfo = deviceInfo;
    camera_metadata_entry partialResultsCount =
            mDeviceInfo.find(ANDROID_REQUEST_PARTIAL_RESULT_COUNT);
    if (partialResultsCount.count > 0) {
        mNumPartialResults = partialResultsCount.data.i32[0];
    }
    mResultBatcher.setNumPartialResults(mNumPartialResults);
 
    camera_metadata_entry aeLockAvailableEntry = mDeviceInfo.find(
            ANDROID_CONTROL_AE_LOCK_AVAILABLE);
    if (aeLockAvailableEntry.count > 0) {
        mIsAELockAvailable = (aeLockAvailableEntry.data.u8[0] ==
                ANDROID_CONTROL_AE_LOCK_AVAILABLE_TRUE);
    }
 
    // Determine whether we need to derive sensitivity boost values for older devices.
    // If post-RAW sensitivity boost range is listed, so should post-raw sensitivity control
    // be listed (as the default value 100)
    if (mDeviceInfo.exists(ANDROID_CONTROL_POST_RAW_SENSITIVITY_BOOST_RANGE)) {
        mDerivePostRawSensKey = true;
    }
 
    mInitFail = initialize();
}

CameraDeviceSession 中的 mResultBatcher 类构造中传入了这个 callback，现在由 CameraDeviceSession::ResultBatcher 来持有 callback了。看下ResultBatcher全局代码，在CameraDeviceSession.h中。

那以后底层要回调到上层必定要经过 CameraDeviceSession::ResultBatcher的mCallback来完成了。

class ResultBatcher {
public:
    ResultBatcher(const sp<ICameraDeviceCallback>& callback);
    void setNumPartialResults(uint32_t n);
    void setBatchedStreams(const std::vector<int>& streamsToBatch);
    void setResultMetadataQueue(std::shared_ptr<ResultMetadataQueue> q);
 
    void registerBatch(uint32_t frameNumber, uint32_t batchSize);
    void notify(NotifyMsg& msg);
    void processCaptureResult(CaptureResult& result);
 
protected:
    struct InflightBatch {
        // Protect access to entire struct. Acquire this lock before read/write any data or
        // calling any methods. processCaptureResult and notify will compete for this lock
        // HIDL IPCs might be issued while the lock is held
        Mutex mLock;
 
        bool allDelivered() const;
 
        uint32_t mFirstFrame;
        uint32_t mLastFrame;
        uint32_t mBatchSize;
 
        bool mShutterDelivered = false;
        std::vector<NotifyMsg> mShutterMsgs;
 
        struct BufferBatch {
            BufferBatch(uint32_t batchSize) {
                mBuffers.reserve(batchSize);
            }
            bool mDelivered = false;
            // This currently assumes every batched request will output to the batched stream
            // and since HAL must always send buffers in order, no frameNumber tracking is
            // needed
            std::vector<StreamBuffer> mBuffers;
        };
        // Stream ID -> VideoBatch
        std::unordered_map<int, BufferBatch> mBatchBufs;
 
        struct MetadataBatch {
            //                   (frameNumber, metadata)
            std::vector<std::pair<uint32_t, CameraMetadata>> mMds;
        };
        // Partial result IDs that has been delivered to framework
        uint32_t mNumPartialResults;
        uint32_t mPartialResultProgress = 0;
        // partialResult -> MetadataBatch
        std::map<uint32_t, MetadataBatch> mResultMds;
 
        // Set to true when batch is removed from mInflightBatches
        // processCaptureResult and notify must check this flag after acquiring mLock to make
        // sure this batch isn't removed while waiting for mLock
        bool mRemoved = false;
    };
 
 
    // Get the batch index and pointer to InflightBatch (nullptrt if the frame is not batched)
    // Caller must acquire the InflightBatch::mLock before accessing the InflightBatch
    // It's possible that the InflightBatch is removed from mInflightBatches before the
    // InflightBatch::mLock is acquired (most likely caused by an error notification), so
    // caller must check InflightBatch::mRemoved flag after the lock is acquried.
    // This method will hold ResultBatcher::mLock briefly
    std::pair<int, std::shared_ptr<InflightBatch>> getBatch(uint32_t frameNumber);
 
    static const int NOT_BATCHED = -1;
 
    // move/push function avoids "hidl_handle& operator=(hidl_handle&)", which clones native
    // handle
    void moveStreamBuffer(StreamBuffer&& src, StreamBuffer& dst);
    void pushStreamBuffer(StreamBuffer&& src, std::vector<StreamBuffer>& dst);
 
    void sendBatchMetadataLocked(
            std::shared_ptr<InflightBatch> batch, uint32_t lastPartialResultIdx);
 
    // Check if the first batch in mInflightBatches is ready to be removed, and remove it if so
    // This method will hold ResultBatcher::mLock briefly
    void checkAndRemoveFirstBatch();
 
    // The following sendXXXX methods must be called while the InflightBatch::mLock is locked
    // HIDL IPC methods will be called during these methods.
    void sendBatchShutterCbsLocked(std::shared_ptr<InflightBatch> batch);
    // send buffers for all batched streams
    void sendBatchBuffersLocked(std::shared_ptr<InflightBatch> batch);
    // send buffers for specified streams
    void sendBatchBuffersLocked(
            std::shared_ptr<InflightBatch> batch, const std::vector<int>& streams);
   // End of sendXXXX methods
 
    // helper methods
    void freeReleaseFences(hidl_vec<CaptureResult>&);
    void notifySingleMsg(NotifyMsg& msg);
    void processOneCaptureResult(CaptureResult& result);
    void invokeProcessCaptureResultCallback(hidl_vec<CaptureResult> &results, bool tryWriteFmq);
 
    // Protect access to mInflightBatches, mNumPartialResults and mStreamsToBatch
    // processCaptureRequest, processCaptureResult, notify will compete for this lock
    // Do NOT issue HIDL IPCs while holding this lock (except when HAL reports error)
    mutable Mutex mLock;
    std::deque<std::shared_ptr<InflightBatch>> mInflightBatches;
    uint32_t mNumPartialResults;
    std::vector<int> mStreamsToBatch;
    const sp<ICameraDeviceCallback> mCallback;
    std::shared_ptr<ResultMetadataQueue> mResultMetadataQueue;
 
    // Protect against invokeProcessCaptureResultCallback()
    Mutex mProcessCaptureResultLock;
 
} mResultBatcher;

到这里，openSession工作就完成了，这个主要是设置了上层的回调到底层，并且底层返回可用的camera session到上层来，实现底层和上层的交互通信。

1.获取的session是什么？为什么这个重要？

此session是 ICameraDeviceSession 对象，这个对象是指为了操作camera device，camera provider与 camera service之间建立的一个会话机制，可以保证camera service IPC调用到camera provider进程中的代码。

1.1.获取session当前请求原数组队列

auto requestQueueRet = session->getCaptureRequestMetadataQueue(
    [&queue](const auto& descriptor) {
        queue = std::make_shared<RequestMetadataQueue>(descriptor);
        if (!queue->isValid() || queue->availableToWrite() <= 0) {
            ALOGE("HAL returns empty request metadata fmq, not use it");
            queue = nullptr;
            // don't use the queue onwards.
        }
    });

到HAL层的 CameraDeviceSession.cpp中调用 getCaptureRequestMetadataQueue

Return<void> CameraDeviceSession::getCaptureRequestMetadataQueue(
    ICameraDeviceSession::getCaptureRequestMetadataQueue_cb _hidl_cb) {
    _hidl_cb(*mRequestMetadataQueue->getDesc());
    return Void();
}

这个mRequestMetadataQueue 是在 CameraDeviceSession::initialize 执行的时候初始化的。

int32_t reqFMQSize = property_get_int32("ro.camera.req.fmq.size", /*default*/-1);
if (reqFMQSize < 0) {
    reqFMQSize = CAMERA_REQUEST_METADATA_QUEUE_SIZE;
} else {
    ALOGV("%s: request FMQ size overridden to %d", __FUNCTION__, reqFMQSize);
}
 
mRequestMetadataQueue = std::make_unique<RequestMetadataQueue>(
        static_cast<size_t>(reqFMQSize),
        false /* non blocking */);
if (!mRequestMetadataQueue->isValid()) {
    ALOGE("%s: invalid request fmq", __FUNCTION__);
    return true;
}

首先读取 ro.camera.req.fmq.size 属性，如果没有找到，则直接赋给一个 1M 大小的 请求原数组队列。这个队列很重要，后续的camera capture请求都是通过这个队列处理的。

1.2.获取session 当前结果原数组队列

这个和 请求原数组队列相似，不过结果原数组中保留的是 camera capture的结果数据。大家可以看下源码，这儿就不贴源码了

2.开始运转capture request线程

camera service 与 camera provider 建立session 会话之后，开始运转capture request请求线程，之后发送的capture request都会到这个线程中执行，这就是熟知的capture request轮转。

在Camera3Device::initializeCommonLocked 中执行了 capture request轮转。

/** Start up request queue thread */
mRequestThread = new RequestThread(this, mStatusTracker, mInterface, sessionParamKeys);
res = mRequestThread->run(String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
if (res != OK) {
    SET_ERR_L("Unable to start request queue thread: %s (%d)",
            strerror(-res), res);
    mInterface->close();
    mRequestThread.clear();
    return res;
}

开始启动当前的capture request 队列，放在 RequestThread线程中执行，这个线程会一直执行，当有新的capture request发过来，会将capture request放进当前会话的请求队列中，继续执行。这个轮转很重要，这是camera能正常工作的前提。

轮转的主要工作在Camera3Device::RequestThread::threadLoop 函数中完成，这是native中定义的一个 线程执行函数块。

bool Camera3Device::RequestThread::threadLoop() {
    ATRACE_CALL();
    status_t res;
 
    // Handle paused state.
    if (waitIfPaused()) {
        return true;
    }
 
    // Wait for the next batch of requests.
    waitForNextRequestBatch();
    if (mNextRequests.size() == 0) {
        return true;
    }
    //......
 
    // Prepare a batch of HAL requests and output buffers.
    res = prepareHalRequests();
    if (res == TIMED_OUT) {
        // Not a fatal error if getting output buffers time out.
        cleanUpFailedRequests(/*sendRequestError*/ true);
        // Check if any stream is abandoned.
        checkAndStopRepeatingRequest();
        return true;
    } else if (res != OK) {
        cleanUpFailedRequests(/*sendRequestError*/ false);
        return false;
    }
 
    // Inform waitUntilRequestProcessed thread of a new request ID
    {
        Mutex::Autolock al(mLatestRequestMutex);
 
        mLatestRequestId = latestRequestId;
        mLatestRequestSignal.signal();
    }
    //......
 
    bool submitRequestSuccess = false;
    nsecs_t tRequestStart = systemTime(SYSTEM_TIME_MONOTONIC);
    if (mInterface->supportBatchRequest()) {
        submitRequestSuccess = sendRequestsBatch();
    } else {
        submitRequestSuccess = sendRequestsOneByOne();
    }
    //......
 
    return submitRequestSuccess;
}

waitForNextRequestBatch() 不断去轮训底层是否有InputBuffer数据，获取的inputBuffer数据放在request中，这些数据会在之后被消费。
这儿先列个调用过程：对照着代码看一下，camera的producer与consumer模型之后还会详细讲解的。

camera request轮转流程.jpg

 

这儿也为了大家快速进入代码，也列出来代码的对应位置：

Camera3Device::RequestThread : frameworks/av/services/camera/libcameraservice/device3/Camera3Device.cpp 中有内部类 RequestThread，这是一个线程类。

Camera3Stream : frameworks/av/services/camera/libcameraservice/device3/Camera3Stream.cpp

Camera3InputStream : frameworks/av/services/camera/libcameraservice/device3/Camera3InputStream.cpp

Camera3IOStreamBase : frameworks/av/services/camera/libcameraservice/device3/Camera3IOStreamBase.cpp

BufferItemConsumer : frameworks/native/libs/gui/BufferItemConsumer.cpp

ConsumerBase : frameworks/native/libs/gui/ConsumerBase.cpp

BnGraphicBufferConsumer : frameworks/native/libs/gui/IGraphicBufferConsumer.cpp

上层发过来来的capture request，手下到底层申请Consumer buffer，这个buffer数据存储在capture request缓存中，后期这些buffer数据会被复用，不断地生产数据，也不断地被消费。

capture request开启之后，camera hal层也会受到capture request批处理请求，让camera hal做好准备，开始和camera driver层交互。hal层的请求下一章讲解。

 

小礼物走一走，来简书关注我