Android MTK三方算法集成学习

MTK官方资料/优秀学习博客分享

MTK HAL算法集成之单帧算法
MTK HAL算法集成之双摄算法
3rd Party ALGO Integrate Capture 3rd ALGO

HAL三方算法集成

1.三方算法类型

      超级夜景首先是集成在拍照上的即需要挂载到CaptureNode，其次类似超夜，HDR都需要多帧EV- EV0 EV+多帧处理，因此还需要挂载到MultiFrame中，以下可以仿照FID_AIHDR进行添加
vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp

    case FID_AIHDR:                 return "aihdr";
+   case FID_SUPERNIGHT:            return "supernight";

vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/MultiFrameNode.cpp

    else if (rProperty.mFeatures & MTK_FEATURE_AIHDR)
        featId = FID_AIHDR;
+   else if (rProperty.mFeatures & TP_FEATURE_SUPERNIGHT)
+       featId = FID_SUPERNIGHT;

vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h

    TP_FEATURE_ZOOM_FUSION  = 1ULL << 47,   // used by zoom scenario
+   TP_FEATURE_SUPERNIGHT   = 1ULL << 48,
     // TODO: reserve for customer feature index (bit 32-63)

vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h

    FID_AIHDR,
+   FID_SUPERNIGHT,

vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp

    if (feature & MTK_FEATURE_AIHDR)
        pCapRequest->addFeature(FID_AIHDR);
+   if (feature & TP_FEATURE_SUPERNIGHT)
+        pCapRequest->addFeature(FID_SUPERNIGHT);

2.三方算法挂载到对应的SCENARIO

      get_capture_scenario根据上层的提示或者AppMeta信息，决定senario，超夜原生就有对应的senario为CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT，可通过pParsedAppConfiguration->isSuperNightMode去控制这个senario走不走，默认是
vendor/mediatek/proprietary/custom/mt6853/hal/camera/camera_custom_feature_table.cpp

+   #define MTK_FEATURE_COMBINATION_SUPERNIGHT     (TP_FEATURE_SUPERNIGHT   |MTK_FEATURE_NR| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE)

+   CAMERA_SCENARIO_START(CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT)//
+   ADD_CAMERA_FEATURE_SET(TP_FEATURE_SUPERNIGHT,    MTK_FEATURE_COMBINATION_SUPERNIGHT)
+   ADD_CAMERA_FEATURE_SET(MTK_FEATURE_MFNR,     MTK_FEATURE_COMBINATION_MFNR)
+   ADD_CAMERA_FEATURE_SET(NO_FEATURE_NORMAL,    MTK_FEATURE_COMBINATION_SINGLE)
+   CAMERA_SCENARIO_END

      以下可通过log去确认超夜挂载的feature和scenario是否正确

mtkcam-customer_scenario_mgr: [customer_get_capture_scenario] scenarioHint(isCShot:0, isDualCam:0, dualDevicePath:0, multiCamFeatureMode:-1, isLogical:1 captureScenarioIndex:-1)
mtkcam-customer_scenario_mgr: [customer_get_capture_scenario] capture OpMode:0x10600
mtkcam-customer_scenario_mgr: [customer_get_capture_scenario] scenario:103
mtkcam-customer_scenario_mgr: [customer_get_features_table_by_scenario] scenario:103, table size:1
mtkcam-customer_scenario_mgr: [customer_get_features_table_by_scenario] find features for scenario(103 : CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT) in logical map
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] find scenario:CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT for (openId:0, scenario:103)
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] scenario(CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT) support feature:TP_FEATURE_SUPERNIGHT(0x2000000000000), feature combination:MTK_FEATURE_COMBINATION_SUPERNIGHT(0x20000000300d0)
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] scenario(CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT) support feature:MTK_FEATURE_MFNR(0x1), feature combination:MTK_FEATURE_COMBINATION_MFNR(0x300f1)
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] scenario(CUSTOMER_CAMERA_SCENARIO_CAPTURE_SUPER_NIGHT) support feature:NO_FEATURE_NORMAL(0), feature combination:MTK_FEATURE_COMBINATION_SINGLE(0x300f0)
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] support features:0x20000000300f1
mtkcam-FeatureSettingPolicy: [evaluateCaptureConfiguration] (0xb4000072dbf6dc70) uniqueKey:82405207

3.三方算法库，build动态库

      新建一个floder用于存放三方算法的sdk库，包括cfg，头文件，允许库等方便三方算法接口执行函数的调用
vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/supernight_sdk/Android.mk

include $(CLEAR_VARS)
LOCAL_MODULE := libSupperLongExposure
LOCAL_SRC_FILES_64 := arm64-v8a/libSupperLongExposure.so
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_SUFFIX := .so
LOCAL_SHARED_LIBRARIES := liblog libcutils libmtk_drvb libutils
LOCAL_PROPRIETARY_MODULE := true
LOCAL_MODULE_OWNER := mtk
LOCAL_MULTILIB := 64
LOCAL_CHECK_ELF_FILES := false
include $(BUILD_PREBUILT)

4.三方算法接口，build静态库

      新建一个floder用于存放三方算法的接口PipelinePlugin API
vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/supernight/Android.mk

LOCAL_MODULE := libmtkcam.plugin.supernight
LOCAL_PROPRIETARY_MODULE := true
LOCAL_MODULE_OWNER := mtk
LOCAL_MODULE_TAGS := optional
include $(MTK_STATIC_LIBRARY)

      超夜三方算法Provider继承自MultiFramePLugin:IProvider，API接口主要有property negotiate init process abort uninit，每种类型的PipelinePlugin，Property成员的定义都不一样，具体说明可以参考DCC文档“MTK_camera_plugin_interface_user_guide”中4.4 CPI PipelinePlugin API的内容，其中一定需要注意填写的是mName与mFeature字段，在negotiate函数中配置算法需要的输入、输出图像的格式、尺寸，raw算法则输入输出对应eImgFmt_BAYER10_UNPAK。

prop.mName = "THIRD_PARTY_SuperNight";
prop.mFeature = TP_FEATURE_SUPERNIGHT;

      目前超夜算法的实现，已知有两种
            方案一:Exp time + ISO (AE Lock + AE mode off (AE mode = 0))
            方案二 EV compensation(AE lock)

5.三方算法链接

vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk

LOCAL_C_INCLUDES +=$(TOP)/$(MTK_PATH_SOURCE)/hardware/mtkcam3/3rdparty/supernight_sdk/inc
LOCAL_WHOLE_STATIC_LIBRARIES += libmtkcam.plugin.supernight
LOCAL_SHARED_LIBRARIES += libSupperLongExposure

三方算法参数区分

1.解析App下发OpMode的区分是否进去超夜模式

vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/model/session/PipelineModelSessionFactory.cpp

if(pParsedAppConfiguration->operationMode == 自定义){
    pParsedAppConfiguration->isSuperNightMode = true;
    property_set("vendor.debug.camera.supernightmode","1");
}else{
    pParsedAppConfiguration->isSuperNightMode = false;
    property_set("vendor.debug.camera.supernightmode","0");   
}

      或者通过App下发VendorTag的方式控制，tryGetMetadata(pAppMeta, CONTROL_SUPERNIGHT_MODE, superNight);

2.添加一个新的EApp用于超夜参数区分

vendor/mediatek/proprietary/custom/mt6853/hal/inc/tuning_mapping/cam_idx_struct_ext.h

#define Comm_EApp_SuperNight 14

typedef enum
{
    EApp_MTKCam = Comm_EApp_MTKCam,
    EApp_Facebook = Comm_EApp_Facebook,
    ...
    EApp_3rd_party = Comm_EApp_3rd_party,
    EApp_END2ENDHDR = Comm_EApp_END2ENDHDR,
+   EApp_SuperNight = Comm_EApp_SuperNight,
    EApp_NUM,
} EApp_T;

3.通过map_AppName_to_MappingInfo区分找到对应模式的EApp

3.1 不传参AppName，直接控制判断，返回对应的模式的EApp

vendor/mediatek/proprietary/custom/mt6853/hal/camera_isp/ver1/isp_tuning_custom.cpp

MINT32
IspTuningCustom::
map_AppName_to_MappingInfo(const char* pAppname, const MINT32 feature){
	
+   int supernightmode = 0;
+   supernightmode = property_get_int32("vendor.debug.camera.supernightmode", 0);
+   MY_LOG("[%s] supernightmode:%d \n", __FUNCTION__, supernightmode);

    if (strstr(AppName, "Facebook")){
        return EApp_Facebook;
    ...
    else{
        if (strstr(pAppname, "MTKCam") && (feature & MTK_HDR_FEATURE_HDR_HAL_MODE_MSTREAM_CAPTURE_PREVIEW) ){
            return EApp_HDR;
        }
        //+++++++++++ For customization, please remove this section.
 +      else{
 +          if (supernightmode){
 +              return EApp_SuperNight;
 +          }
            else
                return EApp_MTKCam;
        }
    }

3.2 通过ATMs机制的中的setFactor设置对应模式的EApp

vendor/mediatek/proprietary/custom/mt6853/hal/camera_isp/ver1/isp_tuning_custom.cpp

MINT32
IspTuningCustom::
map_AppName_to_MappingInfo(const char* pAppname, const MINT32 feature){
	
    if (strstr(AppName, "Facebook")){
        return EApp_Facebook;
+   if (strstr(AppName, "SuperNight")){
+       return EApp_SuperNight;
    ...
    

      这个是之前项目上遇到过的别人使用的方案NSCam::ATMsEventMgr::getInstance()->setFactor(“APP_NAME”, “SuperNight”);其中在解问题的时候发现有参数没有区分开，类似log如下，通道会跳转到Scene_Preview

MtkCam/MappingMgr: [query] [Caller: getISPIndex] [Dev:1-Mod:DCE(8)] (Idx 5) (Night_Vision_Capture) (ISO_GROUP 2)
MtkCam/MappingMgr: [query] [Caller: getISPIndex] [Dev:1-Mod:DCE(8)] (Idx 5) (Night_Vision_Capture) (PF Capture_DCE, SM Preview, Bin 0, P2 0, FLASH 0, APP Night_Vision, FD 0, ZOOM 0, LV 3, CT 9, ISO 8, ISO_GROUP 2) (Base_Idx 5)
MtkCam/MappingMgr: [query] [Caller: TaskRequest] [Dev:1-Mod:LTM(36)] (Idx 2) (Scene_Preview) (ISO_GROUP 2)
MtkCam/MappingMgr: [query] [Caller: TaskRequest] [Dev:1-Mod:LTM(36)] (Idx 2) (Scene_Preview) (PF Preview, SM Preview, Bin 0, P2 0, FLASH 0, APP MTKCam, FD 0, ZOOM 0, LV 0, CT 0, ISO 0, ISO_GROUP 2) (Base_Idx 2)

      正常按参数配置表中的ScenarioMap应该是，所以需要修改Flow控制
      IspProfile                  SensorMode            AppName            Scenario
      A_Normal_Capture   Preview                  Night_Vision        Night_Vision_Capture

vendor/mediatek/proprietary/hardware/mtkcam/aaa/source/isp_6s/wrapper/Hal3ARawImp.cpp

MVOID
Hal3ARawImp::
TaskRequest(Request_Service service_id, MVOID* input, MVOID* output)
{
    switch (service_id) {
        case REQ_LTM_INPUT:
        {
            CAM_IDX_QRY_COMB rMapping_InfoLTM = {};
            rMapping_InfoLTM.eIspProfile = m_rParam.eIspProfile;
            rMapping_InfoLTM.eSensorMode = static_cast<NSIspTuning::ESensorMode_T>(m_u4SensorMode);
            
+           IspTuningCustom* pIspTuningCustom = IspTuningCustom::createInstance((ESensorDev_T)m_i4SensorDev, m_i4SensorIdx, m_i4SensorIdx);
+           //MBOOL ret = NSCam::ATMsEventMgr::getInstance()->getFactor("APP_NAME", str_factor_value);
+           EApp_T NV = EApp_MTKCam;

+           if (pIspTuningCustom)
+           {
+                NV = (EApp_T)pIspTuningCustom->map_AppName_to_MappingInfo(str_factor_value.c_str(), -1);
+           }else
+           {
+                CAM_LOGD("[%s] pIspTuningCustom is NULL", __FUNCTION__);
+           }

+           if(NV == EApp_Night_Vision && m_i4SensorDev == 1 ){
+                 rMapping_InfoLTM.eApp = EApp_Night_Vision;
+                 rMapping_InfoLTM.eIspProfile = EIspProfile_Capture;
+                 m_rParam.eIspProfile = EIspProfile_Capture;
            }