和你一起终身学习,这里是程序员Android
经典好文推荐,通过阅读本文,您将收获以下知识点:
算法概览
一、算法集成前的准备
二、 为算法选择feature
三、 将算法对应的feature添加到scenario配置表
四、挂载算法
五、自定义metadata
六、APP调用算法
七、遇到的问题及解决方法
八、结语
算法概览
为了给用户提供更好的成像效果,现在的手机都会接入一些第三方的图像处理算法。MTK平台的HAL3也在P2这一层提供接入的plugin。按图像处理算法需要的帧数和摄像头数量,大体可以分为三类:
单帧算法:
常见的单帧算法有:美颜算法(瘦脸、磨皮、大眼)、广角镜头畸变校正算法、附加表情算法、单摄背景虚化算法(伪双摄算法)等等,仅需单帧图像输入的算法都属于单帧算法。一般情况下,输入一帧图像,算法处理完输出一帧处理后的图像。多帧算法:
常见的多帧算法有:MFNR(多帧降噪)、HDR(高动态范围)等等,需要连续多帧图像输入的算法都属于多帧算法。一般情况下,输入连续多帧图像,算法处理完输出一帧处理后的图像。双摄算法:
最常见的双摄算法是双摄景深算法或者叫双摄背景虚化算法,除此之外,也有彩色+黑白用于增强夜拍效果的双摄算法。单帧算法和多帧算法仅需要获取一个摄像头的图像。而双摄算法需要获取主、辅两个摄像头的图像,并且一般还会要求主、辅摄像头同步。分别获取主、辅摄像头的两帧同步图像,处理后输出一帧主摄图像,用户也仅能看到主摄图像。
根据这个大体上的分类,MTK HAL算法集成系列文章共三篇:
MTK HAL算法集成之单帧算法
MTK HAL算法集成之多帧算法
MTK HAL算法集成之双摄算法
本文是其中的第一篇。这个系列文章均基于Android 9.0,MT6763平台,HAL版本是HAL3。
一、算法集成前的准备
在开展集成工作之前,首先要对算法有一个基本的评估,并且对于集成也应有一定的要求。
1. 1 算法要求及评估
处理效果好,不能比竞品差,超过竞品更佳。(这条和camera调试的主观效果一样,主观性较强,往往一厢情愿,具体看项目要求吧)
各个场景及压力测试下效果稳定。
处理后照片无色差、锐度和饱和度无损失,或者损失在可接受范围。
达到可接受的分辨率,最好可达到摄像头的最大分辨率。
处理时间越快越好,不超过竞品时间、不超过项目和产品的目标时间。
无内存泄露,占用内存少。
提供必要的集成说明文档,包括算法类型、输入及输出图像要求、输入参数要求等等。
注意:如果有条件,处理时间、内存占用、分辨率等等可量化的指标可要求算法提供方给出具体的参考数据,以便集成完后测试验证。
1.2 算法集成要求
编译时可根据项目控制是否集成算法。
运行时可以用参数控制是否启用算法。
集成算法库正常运行、压力测试下效果稳定、无内存泄露。
1.3 算法集成的步骤
(1). 根据算法选择feature类型,如果与MTK提供的feature不能对号入座,则需要添加自定义feature。
(2). 将算法对应的feature类型添加到scenario配置表。
(3). 根据算法选择plugin类型,编写CPP文件实现plugin,挂载算法。
(4). 如果算法不能复用Android和MTK提供的metadata,则还需要为算法配置自定义的metadata以便APP控制是否启用算法。
首先,我准备了一个libwatermark.so,它仅仅实现了一个添加水印的功能,用它来模拟第三方的单帧算法库。如果想了解添加水印的实现代码,可以参考我另外一篇文章:Android 实现图片加水印或logo。接下来,我们就按照集成步骤,逐步详细讲解。
二、 为算法选择feature
2.1 MTK提供的feature
MTK在mtk_feature_type.h和customer_feature_type.h已经提供了一些feature。
vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/mtk/mtk_feature_type.h:
NO_FEATURE_NORMAL = 0ULL,
// MTK (bit 0-31)
MTK_FEATURE_MFNR = 1ULL << 0,
MTK_FEATURE_HDR = 1ULL << 1,
MTK_FEATURE_REMOSAIC = 1ULL << 2,
MTK_FEATURE_ABF = 1ULL << 3,
MTK_FEATURE_NR = 1ULL << 4,
MTK_FEATURE_FB = 1ULL << 5,
MTK_FEATURE_CZ = 1ULL << 6,
MTK_FEATURE_DRE = 1ULL << 7,
MTK_FEATURE_DEPTH = 1ULL << 8,
MTK_FEATURE_BOKEH = 1ULL << 9,
MTK_FEATURE_VSDOF = (MTK_FEATURE_DEPTH|MTK_FEATURE_BOKEH),
MTK_FEATURE_FSC = 1ULL << 10,
MTK_FEATURE_3DNR = 1ULL << 11,
MTK_FEATURE_EIS = 1ULL << 12,
MTK_FEATURE_AINR = 1ULL << 13,
MTK_FEATURE_DUAL_YUV = 1ULL << 14,
MTK_FEATURE_DUAL_HWDEPTH = 1ULL << 15,
MTK_FEATURE_AIS = 1ULL << 16,
MTK_FEATURE_HFG = 1ULL << 17,
MTK_FEATURE_DCE = 1ULL << 18,
vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h:
// ThirdParty (bit 32-63)
TP_FEATURE_HDR = 1ULL << 32,
TP_FEATURE_MFNR = 1ULL << 33,
TP_FEATURE_EIS = 1ULL << 34,
TP_FEATURE_FB = 1ULL << 35,
TP_FEATURE_FILTER = 1ULL << 36,
TP_FEATURE_DEPTH = 1ULL << 37,
TP_FEATURE_BOKEH = 1ULL << 38,
TP_FEATURE_VSDOF = (TP_FEATURE_DEPTH|TP_FEATURE_BOKEH),
TP_FEATURE_FUSION = 1ULL << 39,
TP_FEATURE_HDR_DC = 1ULL << 40, // used by DualCam
TP_FEATURE_DUAL_YUV = 1ULL << 41,
TP_FEATURE_DUAL_HWDEPTH = 1ULL << 42,
TP_FEATURE_PUREBOKEH = 1ULL << 43,
TP_FEATURE_RAW_HDR = 1ULL << 44,
TP_FEATURE_RELIGHTING = 1ULL << 45,
MTK提供的这些feature可以满足绝大多数算法的集成,在可以对号入座的情况下,我们直接使用已有feature即可。如果不能够满足我们的要求,可以参考下节内容添加新的feature。
2.2 添加自定义feature
本来单帧算法对应的feature可以选择MTK提供的MTK_FEATURE_FB和TP_FEATURE_FB,但是为了讲解如何添加新feature,我们选择添加一个自定义feature:TP_FEATURE_WATERMARK。
vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
old mode 100644
new mode 100755
index a41fd864f5..17bc35eea8
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
@@ -59,6 +59,7 @@ enum eFeatureIndexCustomer {
TP_FEATURE_PUREBOKEH = 1ULL << 43,
TP_FEATURE_RAW_HDR = 1ULL << 44,
TP_FEATURE_RELIGHTING = 1ULL << 45,
+ TP_FEATURE_WATERMARK = 1ULL << 46,
// TODO: reserve for customer feature index (bit 32-63)
};
vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
old mode 100644
new mode 100755
index e32f80a609..47273b01c7
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
@@ -599,6 +599,7 @@ const char* FeatID2Name(FeatureID_T fid)
case FID_FUSION_3RD_PARTY: return "fusion_3rd_party";
case FID_PUREBOKEH_3RD_PARTY: return "purebokeh_3rd_party";
case FID_RELIGHTING_3RD_PARTY: return "relighting_3rd_party";
+ case FID_WATERMARK_3RD_PARTY: return "watermark_3rd_party";
default: return "unknown";
};
vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
index 8bb794ba02..d4343aaccf 100755
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
@@ -779,7 +779,8 @@ MBOOL YUVNode::onInit()
featId = FID_FB_3RD_PARTY;
else if (rProperty.mFeatures & TP_FEATURE_RELIGHTING)
featId = FID_RELIGHTING_3RD_PARTY;
-
+ else if (rProperty.mFeatures & TP_FEATURE_WATERMARK)
+ featId = FID_WATERMARK_3RD_PARTY;
if (featId != NULL_FEATURE) {
MY_LOGD_IF(mLogLevel, "%s finds plugin:%s, priority:%d",
vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
old mode 100644
new mode 100755
index 2f1ad8a665..ab47aae456
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
@@ -172,6 +172,7 @@ enum CaptureFeatureFeatureID {
FID_FUSION_3RD_PARTY,
FID_PUREBOKEH_3RD_PARTY,
FID_RELIGHTING_3RD_PARTY,
+ FID_WATERMARK_3RD_PARTY,
NUM_OF_FEATURE,
NULL_FEATURE = 0xFF,
};
vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
old mode 100644
new mode 100755
index cc1dc549fd..00559cbc30
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
@@ -428,6 +428,9 @@ MBOOL CaptureProcessor::onEnque(const sp<P2FrameRequest> &pP2Frame)
pCapRequest->addFeature(FID_HFG);
if (feature & MTK_FEATURE_DCE)
pCapRequest->addFeature(FID_DCE);
+ if (feature & TP_FEATURE_WATERMARK)
+ pCapRequest->addFeature(FID_WATERMARK_3RD_PARTY);
+
}
}
三、 将算法对应的feature添加到scenario配置表
在我们打开camera进行预览和拍照的时候,MTK HAL3会执行vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/policy/FeatureSettingPolicy.cpp的代码,会分别调用
vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/scenario_mgr.cpp的
get_streaming_scenario函数和get_capture_scenario函数。它们会读取一个scenario的feature配置表,遍历所有的feature,决定哪些feature会被执行。这个配置表中有许多的scenario,一个scenario可能对应多个feature。因此添加自定义feature后,还需将自定义的feature添加到配置表中。MTK feature 对应的配置表是 gMtkScenarioFeaturesMaps,customer feature 对应的配置表是 gCustomerScenarioFeaturesMaps。
vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp:
diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
old mode 100644
new mode 100755
index f8d081e433..577f85797e
--- a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
+++ b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
@@ -93,30 +93,30 @@ using namespace NSCam::v3::pipeline::policy::scenariomgr;
// #define <feature combination> (key feature | post-processing features | ...)
//
// single cam capture feature combination
-#define TP_FEATURE_COMBINATION_SINGLE (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_HDR (TP_FEATURE_HDR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_AINR (MTK_FEATURE_AINR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_MFNR (MTK_FEATURE_MFNR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_REMOSAIC (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
+#define TP_FEATURE_COMBINATION_SINGLE (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_HDR (TP_FEATURE_HDR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_AINR (MTK_FEATURE_AINR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_MFNR (MTK_FEATURE_MFNR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_REMOSAIC (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
#define TP_FEATURE_COMBINATION_CSHOT (NO_FEATURE_NORMAL | MTK_FEATURE_CZ| MTK_FEATURE_HFG)
-#define TP_FEATURE_COMBINATION_YUV_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_RAW_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB)
+#define TP_FEATURE_COMBINATION_YUV_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_RAW_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
#define TP_FEATURE_COMBINATION_PRO (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE)
-#define TP_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB)
+#define TP_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS (NO_FEATURE_NORMAL | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
// dual cam capture feature combination
// the VSDOF means the combination of Bokeh feature and Depth feature
-#define TP_FEATURE_COMBINATION_TP_VSDOF (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
-#define TP_FEATURE_COMBINATION_TP_VSDOF_HDR (TP_FEATURE_HDR_DC | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
-#define TP_FEATURE_COMBINATION_TP_VSDOF_MFNR (MTK_FEATURE_MFNR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
-#define TP_FEATURE_COMBINATION_TP_FUSION (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_FUSION)
-#define TP_FEATURE_COMBINATION_TP_PUREBOKEH (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_PUREBOKEH)
+#define TP_FEATURE_COMBINATION_TP_VSDOF (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_TP_VSDOF_HDR (TP_FEATURE_HDR_DC | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_TP_VSDOF_MFNR (MTK_FEATURE_MFNR | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_TP_FUSION (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_FUSION| TP_FEATURE_WATERMARK)
+#define TP_FEATURE_COMBINATION_TP_PUREBOKEH (NO_FEATURE_NORMAL | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_PUREBOKEH| TP_FEATURE_WATERMARK)
// streaming feature combination (TODO: it should be refined by streaming scenario feature)
-#define TP_FEATURE_COMBINATION_VIDEO_NORMAL (MTK_FEATURE_FB|TP_FEATURE_FB)
-#define TP_FEATURE_COMBINATION_VIDEO_DUAL_YUV (MTK_FEATURE_FB|MTK_FEATURE_DUAL_YUV|TP_FEATURE_FB|TP_FEATURE_DUAL_YUV)
-#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWDEPTH (MTK_FEATURE_FB|MTK_FEATURE_DUAL_HWDEPTH|TP_FEATURE_FB|TP_FEATURE_DUAL_HWDEPTH)
-#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWVSDOF (MTK_FEATURE_FB|TP_FEATURE_FB)
+#define TP_FEATURE_COMBINATION_VIDEO_NORMAL