UE4 OpenCV+Dlib插件（支持Windows Android）人脸检测（一）

于 2020-05-26 12:08:31 发布
阅读量3.2k 收藏 16
点赞数 1
分类专栏： UE4学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/maxiaosheng521/article/details/106351234
版权

在这里插入图片描述

本篇是OpenCV插件的编写，使用

参考

https://github.com/Brandon-Wilson/OpenCV-Plugin
https://github.com/TianxingWu/OpenVHead
https://unreal.gg-labs.com/wiki-archives/ar-vr/integrating-opencv-into-unreal-engine-4
https://github.com/Batname/UE4OpenCV

相关配置
  • opencv 3.4
  • UE4.24
  • 在这里插入图片描述
  • Android端的话需要申请权限代码，这里要注意申请前需要先检查一下是否已经具备权限，否则会报错，为例省事，直接申请了存储的写入，读取还有摄像机权限
OpenCV.cpp
//Request Android Permission
	TArray<FString> AndroidTotalPermissions = {
		TEXT("android.permission.CAMERA"),
       TEXT("android.permission.READ_EXTERNAL_STORAGE"),
       TEXT("android.permission.WRITE_EXTERNAL_STORAGE"),
       TEXT("android.permission.MOUNT_UNMOUNT_FILESYSTEMS"),
       };
	TArray<FString> AndroidNeedReqPermissions;
	for(int i = 0; i < AndroidTotalPermissions.Num(); i++)
	{
		if(!UAndroidPermissionFunctionLibrary::CheckPermission(AndroidTotalPermissions[i]))
		{
			AndroidNeedReqPermissions.Add(AndroidTotalPermissions[i]);
		}	
	}
	UAndroidPermissionFunctionLibrary::AcquirePermissions(AndroidNeedReqPermissions);
源码

源码
有可能会有遗漏，以下所有内容最终以源码中的结果为准

插件的目录结构

在这里插入图片描述

OpenCV SDK准备

  1. UE4新建Blank插件或者ThirdPartyLibrary插件，我这里是直接Blank

  2. 下载OpenCV SDK，放置Include library到指定目录中

    • OpenCV SDK 根据需要下载指定的版本，我这里下载的是3.4版本的WIndows和Android
    • 需要注意要修改include\opencv2\core\utility.hpp的关于Check代码，否则的话，编译会报错误
      // cofolict with UE4
// #if defined(check)
// #  warning Detected Apple 'check' macro definition, it can cause build conflicts. Please, include this header before any Apple headers.
// #endif

        // cofolict with UE4
  // bool check() const;
    • Windows相对简单拷贝dll lib
      opencv\build\bin中的dll 还有opencv\build\x64\vc15\lib的lib拷贝到 Plugins\OpenCV\Library\Win64中 注意要将两个dll文件拷贝到项目根目录的Binaries/Win64中，否则打开Editor会崩溃
      在这里插入图片描述
    • 拷贝opencv\build\include到Plugins\OpenCV\Include\Win64中
    • Android相对麻烦一些 将OpenCV-android-sdk\sdk\native\3rdparty拷贝到Plugins\OpenCV\Library\Android\3rdparty中
      对应的libs拷贝到对应的libs目录
      注意将liblibpng.a删除掉，会和UE4冲突
      在这里插入图片描述
    • 上面jni目录中只保存libopenc_java3.so 具体看源码中的内容 src目录是后面例子使用的
    • OpenCV-android-sdk\sdk\native\jni\include拷贝到Plugins\OpenCV\Include\Android中，这里注意Windows和Android的.h文件是不同的

    这里插件需要创建对应的Library Include目录

.build.cs的配置

在这里插入图片描述

  • 这里两个模块是用来将OpenCV的Mat数据转换成UTexture2D用的，没有需求可以不加
   if (Target.Platform == UnrealTargetPlatform.Android)
        {
        	//for jni call
    		PrivateDependencyModuleNames.Add("Launch");
        }
  • 这是用来在执行Android JNI函数的，没有需求可以不加
   string OpenCVIncludePath = Path.Combine(ModuleDirectory, "../../Include");
        string OpenCVLibraryPath = Path.Combine(ModuleDirectory, "../../Library");
        
        System.Console.WriteLine("----Architecture -----" + Target.Architecture);
        
        //Load OpenCV Include

        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "Win64");
            
        }
        else if (Target.Platform == UnrealTargetPlatform.Android)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "Android");
        }
        else if (Target.Platform == UnrealTargetPlatform.IOS)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "IOS");
        }
        System.Console.WriteLine("----Include -----" + OpenCVIncludePath);
        PublicIncludePaths.AddRange(new string[] { OpenCVIncludePath });
  • 根据不同的平台 将不同的Include目录添加到PublicIncludePaths中
//Load OpenCV Library
        string OpenCVFinalLibraryPath = "";
        if(Target.Platform == UnrealTargetPlatform.Win64)
        {
	        OpenCVFinalLibraryPath = Path.Combine(OpenCVLibraryPath, "Win64");
	        // Add Library Path 
	        PublicLibraryPaths.Add(OpenCVFinalLibraryPath);
	        
	        PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "opencv_world340.lib"));
	        
            //add dynamic library
            PublicDelayLoadDLLs.Add("opencv_ffmpeg340_64.dll");
            PublicDelayLoadDLLs.Add("opencv_world340.dll");
            RuntimeDependencies.Add(new RuntimeDependency(Path.Combine(OpenCVFinalLibraryPath, "opencv_ffmpeg340_64.dll"))); //for package
            RuntimeDependencies.Add(new RuntimeDependency(Path.Combine(OpenCVFinalLibraryPath, "opencv_world340.dll")));  //for package
            
            System.Console.WriteLine("----Library -----" + OpenCVFinalLibraryPath);
        }
  • Windows平台 加载lib dll
else if(Target.Platform == UnrealTargetPlatform.Android)
        {
	        
            string[] AndroidTarget = { "arm64-v8a", "armeabi", "armeabi-v7a", "mips", "mips64", "x86", "x86_64"  };
            
            string[] ThirdLibName = { "libcpufeatures.a", "libIlmImf.a", "liblibjasper.a", "liblibjpeg.a", 
                "liblibprotobuf.a", "liblibtiff.a", "liblibwebp.a", "libtbb.a", "libtegra_hal.a"
            };
            string[] ThirdLibName_x86 = { "libcpufeatures.a", "libIlmImf.a", "libippicv.a", "libippiw.a", "libittnotify.a",
	            "liblibjasper.a", "liblibjpeg.a",  "liblibprotobuf.a", "liblibtiff.a", "liblibwebp.a", "libtbb.a"
            };
            string[] OpenCVLibName = { "libopencv_calib3d.a", "libopencv_core.a", "libopencv_flann.a", "libopencv_highgui.a",
                "libopencv_imgcodecs.a", "libopencv_imgproc.a", "libopencv_ml.a", "libopencv_objdetect.a",
                "libopencv_photo.a", "libopencv_shape.a", "libopencv_stitching.a", "libopencv_superres.a", "libopencv_video.a",
                "libopencv_videoio.a", "libopencv_videostab.a", "libopencv_dnn.a", "libopencv_features2d.a"
            };
            OpenCVFinalLibraryPath = Path.Combine(OpenCVLibraryPath, "Android");
            
            //add Third lib
            for (int i = 0; i < 3; i++)
            {
	            for (int j = 0; j < ThirdLibName.Length; j++)
	            {
		            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", AndroidTarget[i], ThirdLibName[j]));
		            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", AndroidTarget[i], ThirdLibName[j]));
	            }
            }
			
            for (int j = 0; j < ThirdLibName.Length - 1; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips", ThirdLibName[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips", ThirdLibName[j]));
            }
            for (int j = 0; j < ThirdLibName.Length - 1; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips64", ThirdLibName[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips64", ThirdLibName[j]));
            }
          
            for (int j = 0; j < ThirdLibName_x86.Length ; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86", ThirdLibName_x86[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86", ThirdLibName_x86[j]));
            }
            for (int j = 0; j < ThirdLibName_x86.Length; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86_64", ThirdLibName_x86[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86_64", ThirdLibName_x86[j]));
            }
			
            //add core lib
            for (int i = 0; i < AndroidTarget.Length; i++)
            {
               
                for (int k = 0; k < OpenCVLibName.Length; k++)
                {
                 PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "libs", AndroidTarget[i], OpenCVLibName[k]));
                 System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "libs", AndroidTarget[i], OpenCVLibName[k]));
                }
            }
            
            //load APL.xml
            string APLXmlPath = Utils.MakePathRelativeTo(ModuleDirectory, Target.RelativeEnginePath);
            AdditionalPropertiesForReceipt.Add("AndroidPlugin", Path.Combine(APLXmlPath, "OpenCV_APL.xml"));
            System.Console.WriteLine("----APLXmlPath -----" + Path.Combine(APLXmlPath, "OpenCV_APL.xml"));
        }
        else if(Target.Platform == UnrealTargetPlatform.IOS)
        {

        }

  • Android平台 遍历加载所有的.a库，最后一步需要设置OpenCV_APL.xml 用来指定将so库拷贝到android中，以及Android编译的一些相关配置

  • OpenCV_APL.xml

<?xml version="1.0" encoding="utf-8"?>
<root xmlns:android="http://schemas.android.com/apk/res/android">
	<init>
		<log text="OpenCV Init Begin"/>
	</init>
	
	<prebuildCopies>
		<log text="OpenCV Copy Begin"/>
		<copyDir src="$S(PluginDir)/../../Library/Android" dst="$S(BuildDir)" />
	</prebuildCopies>
	<androidManifestUpdates>
		<log text="OpenCV Add read Camera Permission"/>
		<addPermission android:name="android.permission.CAMERA"/>
	</androidManifestUpdates>
	<soLoadLibrary>
		<loadLibrary name="opencv_java3" failmsg="ARWrapper library not loaded and required!"/>
	</soLoadLibrary>

</root>

  • prebuildCopies用来配置将我们自定义的so还有java文件添加到Android工程中

  • androidManifestUpdates 用来添加相机权限到Android中，没有需求可以不加

  • soLoadLibrary 添加so库

  • 进入Library\Android\jni 创建Android.mk

LOCAL_PATH := $(call my-dir)

PATH_TO_LIBS = libs
 


include $(CLEAR_VARS)
LOCAL_MODULE := opencv_java3
LOCAL_SRC_FILES := $(TARGET_ARCH_ABI)/libopencv_java3.so
include $(PREBUILT_SHARED_LIBRARY)


include $(CLEAR_VARS)
LOCAL_MODULE := UE4
LOCAL_SRC_FILES := $(TARGET_ARCH_ABI)/libUE4.so
include $(PREBUILT_SHARED_LIBRARY)
  • 添加libopencv_java3.so
模块加载
  • 针对Windows平台，需要在StartupModule中加载dll文件 ShutdownModule 释放dll
void FOpenCVModule::StartupModule()
{
	// This code will execute after your module is loaded into memory; the exact timing is specified in the .uplugin file per-module
	const FString PluginDir = IPluginManager::Get().FindPlugin(TEXT("OpenCV"))->GetBaseDir();
	FString LibraryPath;
#if PLATFORM_WINDOWS
	LibraryPath = FPaths::Combine(*PluginDir, TEXT("Library/Win64/"));
	UE_LOG(LogTemp, Warning, TEXT("opencv world LibraryPath == %s"), *(LibraryPath + TEXT("opencv_world340.dll")));
	OpenCV_World_Handler = FPlatformProcess::GetDllHandle(*(LibraryPath + TEXT("opencv_world340.dll")));
	OpenCV_FFmpeg_Handler = FPlatformProcess::GetDllHandle(*(LibraryPath + TEXT("opencv_ffmpeg340_64.dll")));
	if (!OpenCV_World_Handler || !OpenCV_FFmpeg_Handler)
	{
		UE_LOG(LogTemp, Error, TEXT("Load OpenCV dll failed!"));
	}
#elif PLATFORM_ANDROID

#elif PLATFORM_IOS

#endif
}

void FOpenCVModule::ShutdownModule()
{
	// This function may be called during shutdown to clean up your module.  For modules that support dynamic reloading,
	// we call this function before unloading the module.
#if PLATFORM_WINDOWS
	FPlatformProcess::FreeDllHandle(OpenCV_World_Handler);
	OpenCV_World_Handler = nullptr;
	FPlatformProcess::FreeDllHandle(OpenCV_FFmpeg_Handler);
	OpenCV_FFmpeg_Handler = nullptr;
#elif PLATFORM_ANDROID

#elif PLATFORM_IOS

#endif
}
  • 没有什么好解释的，就是一些路径的拼接，以上基本上就是全部的OpenCV插件的接入了，如果编译有问题的话，大概率是库的路径方面有问题，一定要注意拷贝dll文件到Binaries中。

下面是具体的使用例子，其中对摄像机的使用需要用到JNI与Java交互比较麻烦

OpenCVTestActor

使用OpenCV 加载图片

主要知识点:

  • 涉及到在ProjectSetting中设置不被打包成Pack的文件夹

在这里插入图片描述

  • Android端获得Content的绝对路径
extern const FString &GetFileBasePath();
	
	FString ProjectPath = GetFileBasePath();
	FString TestImgPath = FPaths::Combine(ProjectPath, FApp::GetProjectName(), TEXT("Content/TestRes/test.jpg"));
	IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
	if(!PlatformFile.FileExists(*TestImgPath))
	{
		GEngine->AddOnScreenDebugMessage(-1, 10, FColor::Red, FString::Printf(TEXT("TestImgPath == %s not exist "), *TestImgPath));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 10, FColor::Green, FString::Printf(TEXT("TestImgPath == %s exist "), *TestImgPath));
	}

	std::string temp(TCHAR_TO_UTF8(*TestImgPath));

	Mat img = imread(temp, IMREAD_UNCHANGED);
	if(img.empty())
	{	
		GEngine->AddOnScreenDebugMessage(-1, 5, FColor::Red, TEXT("Opencv open img failed!"));
		return;
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 5, FColor::Green, TEXT("Opencv open img Success!"));
	}
  • 使用GetFileBasePath()获得存储卡上UE4Game 路径
WebcamReader
  • Windows端可以直接使用OpenCV的 VideoCapture.open(id)来获取摄像机
  • Android端就麻烦一些了，需要通过JNI调用Android 的Camera进行图像获取，然后回调会C++

这里大致说明一下Android的流程
主要函数在CameraFrame中
captureCamera() == 》Java startPreview() 配置Camera相关
Java onPreviewFrame()更新Data数据 ==> FrameProcessing() 将数据从Java端传递到C++中
GetFrame*( 将经过转换过格式还有翻转方向后的图像数据返回给WebcamReader,用来更新Texture2D内容

最终效果

在这里插入图片描述

PS:
  • 错误 C2872 “ACCESS_MASK”：不是使用using namespace cv,会与windows中的头文件起冲突
  • 错误C4668 没有将“_WIN32_WINNT_WIN10_TH2”定义为预处理器宏，用“0”替换“#if/#elif”
    .build.cs中添加 bEnableUndefinedIdentifierWarnings = false;
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值