java 深度包检测_opencv for java之——深度学习目标检测MobileNet-SSD

前言

当前，在目标检测领域，基于深度学习的目标检测方法在准确度上碾压传统的方法。基于深度学习的目标检测先后出现了RCNN,FastRCNN,FasterRCNN, 端到端目标检测方法YOLO,YOLO-9000,YOLO-v3, MobileNet-SSD,以及Mask-RCNN等。MobileNet是一种轻量级的网络，本文基于MobileNet-SSD+opencv实现目标检测。

开发环境

windows10 x64

IntellJ IDEA

opencv3.4.2

Visual Studio 2017(测试C++版本MobileNet-SSD)

MobileNet-SSD简介

opencv调用MobileNet-SSD

C++版本MobileNet-SSD的运行

目前MobileNet有基于caffe框架训练好的，caffe本身就是C++实现的，因此网上的大部分opencv调用MobileNet都是C++代码。本人先采用vs+opencv3.4.1成功测试之后，再用Java代码进行移植。

Visual Stuido 2017配置opencv过程就不赘述了

MobileNet-SSD训练好的caffe模型在上面的MobileNet-SSD caffe链接下载

C++代码：

#include

#include

#include

using namespace std;

using namespace cv;

using namespace cv::dnn;

class Object

{

public:

Object();

Object(int index, float confidence, String name, Rect rect);

~Object();

public:

int index;

String name;

float confidence;

Rect rect;

private:

};

Object::Object() {

}

Object::Object(int index,float confidence,String name,Rect rect) {

this->index = index;

this->confidence = confidence;

this->name = name;

this->rect = rect;

}

Object::~Object() {

}

//----------------------------全局常量----------------------------------

//配置好protxt文件，网络结构描述文件

//配置好caffemodel文件，训练好的网络权重

const String PROTOTX_FILE ="MobileNetSSD\\MobileNetSSD_deploy.prototxt";

const String CAFFE_MODEL_FILE = "MobileNet-SSD\\MobileNetSSD_deploy.caffemodel";

const String classNames[] = { "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair",

"cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor" };

const float CONF_THRESH = 0.7f;

int main() {

//------------------------实例化网络----------------------------

Net mobileNetSSD = readNetFromCaffe(PROTOTX_FILE, CAFFE_MODEL_FILE);

if (mobileNetSSD.empty()) {

cerr << "加载网络失败！" << endl;

return -1;

}

TickMeter t;

//----------------------设置网络输入-----------------------

Mat srcImg = imread("D:\\小可爱\\Java学习\\day-6\\代码\\opencv调用MobileNet-SSD\\MobileNet-test.jpg");

//将二维图像转换为CNN输入的张量Tensor,作为网络的输入

mobileNetSSD.setInput(blobFromImage(srcImg, 1.0 / 127.5, Size(300, 300), Scalar(127.5, 127.5, 127.5), true, false));

t.start();

//--------------------CNN网络前向计算----------------------

Mat netOut = mobileNetSSD.forward();

t.stop();

cout << "检测时间=" << t.getTimeMilli() << "ms" << endl;

//----------------------解析计算结果-----------------------

vector detectObjects;

Mat detectionResult(netOut.size[2], netOut.size[3], CV_32F, netOut.ptr());

for (int i = 0; i < detectionResult.rows; i++) {

//目标类别的索引

int objectIndex = detectionResult.at(i, 1);

//检测结果置信度

float confidence = detectionResult.at(i, 2);

//根据置信度阈值过滤掉置信度较小的目标

if (confidence

continue;

}

//反归一化，得到图像坐标

int xLeftUp = static_cast(detectionResult.at(i, 3)*srcImg.cols);

int yLeftUp = static_cast(detectionResult.at(i, 4)*srcImg.rows);

int xRightBottom = static_cast(detectionResult.at(i, 5)*srcImg.cols);

int yRightBottom = static_cast(detectionResult.at(i, 6)*srcImg.rows);

//矩形框

Rect rect(Point{ xLeftUp,yLeftUp }, Point{ xRightBottom,yRightBottom });

//保存结果

detectObjects.push_back(Object{ objectIndex,confidence,classNames[objectIndex],rect });

}

//------------------------显示结果-----------------------------------

int count = 0;

for (auto& i:detectObjects) {

rectangle(srcImg, i.rect, Scalar(0, 255, 255), 2);

putText(srcImg, i.name, i.rect.tl(), 1, 1.8, Scalar(255, 0, 0),2);

cout << "第" << count << "个目标：" << i.name << "\t" << i.rect << "\t" << i.confidence << endl;

count++;

}

imshow("MobileNet-SSD", srcImg);

waitKey(0);

}

测试图像：

MobileNet-test.jpg

代码说明

相关API

blobFromImage() 将输入的二维图像转换为一个4维的张量/高维矩阵，4维张量的顺序为NCHW(N个数,C通道数，H高度，W宽度 )

/** @brief Creates 4-dimensional blob from series of images. Optionally resizes and

* crops @p images from center, subtract @p mean values, scales values by @p scalefactor,

* swap Blue and Red channels.

* @param images input images (all with 1-, 3- or 4-channels).

* @param size spatial size for output image

* @param mean scalar with mean values which are subtracted from channels. Values are intended

* to be in (mean-R, mean-G, mean-B) order if @p image has BGR ordering and @p swapRB is true.

* @param scalefactor multiplier for @p images values.

* @param swapRB flag which indicates that swap first and last channels

* in 3-channel image is necessary.

* @param crop flag which indicates whether image will be cropped after resize or not

* @details if @p crop is true, input image is resized so one side after resize is equal to corresponing

* dimension in @p size and another one is equal or larger. Then, crop from the center is performed.

* If @p crop is false, direct resize without cropping and preserving aspect ratio is performed.

* @returns 4-dimansional Mat with NCHW dimensions order.

*/

CV_EXPORTS_W Mat blobFromImages(const std::vector& images, double scalefactor=1.0,

Size size = Size(), const Scalar& mean = Scalar(), bool swapRB=true, bool crop=true);

net.forward() 深度神经网络的前向传播计算,返回值也是一个4D的张量

/** @brief Runs forward pass to compute output of layer with name @p outputName.

* @param outputName name for layer which output is needed to get

* @return blob for first output of specified layer.

* @details By default runs forward pass for the whole network.

*/

CV_WRAP Mat forward(const String& outputName = String());

调试步骤，输入图像为RGB图像，包含dog,person

image.png

调试步骤，网络的最终计算输出是一个张量，需要转换为一个2维矩形Mat,是一个7(width)*6(height)的float矩阵，第2列的整数代表目标类别的索引，第3列代表检测结果的置信度，最后4列是归一化的目标矩形框。网络只输出置信度最大的前6个目标，因此输出矩阵的形状为7x6.

image.png

运行结果

image.png

image.png

检测到了5个目标，2个dog+3个person，后面坐着的2个人以及椅子上中间的人露检了。

Java版本MobileNet-SSD的移植

笔者查看了好几遍Java封装的Mat,没有发现如何将一个高维/4D张量转换为2D矩阵的方法，比较坑。在C++ Mat中，Mat有一个成员变量size,进一步查看之后发现是一个结构体，里面封装了指针，在Java Mat中找不到对应的。卡在这里了，只能用JNI了，好麻烦。

以后有时间再继续倒腾Java版本的 MobileNet-SSD。