OpenCV调用caffe版本MobilenetSSD

转载地址
前言

当前，在目标检测领域，基于深度学习的目标检测方法在准确度上碾压传统的方法。基于深度学习的目标检测先后出现了RCNN,FastRCNN,FasterRCNN, 端到端目标检测方法YOLO,YOLO-9000,YOLO-v3, MobileNet-SSD,以及Mask-RCNN等。MobileNet是一种轻量级的网络，本文基于MobileNet-SSD+opencv实现目标检测。

---

开发环境

• windows10 x64
• IntellJ IDEA
• opencv3.4.2
• Visual Studio 2017(测试C++版本MobileNet-SSD)

---

MobileNet-SSD简介

MobileNet-SSD caffe

---

opencv调用MobileNet-SSD

C++版本MobileNet-SSD的运行

目前MobileNet有基于caffe框架训练好的，caffe本身就是C++实现的，因此网上的大部分opencv调用MobileNet都是C++代码。本人先采用vs+opencv3.4.1成功测试之后，再用Java代码进行移植。

Visual Stuido 2017配置opencv过程就不赘述了
MobileNet-SSD训练好的caffe模型在上面的MobileNet-SSD caffe链接下载

#include<iostream>
#include<opencv2/opencv.hpp>
#include<opencv2/dnn.hpp>
using namespace std;
using namespace cv;
using namespace cv::dnn;
class Object
{
public:
    Object();
    Object(int index, float confidence, String name, Rect rect);
    ~Object();
public:
    int index;
    String name;
    float confidence;
    Rect rect;
private:
};
Object::Object() {
}
Object::Object(int index,float confidence,String name,Rect rect) {
    this->index = index;
    this->confidence = confidence;
    this->name = name;
    this->rect = rect;
}
Object::~Object() {
}
//----------------------------全局常量----------------------------------
//配置好protxt文件，网络结构描述文件
//配置好caffemodel文件，训练好的网络权重
const String PROTOTX_FILE ="MobileNetSSD\\MobileNetSSD_deploy.prototxt";
const String CAFFE_MODEL_FILE = "MobileNet-SSD\\MobileNetSSD_deploy.caffemodel";
const String classNames[] = { "background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair",
"cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor" };
const float CONF_THRESH = 0.7f;
int main() {
    //------------------------实例化网络----------------------------
    Net mobileNetSSD = readNetFromCaffe(PROTOTX_FILE, CAFFE_MODEL_FILE);
    if (mobileNetSSD.empty()) {
        cerr << "加载网络失败！" << endl;
        return -1;
    }
    TickMeter t;
    //----------------------设置网络输入-----------------------
    Mat srcImg = imread("D:\\小可爱\\Java学习\\day-6\\代码\\opencv调用MobileNet-SSD\\MobileNet-test.jpg");
    //将二维图像转换为CNN输入的张量Tensor,作为网络的输入
    mobileNetSSD.setInput(blobFromImage(srcImg, 1.0 / 127.5, Size(300, 300), Scalar(127.5, 127.5, 127.5), true, false));
    t.start();
    //--------------------CNN网络前向计算----------------------
    Mat netOut = mobileNetSSD.forward();
    t.stop();
    cout << "检测时间=" << t.getTimeMilli() << "ms" << endl;
    //----------------------解析计算结果-----------------------
    vector<Object> detectObjects;
    Mat detectionResult(netOut.size[2], netOut.size[3], CV_32F, netOut.ptr<float>());
    for (int i = 0; i < detectionResult.rows; i++) {
        //目标类别的索引
        int objectIndex = detectionResult.at<float>(i, 1);
        //检测结果置信度
        float confidence = detectionResult.at<float>(i, 2);
        //根据置信度阈值过滤掉置信度较小的目标
        if (confidence<CONF_THRESH) {
            continue;
        }
        //反归一化，得到图像坐标
        int xLeftUp = static_cast<int>(detectionResult.at<float>(i, 3)*srcImg.cols);
        int yLeftUp = static_cast<int>(detectionResult.at<float>(i, 4)*srcImg.rows);
        int xRightBottom = static_cast<int>(detectionResult.at<float>(i, 5)*srcImg.cols);
        int yRightBottom = static_cast<int>(detectionResult.at<float>(i, 6)*srcImg.rows);
        //矩形框
        Rect rect(Point{ xLeftUp,yLeftUp }, Point{ xRightBottom,yRightBottom });
        //保存结果
        detectObjects.push_back(Object{ objectIndex,confidence,classNames[objectIndex],rect });
    }
    //------------------------显示结果-----------------------------------
    int count = 0;
    for (auto& i:detectObjects) {
        rectangle(srcImg, i.rect, Scalar(0, 255, 255), 2);
        putText(srcImg, i.name, i.rect.tl(), 1, 1.8, Scalar(255, 0, 0),2);
        cout << "第" << count << "个目标：" << i.name << "\t" << i.rect << "\t" << i.confidence << endl;
        count++;
    }
    imshow("MobileNet-SSD", srcImg);
    waitKey(0);
}

测试图像：

MobileNet-test.jpg

代码说明

• 相关API
  • blobFromImage() 将输入的二维图像转换为一个4维的张量/高维矩阵，4维张量的顺序为NCHW(N个数,C通道数，H高度，W宽度 )

   /** @brief Creates 4-dimensional blob from series of images. Optionally resizes and
     *  crops @p images from center, subtract @p mean values, scales values by @p scalefactor,
     *  swap Blue and Red channels.
     *  @param images input images (all with 1-, 3- or 4-channels).
     *  @param size spatial size for output image
     *  @param mean scalar with mean values which are subtracted from channels. Values are intended
     *  to be in (mean-R, mean-G, mean-B) order if @p image has BGR ordering and @p swapRB is true.
     *  @param scalefactor multiplier for @p images values.
     *  @param swapRB flag which indicates that swap first and last channels
     *  in 3-channel image is necessary.
     *  @param crop flag which indicates whether image will be cropped after resize or not
     *  @details if @p crop is true, input image is resized so one side after resize is equal to corresponing
     *  dimension in @p size and another one is equal or larger. Then, crop from the center is performed.
     *  If @p crop is false, direct resize without cropping and preserving aspect ratio is performed.
     *  @returns 4-dimansional Mat with NCHW dimensions order.
     */
    CV_EXPORTS_W Mat blobFromImages(const std::vector<Mat>& images, double scalefactor=1.0,
                                    Size size = Size(), const Scalar& mean = Scalar(), bool swapRB=true, bool crop=true);

• net.forward() 深度神经网络的前向传播计算,返回值也是一个4D的张量

        /** @brief Runs forward pass to compute output of layer with name @p outputName.
         *  @param outputName name for layer which output is needed to get
         *  @return blob for first output of specified layer.
         *  @details By default runs forward pass for the whole network.
         */
        CV_WRAP Mat forward(const String& outputName = String());

---

• 调试步骤，输入图像为RGB图像，包含dog,person

  
  
  

  image.png

• 调试步骤，网络的最终计算输出是一个张量，需要转换为一个2维矩形Mat,是一个7(width)*6(height)的float矩阵，第2列的整数代表目标类别的索引，第3列代表检测结果的置信度，最后4列是归一化的目标矩形框。网络只输出置信度最大的前6个目标，因此输出矩阵的形状为7x6.

  
  
  

  image.png

运行结果

image.png

image.png

检测到了5个目标，2个dog+3个person，后面坐着的2个人以及椅子上中间的人露检了。

---

Java版本MobileNet-SSD的移植

笔者查看了好几遍Java封装的Mat,没有发现如何将一个高维/4D张量转换为2D矩阵的方法，比较坑。在C++ Mat中，Mat有一个成员变量size,进一步查看之后发现是一个结构体，里面封装了指针，在Java Mat中找不到对应的。卡在这里了，只能用JNI了，好麻烦。

以后有时间再继续倒腾Java版本的 MobileNet-SSD。