caffe中解析solver.prototxt到类SolverParameter的过程解析

caffe中解析solver.prototxt到类SolverParameter的过程解析

此篇博客主要是重现了caffe中从solver.prototxt文件中获取参数到类Solver（该类控制着网络的前向计算以及反向计算的过程，网络计算的核心流程都在这里实现，管理着网络反向传播时使用的各种优化器，比如：SGD，Adam等接口，以及相应的网络配置参数，比如：学习率，动量等）的过程。实验过程从c++源码开始构建，不依赖于编译好的caffe文件。主要是将caffe源码中实现该过程的源码进行重现。

• 实验环境：Ubuntu14.0.4
• 编译工具：gcc 4.8.4

---

一：源码中涉及的源文件简介

• caffe-master/src/caffe/proto/caffe.proto:该文件主要定义了在各种.prototxt文件中定义变量的格式。该文件用于生成c++的源代码，使用方法如下：protoc –proto_path=src/caffe/proto –cpp_out=.build_release/src/caffe/proto src/caffe/proto/caffe.proto 。(此处的短横线是两个短横线，Markdown编辑器的原因)
• caffe-master/src/caffe/solver.cpp：该文件定义了Solver类
• caffe-master/src/caffe/util/upgrade_proto.cpp：该文件主要是Solver类做一些异常处理，判断该Solver类型是否需要更新
• caffe-master/src/caffe/util/io.cpp：该文件主要是对文件以及数据的读取操作。

二：该过程涉及到的动态链接库

• libglog：该库主要用于打印信息，Google公司开发。源代码涉及的CHECK（）等宏都是来于该库。
• libprotobuf：该库主要用于解析.prototxt文件。

三：代码重建过程

（一）：工程的文件结构

proto/

caffe.proto

caffe.pb.h

caffe.pb.cc

io/

io.hpp

io.cpp

upgrade_proto/

upgrade_proto.hpp

upgrade_proto.cpp

solver.hpp solver.cpp makefile

注意：在文件夹proto/下的caffe.pb.h以及caffe.pb.cc文件由protoc编译器依据caffe.proto文件生成

（二）：各个文件的代码

io/io.hpp

#ifndef CAFFE_UTIL_IO_H_
#define CAFFE_UTIL_IO_H_
#include <iostream>
#include "google/protobuf/message.h"
namespace caffe {
    using ::google::protobuf::Message;
    bool ReadProtoFromTextFile(const char* filename,Message* proto);
    inline bool ReadProtoFromTextFile(const std::string& filename, Message* proto){
        return ReadProtoFromTextFile(filename.c_str(), proto);
    }
}
#endif

io/io.cpp

#include <fcntl.h>
#include <fstream>
#include <google/protobuf/io/coded_stream.h>
#include <google/protobuf/io/zero_copy_stream_impl.h> //including FileInputStream
#include <google/protobuf/text_format.h>
#include "io.hpp"
#include <glog/logging.h> // including CHECK macro 
namespace caffe {
using google::protobuf::Message; //Message is a abstract class, you must impliment a specific class.
using google::protobuf::io::FileInputStream;
bool ReadProtoFromTextFile(const char* filename, Message* proto) {
    int fd = open(filename, O_RDONLY);
    CHECK_NE(fd, -1) << "File not found: "<< filename;
    FileInputStream* input = new FileInputStream(fd);
    bool success = google::protobuf::TextFormat::Parse(input, proto);
    delete input;
    close(fd);
    return success;
}

upgrade_proto/upgrade_proto.hpp

#ifndef CAFFE_UTIL_UPGRADE_PROTO_H_
#define CAFFE_UTIL_UPGRADE_PROTO_H_
#include <iostream>
#include "../proto/caffe.pb.h"
namespace caffe {
void ReadSolverParamsFromTextFileOrDie(const std::string& param_file,SolverParameter* param);
}
#endif

upgrade_proto/upgrade_proto.cpp

#include <google/protobuf/text_format.h>
#include <google/protobuf/io/coded_stream.h>
#include <google/protobuf/io/zero_copy_stream_impl.h>
#include "upgrade_proto.hpp"
#include <glog/logging.h>
#include "../io/io.hpp"
namespace caffe{
bool SolverNeedsTypeUpgrade(const SolverParameter& solver_param){
    if(solver_param.has_solver_type()){
        std::cout << "return true" << std::endl;
        return true;
    }
    std::cout << "return false" << std::endl;
    return false;
}
bool UpgradeSolverType(SolverParameter* solver_param) {
    CHECK(!solver_param->has_solver_type() || !solver_param->has_type())
        << "Failed to upgrade solver: old solver_type field (enum) and new type "
        << "field (string) cannot be both specified in solver proto text.";
    if(solver_param->has_solver_type()){
        std::string type;
        switch(solver_param->solver_type()){
            case SolverParameter_SolverType_SGD:
                type = "SGD";
                break;
            case SolverParameter_SolverType_NESTEROV:
                type = "Nesterov";
             break;
            case SolverParameter_SolverType_ADAGRAD:
                type = "AdaGrad";
                break;
            case SolverParameter_SolverType_RMSPROP:
                type = "RMSProp";
                break;
            case SolverParameter_SolverType_ADADELTA:
                type = "AdaDelta";
                break;
            case SolverParameter_SolverType_ADAM:
                type = "Adam";
                break;
            default:
                LOG(FATAL) << "Unknown SolverParameter solver_type: " << type;
        }
        solver_param->set_type(type);
        solver_param->clear_solver_type();
    } else {
        LOG(ERROR) << "Warning: solver type already up to date. ";
        return false;
    }
    return true;
}
bool UpgradeSolverAsNeeded(const std::string& param_file, SolverParameter* param){
    bool success = true;
    if(SolverNeedsTypeUpgrade(*param)){
        std::cout << "in UpgradeSolverAsNeeded" << std::endl;
        LOG(INFO) << "Attempting to upgrade input file specified using deprecated "
            << "'solver_type'field (enum)': " <<param_file;
        if (!UpgradeSolverType(param)){
            success = false;
            LOG(ERROR) << "Warning: had one or more problems upgrading "
                << "SolverType (see above). ";
        } else {
            LOG(INFO) << "Successfully upgraded file specified using deprecated "
                << "'solver_type' field (enum) to 'type' field (string).";
            LOG(WARNING) << "Note that futher Caffe release will only support "
                << "'type' field (string) for a solver's type.";
        }
    }
    return success;
}
void ReadSolverParamsFromTextFileOrDie(const std::string& param_file, SolverParameter* param){
    CHECK(ReadProtoFromTextFile(param_file, param))
        << "Failed to parse SolverParameter file: " << param_file;
    std::cout << "In ReadSolverParamsFromTextFileOrDie" << std::endl;
    UpgradeSolverAsNeeded(param_file, param);

}
}

solver.hpp

#ifndef CAFFE_SOLVER_HPP_
#define CAFFE_SOLVER_HPP_
#include <string>
namespace caffe{
template <typename Dtype>
    class Solver{
        public:
            explicit Solver(const std::string& param_file);
    };
}
#endif

solver.cpp

#include "solver.hpp"
#include <iostream>
#include "proto/caffe.pb.h"
#include "./upgrade_proto/upgrade_proto.hpp"
namespace caffe{
    template<typename Dtype>
    Solver<Dtype>::Solver(const std::string& param_file){
        //std::cout<<"hello"<<param_file<<std::endl;
        SolverParameter param;
        ReadSolverParamsFromTextFileOrDie(param_file, &param);
        std::cout << param.ADAM << std::endl;
        //std::cout << param.SolverType_Name(param.ADAM) << std::endl;
    }
}
int main(){
    caffe::Solver<float> solver("/home/wanghao/workspace/caffe-source-build/examples/mnist/lenet_adadelta_solver.prototxt");
    //std::cout << solver.ADAM << std::endl;
    return 0;
}

makefile

obj_file := $(shell find . -name '*.cpp')
pro_file := $(shell find . -name '*.pb.cc')
inc_dire := /home/wanghao/workspace/caffe-source-study/solver/
all:$(obj_file)
    g++ $(obj_file) $(pro_file) -o solver -lglog -lprotobuf -I$(inc_dire) -DDEBUG -O0 -g

由于水平有限，因此对代码的原理不过多解释，此工程按照文件结构，直接make即可成功，注意此处编译的代码为调试模式。可以gdb进行调试。