[caffe笔记006]：caffe中的softmax层推导

1. SoftmaxParameter

• engine
  
  • CAFFE = 1
  • CUDNN = 2
• axis
  沿着第几维做softmax，可以是正数(正序)和负数(倒序)

// Message that stores parameters used by SoftmaxLayer, SoftmaxWithLossLayer
message SoftmaxParameter {
  enum Engine {
    DEFAULT = 0;
    CAFFE = 1;
    CUDNN = 2;
  }
  optional Engine engine = 1 [default = DEFAULT];

  // The axis along which to perform the softmax -- may be negative to index
  // from the end (e.g., -1 for the last axis).
  // Any other axes will be evaluated as independent softmaxes.
  optional int32 axis = 2 [default = 1];
}

2. 结构

caffe里的softmax层值关注softmax函数本身，是一个二部图。对于输入

$$z_i (i=1,\cdots ,k),$$
其对应输出为
$${a_i} = {{{e^{{z_i}}}} \over {\sum\limits_{j = 1}^k {{e^{{z_j}}}} }}.$$

• bottom
  如果[N C H W]的矩阵，N为batchsize，C为类别数，HxW被称作spatial dimention，此时$z_i$就是第二维上的元素
• top
  [N C H W]的矩阵，为对应输出

3. 正向传播

计算过程
Step1: 计算输入最大值

$$z = \max \{ {z_i}\}$$
Step2: 减去最大值
$${z_i} = {z_i} - z$$
Step3: 求指数
$${z_i} = {e^{{z_i}}}$$
Step4: 求和
$${z_{sum}} = \sum\limits_{i = 1}^k {{e^{{z_i}}}}$$
Step5: softmax
$${z_i} = {{{z_i}} \over {{z_{sum}}}}$$

代码

template <typename Dtype>
void SoftmaxLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom,
    const vector<Blob<Dtype>*>& top) {
  const Dtype* bottom_data = bottom[0]->cpu_data();
  Dtype* top_data = top[0]->mutable_cpu_data();
  Dtype* scale_data = scale_.mutable_cpu_data();
  // channles对应类别的数目
  int channels = bottom[0]->shape(softmax_axis_);
  int dim = bottom[0]->count() / outer_num_;
  // 对top进行初始化，初始化为bottom的值
  caffe_copy(bottom[0]->count(), bottom_data, top_data);
  // We need to subtract the max to avoid numerical issues, compute the exp,
  // and then normalize.
  // 对outer_num_个数据依次处理
  for (int i = 0; i < outer_num_; ++i) {
    // 初始化scale_data为第i个数据，并找到softmax对应维度上的数据最大值
    // Step1:
    // initialize scale_data to the first plane
    caffe_copy(inner_num_, bottom_data + i * dim, scale_data);
    for (int j = 0; j < channels; j++) {
      for (int k = 0; k < inner_num_; k++) {
        scale_data[k] = std::max(scale_data[k],
            bottom_data[i * dim + j * inner_num_ + k]);
      }
    }
    // subtraction, Step2:
    caffe_cpu_gemm<Dtype>(CblasNoTrans, CblasNoTrans, channels, inner_num_,
        1, -1., sum_multiplier_.cpu_data(), scale_data, 1., top_data);
    // exponentiation, Step3:
    caffe_exp<Dtype>(dim, top_data, top_data);
    // sum after exp, Step4:
    caffe_cpu_gemv<Dtype>(CblasTrans, channels, inner_num_, 1.,
        top_data, sum_multiplier_.cpu_data(), 0., scale_data);
    // division, Step5:
    for (int j = 0; j < channels; j++) {
      caffe_div(inner_num_, top_data, scale_data, top_data);
      top_data += inner_num_;
    }
  }
}

4. 反向传播

计算过程
Step1: 求top_data和top_diff内积
Step2: 求差值，对应推导过程$\left( {{{\partial l} \over {\partial {a_i}}} - {{\partial l} \over {\partial a}} \cdot a} \right)$
Step3: 相乘，求偏导

推导过程

$${{\partial l} \over {\partial z}} = {{\partial l} \over {\partial a}}{{\partial a} \over {\partial z}}$$
其中： ${{\partial l} \over {\partial a}} =\rm{top-diff}$， $a = {\rm{top-data}}$。
而：
$${{\partial {a^i}} \over {\partial {z^j}}} = {{\partial \left( {{{{e^{{z_i}}}} \over {\sum\limits_k {{e^{{z_k}}}} }}} \right)} \over {\partial {z^j}}}$$
当 $i\ne j$时
$${{\partial {a^i}} \over {\partial {z^j}}} = - {{{e^{{z_i}}}{e^{{z_j}}}} \over {{{\left( {\sum\limits_k {{e^{{z_k}}}} } \right)}^2}}} = - {a_i}{a_j}$$
当 $i=j$时：
$${{\partial {a^i}} \over {\partial {z^j}}} = {{{e^{{z_i}}}} \over {\sum\limits_k {{e^{{z_k}}}} }} - {{{e^{{z_i}}}{e^{{z_j}}}} \over {{{\left( {\sum\limits_k {{e^{{z_k}}}} } \right)}^2}}} = {a_j} - {a_j}{a_j}$$
可以推出：
$${{\partial l} \over {\partial {z^j}}} = {{\partial l} \over {\partial a}}{{\partial a} \over {\partial {z^j}}} = \sum\limits_i {{{\partial l} \over {\partial {a_i}}}} {{\partial {a_i}} \over {\partial {z^j}}} = - \left( {{{\partial l} \over {\partial a}} \cdot a} \right){a_j} + {{\partial l} \over {\partial {a_j}}}{a_j} = \left( {{{\partial l} \over {\partial {a_j}}} - {{\partial l} \over {\partial a}} \cdot a} \right){a_j}$$

代码

template <typename Dtype>
void SoftmaxLayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
    const vector<bool>& propagate_down,
    const vector<Blob<Dtype>*>& bottom) {
  const Dtype* top_diff = top[0]->cpu_diff();
  const Dtype* top_data = top[0]->cpu_data();
  Dtype* bottom_diff = bottom[0]->mutable_cpu_diff();
  Dtype* scale_data = scale_.mutable_cpu_data();
  int channels = top[0]->shape(softmax_axis_);
  int dim = top[0]->count() / outer_num_;
  caffe_copy(top[0]->count(), top_diff, bottom_diff);
  for (int i = 0; i < outer_num_; ++i) {
    // compute dot(top_diff, top_data) and subtract them from the bottom diff, Step1:
    for (int k = 0; k < inner_num_; ++k) {
      scale_data[k] = caffe_cpu_strided_dot<Dtype>(channels,
          bottom_diff + i * dim + k, inner_num_,
          top_data + i * dim + k, inner_num_);
    }
    // subtraction, Step2:
    caffe_cpu_gemm<Dtype>(CblasNoTrans, CblasNoTrans, channels, inner_num_, 1,
        -1., sum_multiplier_.cpu_data(), scale_data, 1., bottom_diff + i * dim);
  }
  // elementwise multiplication, Step3:
  caffe_mul(top[0]->count(), bottom_diff, top_data, bottom_diff);
}