Log-Sum-Exp Pooling

Log-Sum-Exp Pooling

Papers

1. From Image-level to Pixel-level Labeling with Convolutional Networks
2. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases

LSE Pooling

在阅读这两篇文章之前，我印象中常用的 Pooling 有 Max Pooling 和 Average Pooling，而这两篇文章中用到了 Log-Sum-Exp Pooling，其定义为：

$$x_p=\frac{1}{r}\cdot log[\frac{1}{S}\cdot \sum _{(i,j)\in \mathbf{S}}exp(r\cdot x_{ij})]$$

其中，$x_{ij}$ 表示在$(i,j)$的激活值，$(i,j)$ 是池化区域 $\mathbf{S}$ 的一点并且 $S=s\times s$ 是池化区域 $\mathbf{S}$ 总点数，$r$ 是超参数。

在第一篇文章中，作者提到 LSE Pooling 的作用为：

The hyper-parameter r controls how smooth one wants the approximation to be: high r values implies having an effect similar to the max, very low values will have an effect similar to the score averaging. The advantage of this aggregation is that pixels having similar scores will have a similar weight in the training procedure, r controlling this notion of “similarity”.

在第二篇文章中，作者提到 LSE Pooling 的作用为：

By controlling the hyper-parameter, r, the pooled value ranges from the maximum in S (when $r\to \infty$) to average ($r\to 0$).

一个直观的理解可以看下图：

数学证明

作为一个严谨的大学僧，肯定不会止步于直观理解啦，数学证明走起！

在证明前，不妨把式子简化一点：

$$x_p=\frac{1}{r}\cdot log[\frac{1}{n}\cdot \sum _{i=1}^{n}exp(r\cdot x_i)]$$

证明 $r\to 0$ 相当于 Average Pooling

首先，我们需要借助均值不等式：

$$\frac{a_1+a_2+...+a_n}{n}\ge \sqrt[n]{a_1\cdot a_2...a_n}$$

当且仅当 $a_1=a_2=...=a_n$ 时取等号。
$$\begin{array}{rl}{x}_p& =\frac{1}{r}\cdot log[\frac{1}{n}\cdot \sum _{i=1}^{n}exp(r\cdot x_i)]\\ & =log(\frac{1}{n}\cdot \sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}\end{array}$$

应用均值不等式：

$$\begin{array}{rl}(\frac{1}{n}\cdot \sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}& \ge (\prod _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{n}\cdot \frac{1}{r}}\\ & =(\prod _{i=1}^n{e}^{x_i}{)}^{\frac{1}{n}}\end{array}$$
当 $r=0$ 时，可取等号。代入整个式子：

$$\begin{array}{rl}{x}_p& =log(\frac{1}{n}\cdot \sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}\\ & \ge log(\prod _{i=1}^n{e}^{x_i}{)}^{\frac{1}{n}}\\ & =\frac{1}{n}\sum _{i=1}^n{x}_i\end{array}$$
于是 $r\to 0$ 相当于 Average Pooling 得证。

证明 $r\to \infty$ 相当于 Max Pooling

$$\begin{array}{rl}{x}_p& =\frac{1}{r}\cdot log[\frac{1}{n}\cdot \sum _{i=1}^{n}exp(r\cdot x_i)]\\ & =log(\sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}-\frac{1}{r}\cdot log(n)\end{array}$$

因为 $r>0$，我们有：

$$\begin{array}{r}max(e^{r\cdot x_i}{)}^{\frac{1}{r}}\le (\sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}\le [n\cdot max(e^{r\cdot x_i}){]}^{\frac{1}{r}}\end{array}$$
代入整个式子，得：

$$max(x_i)\le log(\sum _{i=1}^n{e}^{r\cdot x_i}{)}^{\frac{1}{r}}\le \frac{1}{r}\cdot log(n)+max(x_i)$$
当 $r\to \infty$ 时有：$\frac{1}{r}\cdot log(n)\to 0$，故 $r\to \infty$ 相当于 Max Pooling 得证