[笔记][ProtoPNet]This Looks Like That: Deep Learning for Interpretable Image Recognition

Chen, C., et al. (2019). “This looks like that: deep learning for interpretable image recognition.” Advances in neural information processing systems 32.

has a ‘transparent reasoning process’

our model is able to identify several parts of the image where it thinks that this part of the image looks like that prototypical part of some class, and makes its prediction based on a weighted combination of the similarity scores between parts of the image and the learned prototypes.

Architecture

$f\to g_p\to h$

$f$: conv layer

$g_p$: prototype layer

$h$: FC layer

conv layer

$H_0\times W_0\times D_0$: input dim

$ H\times W\times D$: output dim

$224\times 224\times 3\to 7\times 7\times 128(256,512)$ this work

$W_{conv}$: parameters

prototype layer

P = { p j } j = 1 m P=\{p_j\}_{j=1}^m P={pj​}j=1m​: prototype set

$P_k\subset P$: prototype set w.r.t. class k. $|P_k|=m_k,\sum m_k=m$

k ∈ { 1 , . . . , K } k\in\{1,...,K\} k∈{1,...,K} $K=10$ this work

$ H_1\times W_1\times D$: prototype shape $H_1\le H,W_1\le W$

$H_1=W_1=1$ this work

each prototype will be used to represent some prototypical activation pattern in a patch of the convolutional output, which in turn will correspond to some prototypical image patch in the original pixel space

$g_{p_j}(z)={\max }_{\tilde{z}\in patches(z)}\log (\frac{||\tilde{z}-p_j|{|}_2^2+1}{||\tilde{z}-p_j|{|}_2^2+ϵ})$

all patches of $z$ have the same shape as $p_j$

FC layer

$m$: input(m scores produced by $P$)

$w_h$: weight matrix

Training

possible to cycle following 3 steps more than once

SDG before last layer

fix $w_h$

$w_h^{(k,j)}=1,when{p}_j\in P_k$

$w_h^{(k,j)}=-0.5,when{p}_j\notin P_k$

loss func.

$\underset{P,w_{conv}}{\min }\frac{1}{n}\sum _{i=1}^{n}CrsEnt(h\circ g_P\circ f(x_i),y_i)+{\lambda }_{1}Clst+{\lambda }_{2}Sep$

$Clst=\frac{1}{n}\sum _{i=1}^n\underset{j:p_j\in P_{y_i},z\in patches(f(x_i))}{\min }||z-p_j|{|}_2^2$

$Sep=-\frac{1}{n}\sum _{i=1}^n\underset{j:p_j\in P_{y_i},z\notin patches(f(x_i))}{\min }||z-p_j|{|}_2^2$

prototype projection

$p_j\leftarrow \arg \underset{z\in {\mathcal{Z}}_j}{\min }||z-p_j|{|}_2$

Z j = { z ~ : z ~ ∈ p a t c h e s ( f ( x i ) )   ∀ i   s . t .   y i = k } \mathcal Z_j=\{\tilde z:\tilde z\in patches(f(x_i))\ \forall i\ s.t.\ y_i=k\} Zj​={z~:z~∈patches(f(xi​)) ∀i s.t. yi​=k}

convex optimization of last layer

fix $P,w_{conv}$

$\underset{w_h}{\min }\frac{1}{n}\sum _{i=1}^{n}CrsEnt(h\circ g_P\circ f(x_i),y_i)+\lambda \sum _{k=1}^K\sum _{j:p_j\notin P_k}|w_h^{(k,j)}|$