损失函数整理

---

DSH中的哈希函数

$L_r(b_1, b_2, y) = \sum_{i=1}^N \{\frac{1}{2}(1-y_i)||b_{i,1}-b_{i,2}||_2^2+\frac{1}{2}y_i max(m-||b_{i,1}-b_{i,2}||_2^2, 0) + \alpha(|||b_{i,1}|-1||_1+|||b_{i,2}|-1||_1)\}$

Specially, for a pair of images $I_1,I_2\in \Omega$ and the corresponding binary network outputs $b_1, b_2$, we dfine $y=0$ if they are similar, and $y=1$ otherwise.

---

L1Loss

creates a criterion that measures the mean absolute value of the element-wise difference between input x and target y:
$loss(x,y)=\frac{1}{n}\sum|x_i-y_i|$

---

MSELoss

$loss(x,y)=\frac{1}{n}|x_i-y_i|^2$
measures the mean squared error between n elements in the input x and target y .

---

CrossEntropyLoss

the loss can be described as :
$loss(x, class)=-\log\frac{e^{x_{class}}}{\sum_je^{x_j}}=-x_{class}+\log(\sum_je^{x_j})$
or in the case of the weights argument being specified :
$loss(x,class)=w_{class}*(-x_{class}+\log(\sum_je^{x_j}))$

---

NLLLoss

the loss can be described as :
$loss(x, class) = -x_{class}$
or ine the case of the weights argument it is specified as follows:
$loss(x, class)=-w_{class}*x_{class}$

---

KLDivLoss

the loss can be described as:
$loss(x,target)=\frac{1}{n}\sum(target_i*(log(target_i)-x_i))$

---

BCELoss

creates a criterion that measures the Binary Cross Entropy between the target and the output:
$loss(o,t)=-\frac{1}{n}\sum_i(t_i*\log(o_i)+(1-t_i)*\log(1-o_i))$
or in the case of the weights argument being specified:
$loss(o,t)=-\frac{1}{n}\sum_iw_i(t_i*\log(o_i)+(1-t_i)*\log(1-o_i))$
This is used for measureing the error of a reconstruction in for example an auto-encoder. Note that the targets $t_i$ should be numbers between 0 and 1.

---

BCEWithLogitsLoss

this Binary Cross Entropy between the target and the output logits( no sigmoid applied) is:
$loss(o,t)=-\frac{1}{n}\sum_i(t_i*\log(sigmoid(o_i))+(1-t_i)*\log(1-sigmoid(o_i)))$
or in the case of the weights argument being specified:
$loss(o,t)=-\frac{1}{n}\sum_iw_i(t_i*\log(sigmoid(o_i))+(1-t_i)*\log(1-sigmoid(o_i)))$

---

MarginRankingLoss

if y==1 then it assumed the first input should be ranked higher(have a larger ) than the second input , and vice-versa for y==-1.
the loss function for each sample in the mini-batch is:
$loss(x,y)=max(0, -y*(x_1-x_2)+margin)$

---

HingeEmbeddingLoss

$loss(x,y)=\frac{1}{n} \begin{cases}x_i, & y_i==1 \\\ max(0, margin-x_i), &y_i==-1\end{cases}$

---

MultiLLabelMarginLoss

$loss(x,y)=\frac{\sum_{i,j}(max(0, 1-(x_{y_j}-x_i)))}{x.size(0)}$
where i=0 to x.size(0), j=0 to y.size(0), $y_j!=0$, and $i!=y_j$ for all i and j.
x and y must have the same size.
The criterion only considers the first non zero $y_j$ targets. This allows for different samples to have variable amounts of target calsses

---

SmoothL1Loss

Creates a criterion that uses a squared term if the absolute element-wise error falls below 1 and an L1 term otherwise. It is less sensitive to outliers than the MSELoss and in some cases prevents exploding gradients (e.g. see “Fast R-CNN” paper by Ross Girshick). Also known as the Huber loss:
$loss(x,y)=\frac{1}{n}\sum\begin{cases}0.5*(x_i-y_i)^2, |x_i- y_i|<1\\\ |x_i-y_i|-0.5, otherwise\end{cases}$

---

SoftMarginLoss

$loss(x,y)=\frac{\sum_i\log(1+e^{-y_i*x_i})}{x.nelement()}$

---

MultiLabelSoftMarginLoss

$loss(x,y)=-\sum_i(y_i*\log\frac{1}{1+e^{-x_i}}+(1-y_i)*\log\frac{e^{-x_i}}{1+e^{-x_i}})$
where i==0 to x.nelement-1, $y_i$ in {0, 1}
and x must have the same size.

---

CosineEmbeddingLoss

$loss(x,y)=\begin{cases}1-cos(x_1, x_2), y==1\\\ max(0, cos(x_1, x_2)-margin), y==-1\end{cases}$

---

MultiMarginLoss

$loss(x,y)=\frac{\sum_imax(0,(margin-x_y+x_i))^p}{x.size(0)}$
where i==0 to x.size(0) and i!=y.
Optionally, you can give non-equal weighting on classes by passing a 1D weights tensor into the constructor.
The loss function then becomes:
$loss(x,y)=\frac{\sum_imax(0,w_y*(margin-x_y+x_i))^p}{x.size(0)}$

---

TripletMarginLoss

The distance swap is described in detail in the paper Learning shallow convolutional feature descriptors with triplet losses by V.Balntas, E.Riba et al.
$L(a,p,n) = \frac{1}{N}(\sum_{i=1}^Nmax\{d(a_i, p_i)-d(a_i, n_i)+margin, 0\})$
where $d(x_i, y_i)=||x_i-y_i||_2^2$