Cross-Entropy Loss

Reference:

https://en.wikipedia.org/wiki/Cross_entropy

https://d2l.ai/chapter_linear-networks/softmax-regression.html#loss-function

Definition: Cross-Entropy

The cross-entropy of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows:
$$H(p,q)=-E_p[\log q]\text{\hspace{1em}(1)}$$
where $E_p[\cdot ]$ is the expected value operator with respect to the distribution $p$.

For discrete probability distribution $p$ and $q$ with the same support $\mathcal{X}$ this means:
$$H(p,q)=-\sum _{x\in \mathcal{X}}p(x)\log q(x)\text{\hspace{1em}(2)}$$
The situation for continuous distributions is analogous:
$$H(p,q)=-{\int }_{\mathcal{X}}P(x)\log Q(x)dr(x)\text{\hspace{1em}(3)}$$
N.B: The notation $H(p,q)$ is also used for the joint entropy of $p$ and $q$.

Relation to Log-likelihood

In classification problems we want to estimate the probability of different outcomes. Suppose that the entire dataset { X , y } \{\mathbf X, \mathbf y\} { X,y}​​​​​​ has $N$​​​​​​​ samples, where the sample indexed by $i$​​​​​ consists of a feature vector ${\mathbf{x}}^{(i)}$​​​​​ and a label $y^{(i)}$​​​​​​. Let the estimated probability of outcome $k\in \mathcal{K}$​​​ be $\hat{p}(y=k|\mathbf{x};\mathbf{w})$​​​ and let the frequency (empirical probability) of outcome $k$​​​ in the training set be $q(y=k|\mathbf{x})$​​​. The likelihood of the parameters $\mathbf{w}$​​​​​ is
L ( w ) = ∏ k ∈ K ( est. prob. of  k ) num. of oc