线性链条件随机场CRF

Advantages of CRF

CRF is popularly used in classification problems. There are 5 important reasons:
1. The chief advantage of CRF lies in the fact that it models the conditional distribution $P(y|x)$ rather than the joint distribution $P(y,x)$.
2. A CRF can be used to capture arbitrary dependencies among components of $x$ and $y$.
3. The major difference between CRF with some other existing methods is that it is a “global” model that considers all residues of the protein as a whole rather than focus merely on a local window around the tag to be labeled.
4. In the inference, the states of all tags are predicted simultaneously in a way that maximizes the overall likelihood. The interdependence between the states of adjacent tags is also explicitly exploited through doublet feature functions used in the model.
5. For some cases that chain CRF cannot handle, we can use high order CRF to add high order features to describe the long-range interaction.

Chain CRF

Definition of Chain CRF

In the chain CRF, all the nodes in the graph form a linear chain. According to the nature of undirected graphical model, 　

$$P(y_1,...,y_M|x)=\frac{1}{Z_x}\Pi_{i} (\psi_{i}(Y_i,X)\phi_{i}(Y_i,Y_{i-1},X) )$$
where $\psi_{i}(Y_i,X)$ acts over single labels and $\phi_{i}(Y_i,Y_{i-1},X)$ acts over edges.
$$\psi_{i}(Y_i,X)=exp(\Sigma_k\theta_k S_k(y_i,\textbf{x},i))$$
$$\phi_{i}(Y_i,Y_{i-1},X)=exp(\Sigma_j \lambda_j t_j (y_{i-1},y_i,\textbf{x},i) )$$
After merging all parameter into a single vector,
$$P(y_1,...,y_M|x)=\frac{exp({\Lambda}^T \mathcal{F})}{Z_x}$$
where in the $k \times n$ matrix \mathcal{F}, $F_{ji}=f_j(y_i,y_{i−1},{x},i)$, and $d{\Lambda}$ is a vector( $k \times 1$).
$$y^*=argmax_{y}{\Lambda}^T \mathcal{F}$$

The forward and backward vectors

We use dynamic programming to calculate $Z_x$.

$$\alpha(y,i)=\Sigma_{y^,}\alpha(y^,,i-1)exp({\Lambda}^T{f}(y,y^,,{x},i))$$
$$\beta(y,i)=\Sigma_{y^,}\beta(y^,,i+1)exp({\Lambda}^T {f}(y,y^,,{x},i+1))$$
where ${f} (., ., ., i)$ is the feature vector evaluated at the $i^{th}$ sequence position. $\alpha$ and $\beta$ are called the forward and backward vectors respectively, and both have a time complexity of $O(TN^2)$.\
Then we can write the marginals and partition function as below:
$$P(Y_i=y|{x})=\alpha(y,i)\beta(y,i)/Z_x$$
$$P(Y_i=y,Y_{i+1}=y^,|{x})=\alpha(y,i) \beta(y,i+1) exp({\Lambda}^T {f}(y,y^,,{x},i+1))/Z_x$$
$$Z_x=\Sigma_y \alpha(y,T) = \Sigma_y \beta(y,1)$$

Inference of Chain CRF

Inference in linear CRF using the Viterbi algorithm:

$$\delta(y,i)=\max_{y^,} \delta(y^,,i-1)exp({\Lambda}^T {f}(y,y^,,{x},i))$$
where the normalized probability of the best labeling is given by $\frac{\max_y \delta(n,y)}{Z_x}$,and the best labeling is given by $argmax_y \delta(n,y)$
The time complexity is $O(TN^2)$.
We can also use pseudo likelihood to perform the inference.
$$y_t^*=argmax_{y_t}P(y_t|{\Lambda, x})$$

Training of Chain CRF

CRF too suffer from the bane of overfitting, we impose a penalty on large parameter values. The penalized log-likelihood is given by:

$$\mathcal{L}_\Lambda = \Sigma_k({\Lambda^T F({y_k,x_k}) - logZ_{\Lambda}({x^k}) ) - \frac{||\Lambda||^2}{2\sigma^2}}$$
and the gradient is given by:
$$\nabla{{\mathcal{L}_\Lambda}} = \Sigma_k( F(y^k,x^k) - E_{P(y|x^k)[ F(y,x^k)]} ) ) - \frac{\Lambda}{\sigma^2}$$
where $E_{P(y|x^k)[F(y,x^k)]} = \Sigma_i \Sigma_{y,y^,} \alpha(i-1,y^,) \beta(i,y) exp({\Lambda}^T {f}(y,y^,,{x^k},i))$.\
Then Gradient Descent method (you can find in the last blog) can be used to train all the parameters.