Log-Linear Model & CRF 条件随机场详解

往期文章链接目录

Log-Linear model

Let $x$ be an example, and let $y$ be a possible label for it. A log-linear model assumes that

$$p(y|x;w)=\frac{\exp [{\sum }_{j=1}^J{w}_j{F}_j(x,y)]}{Z(x,w)}$$

where the partition function

$$Z(x,w)=\sum _{y^{\prime }}\exp [\sum _{j=1}^J{w}_j{F}_j\left(x,y^{\prime }\right)]$$

Note that in ${\sum }_{y^{\prime }}$, we make a summation over all possible $y$. Therefore, given $x$, the label predicted by the model is

$$\hat{y}=\underset{y}{\mathrm{argmax}}p(y|x;w)=\underset{y}{\mathrm{argmax}}\sum _{j=1}^J{w}_j{F}_j(x,y)$$

Each expression $F_j(x,y)$ is called a feature-function. You can think of it as the $j$-th feature extracted from $(x,y)$.

Remark of the log-linear model:

• a linear combination ${\sum }_{j=1}^J{w}_j{F}_j(x,y)$ can take any positive or negative real value; the exponential makes it positive.

• The division makes the result $p(y|x;w)$ between 0 and 1, i.e. makes them be valid probabilities.

Conditional Random Fields (CRF)

Last time, we talked about Markov Random Fields. In this post, we are going to discuss Conditional Random Fields, which is an important special case of Markov Random Fields arises when they are applied to model a conditional probability distribution $p(y|x)$, where $x$ and $y$ are vactor-valued variables.

Formal definition of CRF

Formally, a CRF is a Markov network which specifies a conditional distribution

$$P(y\mid x)=\frac{1}{Z(x)}\prod _{c\in C}{\varphi }_c(x_c,y_c)$$

with partition function

$$Z=\sum _{y\in \mathcal{Y}}\prod _{c\in C}{\varphi }_c(x_c,y_c)$$

we further assume that the factors ${\varphi }_c(x_c,y_c)$ (maximal cliques) are of the form

$${\varphi }_c(x_c,y_c)=\exp [w_c^T{f}_c(x_c,y_c)]$$

Since we require our potential function $\varphi$ to be non-negative, it’s natural to use the exponential function. $f_c(x_c,y_c)$ can be an arbitrary set of features describing the compatibility between $x_c$ and $y_c$. Note that these feature functions could be designed by manually doing feature engineering or using deep learning, LSTM, etc.

Log-linear model to linear-CRF

As a remainder, let $x$ be an example, and let $y$ be a possible label for it. Then a log-linear model assumes that

$$p(y|x;w)=\frac{\exp [{\sum }_{j=1}^J{w}_j{F}_j(x,y)]}{Z(x,w)}$$

From now on, we use the bar notation for sequences. Then to linear-CRF, we write the above equation as

$$\begin{array}{rlrl}p(\bar{y}|\bar{x};w)& =\frac{\exp [{\sum }_{j=1}^J{w}_j{F}_j(\bar{x},\bar{y})]}{Z(\bar{x},w)}& & \\ & =\frac{\exp [{\sum }_{j=1}^J{w}_j{\sum }_{i=2}^T{f}_j(y_{i-1},y_i,\bar{x})]}{Z(\bar{x},w)}& & (1)\end{array}$$

where $y$ can take values from { 1 , 2 , . . . , m } \{1,2,...,m\} { 1,2,...,m}. Here is an example:

Assume we have a sequence $\bar{x}=(x_1,x_2,x_3,x_4)$ and the corresponding hidden sequence $\bar{y}=(y_1,y_2,y_3,y_4)$.

We can divide each feature-function $F_j(\bar{x},\bar{y})$ into fuctions for each maximal clique. That is,

$$F_j(\bar{x},\bar{y})=\sum _{i=2}^T{f}_j(y_{i-1},y_i,\bar{x})\text{\hspace{1em}(1.1)}$$

Perticularly, from the above figure, since we have $3$ maximal cliques, so

F j ( x ˉ , y ˉ ) = f j ( y 1 , y 2 , x ˉ ) + f j ( y 2 , y 3 , x ˉ ) + f j ( y 3 , y 4 , x