Pattern Recognition course 笔记 -Semi-supervised Learning

仅供个人笔记使用

---

A pattern recognition problem

• goal

there are large “labeled” data online e.g. tweets using hash #
can we use these unlabel data to improve our classifier

• labeled data
  
• unlabeled data

-some applications

• image classification (easy to obtain images e.g, from flicker)
• protein function prediction
• document classification
• part of speech tagging

-semi-supervised classification

• similar but with continuous out come measure
• using some labels to improve a clustering solution
• measure how well the unlabeled data could help to improve

Content

self-learning

One of the earliest studies on SSL (Hartley & Rao 1968):
• Maximum likelihood trying all possible labelings (!)
(the problem of treating unlabeled data is dealing with explosive parameter)

More feasible suggestion (McLachlan 1975):
• Start with supervised solution
• Label unlabeled objects using this classifier
• Retrain classifier treating labels as true labels

Also known as self-training, self-labeling or pseudo-labeling

self-learning $\approx$ EXPECTATION MAXIMIZATION

• Linear Discriminant Analysis (LDA)
  $$p(X,y;\theta )=\prod _{i=1}^L[{\pi }_{0}N(x_i,{\mu }_0,\Sigma ){]}^{1-y_i}[{\pi }_{1}N(x_i,{\mu }_1,\Sigma ){]}^{y_1}$$
  share the covariance $\Sigma$
  $N(x_i,{\mu }_0,\Sigma )$ gaussians for each class
• LDA + unlabeled data

$$\begin{gathered} p(X,y,X_u,h;\theta )=\prod _{i=1}^L[{\pi }_{0}N(x_i,{\mu }_0,\Sigma ){]}^{1-y_i}[{\pi }_{1}N(x_i,{\mu }_1,\Sigma ){]}^{y_1} \\ \times \prod _{i=1}^u[{\pi }_{0}N(x_i,{\mu }_0,\Sigma ){]}^{1-h_i}[{\pi }_{1}N(x_i,{\mu }_1,\Sigma ){]}^{h_1} \end{gathered}$$
But we do not know h… Integrate it out!
$$p(X,y,X_u;\theta )={\int }_{h}p(X,y,X_u,h;\theta )dh$$
LDA +
unlabeled data
$$\begin{gathered} \prod _{i=1}^L[{\pi }_{0}N(x_i,{\mu }_0,\Sigma ){]}^{1-y_i}[{\pi }_{1}N(x_i,{\mu }_1,\Sigma ){]}^{y_1} \\ \times \prod _{i=1}^u\sum _{c=0}^1{\pi }_{c}N(x_i,{\mu }_c,\Sigma ) \end{gathered}$$
Like LDA + a gaussian mixture with the same parameters

EM algorithm

• Log sum makes optimization difficult
• Change goal: find a local maximum of this function

EM algorithm: finding a lower bound

what we want is construct a lower bound and touch exactyl the objective function ,and get the best lower bound which you can get

Jensen’s inequality

If $f(x)$ concave then $f(E[X])\ge E[f(X)]$

Does unlabeled data help?

${\theta }_x\to X$
$X\to Y$
${\theta }_{Y|X}\to Y$

Self-learning and EM conclusions

• For generative models:
• Integrate out the missing variables
• Difficult optimization problem can often be “solved” efficiently using
expectation maximization
• Only guaranteed to improve performance asymptotically, if the model is
correct
• Self-learning is a closely related technique that is applicable to any classifier
• Related: co-training (multi-view learning)
• Use labels predicted by other view(s) as newly labeled objects

Low-density assumption

Low-density assumption conclusion
• “Natural” extension for the SVM
• Local minima may be a problem
• Lots of work on optimization
• My experience: quite sensitive to parameter settings
• Other low-density approaches:
• Entropy Regularization (Bengio & Grandvalet 2005)

manifold assumption

• manifold regularization
  -consistency regularication

$$\Vert f(x;w)-g(x^{\prime };w^t){\Vert }^2$$

Semi-Supervised Conclusion

• Unlabeled data is often available
• Semi-supervised learning attempts to use it to improve classifier
• Often worthwhile, but it does not come for free
• Modeling time
• Computational cost
• Remember: an unlabeled object is less valuable than a labeled one
• Labeling a few more objects can be more effective
• Remember the goal: transductive or inductive?