Generative classfiers And Discrimivative

1.生成模型是学习输入x和标签y的联合概率分布，p(x,y).然后通过贝叶斯规则就算p(y|x)来做预测。

2.判别模型直接计算后验概率p(y|x).

In both the discrete and continuous cases, it is well known that the discriminative analog of naive Bayes is logistic regression. This model has parameters [Beta, Theta] and posits that

Given a test example x, the discriminative logistic regression classifier Hdis: X——y predicts Hdis(x)=T if and only if the linear discriminant function

is positive. Being a discriminative model, the parameters [Beta, Theta] can be fit either to maximize the conditional likelikood on the training set .

or minimize 0-1 traning error 

The error of the discriminative logistic regression is smaller than that of the generative naive Bayes. 

The discriminative logistic regression algorithm has a lower asymptotic error, the generative naive Bayes classifier may also converge more quickly to its (higer) asymptotic error. Thus, as the number of training examples m is increased, one would expect generative naive Bayes to initially do better, but for discriminative logistic regression to eventually catch up to, and quite likely overtake, the performance of naive Bayes.

=============================================================================

产生式模型（Generative）：构建o和s的联合分布p(o,s)，如HMM，BNs，MRF

产生式模型：无穷样本===》概率密度模型=产生模型===》预测

判别式模型（Discriminative）：构建o和s的条件分布p(s|o)，如SVM，CRF，MEMM

判别式模型：有限样本===》判别函数=预测模型===》预测

两种模型的比较：

Generative Model：从统计的角度表示数据的分布情况，能够反映同类数据本身的相似度，不关心判别边界

优点：

实际上带的信息比判别模型丰富，研究单类问题比判别模型灵活性强

能更充分利用先验知识

模型可以通过增量学习得到

缺点：

学习过程比较复杂

在目标分类问题中容易产生较大的错误率

Discriminative Model：寻找不同类别之间的最优分类面，反映的是异类数据之间的差异

优点：

分类边界更灵活，比使用纯概率方法或产生模型得到的更高级

能清晰的分辨出多类或某一类与其他类之间的差异特征

适用于较多类别的识别

缺点：

不能反映训练数据本身的特性

两者关系：

由产生式模型可以得到判别式模型，但有判别式模型得不到产生式模型。