Xue & Shen '2003 [2]用两种序列标注模型——MEMM (Maximum Entropy Markov Model)与CRF (Conditional Random Field)——用于中文分词;看原论文感觉作者更像用的是MaxEnt (Maximum Entropy) 模型而非MEMM。MEMM是由McCallum et al. '2000 [1]提出MEMM,针对于HMM的两个痛点:一是其为生成模型(generative model),二是不能使用更加复杂的feature。
1. 前言
首先,将简要地介绍HMM与MaxEnt模型。
HMM
概率图模型(probabilistic graphical model, PGM)指用图表示变量相关(依赖)关系的概率模型,主要分为两类:
- 有向图模型或贝叶斯网(Bayesian network),使用有向图表示变量间的依赖关系;
- 无向图模型或马尔可夫网(Markov network),使用无向图表示变量间相关关系。
监督学习的任务就是学习一个模型,对于给定的输入
![](https://i-blog.csdnimg.cn/blog_migrate/b61b9c16ee9b9e8a7928d4b4b2ded45b.png)
。所学习到的模型一般可表示为决策函数:
![](https://i-blog.csdnimg.cn/blog_migrate/b61b9c16ee9b9e8a7928d4b4b2ded45b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/329d43eb1e413fba21045bd9c755127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a64514c9553b7a3820b79f568d22ca5c.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e2918013f651930d37d12461f46ef48e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
或者为条件概率
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/760c8c5d82982babbf450a8f943b127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8e719c16b8806be7e85a74c87dfeac76.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a41d925578911827223e58aa2d055639.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0a6f93171a2add28afda019bd506c0d8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/42c41242a72b28e783c2b2c9771534fd.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b61b9c16ee9b9e8a7928d4b4b2ded45b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a64514c9553b7a3820b79f568d22ca5c.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/14dfee35b05e4e5a4f35539d54abc9a9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
监督学习的模型分为生成模型(generative model)与判别模型(discriminative model)。生成模型学习联合概率分布
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/14dfee35b05e4e5a4f35539d54abc9a9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e2918013f651930d37d12461f46ef48e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/14dfee35b05e4e5a4f35539d54abc9a9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
。HMM属于生成模型的有向图PGM,通过联合概率建模:
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3e6bae7341bd271148e16f227f78a805.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/013ae5f3e874e22cb1d47bf2ce75e478.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a59fa215aafd687ab6d6cd76526f694e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a9fb567af23e1218c40f38e071be14d5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/62edeb113aac23e86a5b0f4224420d35.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0d148ab3aa96ade2103826300c28c7b5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
其中,
![](https://i-blog.csdnimg.cn/blog_migrate/013ae5f3e874e22cb1d47bf2ce75e478.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/760c8c5d82982babbf450a8f943b127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8e719c16b8806be7e85a74c87dfeac76.png)
![](https://i-blog.csdnimg.cn/blog_migrate/a41d925578911827223e58aa2d055639.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0a6f93171a2add28afda019bd506c0d8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/516c2b6f3d59c18d9ddd1bdc947ee0f0.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3e6bae7341bd271148e16f227f78a805.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/013ae5f3e874e22cb1d47bf2ce75e478.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/df17f08a61522e75bb7b78c6cac59d5d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
中的概率最大值为
![](https://i-blog.csdnimg.cn/blog_migrate/81a0c87fc79f25a199c65979bd5e3752.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ca5819d9b75fb063080a55d62772bd52.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6a5d7587c2bba2ff64838bfdf0b67e37.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0d148ab3aa96ade2103826300c28c7b5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
则有
![](https://i-blog.csdnimg.cn/blog_migrate/81a0c87fc79f25a199c65979bd5e3752.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ca5819d9b75fb063080a55d62772bd52.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6a5d7587c2bba2ff64838bfdf0b67e37.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ca5819d9b75fb063080a55d62772bd52.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6a5d7587c2bba2ff64838bfdf0b67e37.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5286ebe565175551791f9a16c30c4925.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b18338da32ca67b0323206111daefb81.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0d148ab3aa96ade2103826300c28c7b5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ca5819d9b75fb063080a55d62772bd52.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6a5d7587c2bba2ff64838bfdf0b67e37.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5286ebe565175551791f9a16c30c4925.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b18338da32ca67b0323206111daefb81.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6bd562a8a8a936339a0a125e6b4c0c53.png)
![](https://i-blog.csdnimg.cn/blog_migrate/81a0c87fc79f25a199c65979bd5e3752.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b283dda68754c3de4bff12d9bae674a8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
上述式子即为(用于解决HMM的解码问题的)Viterbi算法的递推式;可以看出HMM是通过联合概率来求解标注问题的。
最大熵模型
最大熵(Maximum Entropy)模型属于log-linear model,在给定训练数据的条件下对模型进行极大似然估计或正则化极大似然估计:
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc7cc9a8ecfef1dad09de2a597ad522a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6ed70225f2c1f1f23a2943d0af421ad2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b0ac74f085ee06a14e57fc92d7717a71.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/32b872421c39024c9af6ba7daa73f28a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6954d1694058246cc9538d3b056ae990.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e4cd2565d50b380fe5c050d3f1714591.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5dd6ad9677a32c1c974a5401b7307d2e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/1bf1b29f2cbe7158ef1bbfcdf0d6bb10.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5dd6ad9677a32c1c974a5401b7307d2e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/329d43eb1e413fba21045bd9c755127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5dd6ad9677a32c1c974a5401b7307d2e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6ed70225f2c1f1f23a2943d0af421ad2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5c96327b4d96a949acc6cd5c3b9f1407.png)
![](https://i-blog.csdnimg.cn/blog_migrate/569bd496fca6b3ff114fccccebfb42f8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc7cc9a8ecfef1dad09de2a597ad522a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8abf2272e4fb5fbcabd85544aab70113.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
其中,
![](https://i-blog.csdnimg.cn/blog_migrate/1bf1b29f2cbe7158ef1bbfcdf0d6bb10.png)
![](https://i-blog.csdnimg.cn/blog_migrate/329d43eb1e413fba21045bd9c755127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5dd6ad9677a32c1c974a5401b7307d2e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6ed70225f2c1f1f23a2943d0af421ad2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6ed70225f2c1f1f23a2943d0af421ad2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
的某一事实。
最大熵模型并没有假定feature相互独立,允许用户根据domain knowledge设计feature。
2. MEMM
MEMM并没有像HMM通过联合概率建模,而是直接学习条件概率
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0d148ab3aa96ade2103826300c28c7b5.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/7bfc97dbbdccbcac5621ef7608906afd.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
因此,有别于HMM,MEMM的当前状态依赖于前一状态与当前观测;HMM与MEMM的图模型如下(图来自于[3]):
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/7bfc97dbbdccbcac5621ef7608906afd.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/8abf2272e4fb5fbcabd85544aab70113.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
则有:
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b0ac74f085ee06a14e57fc92d7717a71.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3a13c9aa1df53226801f4440689d22b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/32b872421c39024c9af6ba7daa73f28a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6954d1694058246cc9538d3b056ae990.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e4cd2565d50b380fe5c050d3f1714591.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b1c1182f75ff3c2693ddd85196ee7b2d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6561dda2f0dced00462a2eb0542de1a4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b1c1182f75ff3c2693ddd85196ee7b2d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/329d43eb1e413fba21045bd9c755127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b1c1182f75ff3c2693ddd85196ee7b2d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5c96327b4d96a949acc6cd5c3b9f1407.png)
![](https://i-blog.csdnimg.cn/blog_migrate/569bd496fca6b3ff114fccccebfb42f8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/d026b0eee8e48fd389eb8af6e4813405.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
其中,
![](https://i-blog.csdnimg.cn/blog_migrate/0a6f93171a2add28afda019bd506c0d8.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/dd714c590810363d802a32d32ab0ef71.png)
![](https://i-blog.csdnimg.cn/blog_migrate/46a7d2fffb622460eb528cb0993dca8e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/031e5a6a79067599b8f2efa83b810a4a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/46a7d2fffb622460eb528cb0993dca8e.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/329d43eb1e413fba21045bd9c755127f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b1c1182f75ff3c2693ddd85196ee7b2d.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
的示例如下(图出自于[6]):
类似于HMM,MEMM的解码问题的递推式:
![](https://i-blog.csdnimg.cn/blog_migrate/81a0c87fc79f25a199c65979bd5e3752.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/40827a6398fcec6b551562eca003ceb9.png)
![](https://i-blog.csdnimg.cn/blog_migrate/ca5819d9b75fb063080a55d62772bd52.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bdb34b9f7e952c2ff426b8a865d932a2.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6a5d7587c2bba2ff64838bfdf0b67e37.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5286ebe565175551791f9a16c30c4925.png)
![](https://i-blog.csdnimg.cn/blog_migrate/b18338da32ca67b0323206111daefb81.png)
![](https://i-blog.csdnimg.cn/blog_migrate/81a0c87fc79f25a199c65979bd5e3752.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/57fb68d31134593780e5c39ebeb66ec7.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ea9a2d81b5b44bd195fe6c3dc5e24b4.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bbd8a060df0534aaa474a54c90dab235.png)
![](https://i-blog.csdnimg.cn/blog_migrate/20b8f7fe76a652aa611d756382773389.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9b93092ecc7e0ed069f1e8deee105e18.png)
![](https://i-blog.csdnimg.cn/blog_migrate/4705aa4ed4b67dc207476fabd1c4cc78.png)
![](https://i-blog.csdnimg.cn/blog_migrate/02f1969cdfc33d065bf472d117b5160a.png)
![](https://i-blog.csdnimg.cn/blog_migrate/e8585e117a543176ae8387b853b940bb.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bc892b316a31902c1d30b96f3d9e685f.png)
![](https://i-blog.csdnimg.cn/blog_migrate/5afff27d9a807ba051985c56928d4f50.png)
![](https://i-blog.csdnimg.cn/blog_migrate/9ed0afb572bef457b579d4cf61879e9b.png)
但是,MEMM存在着标注偏置问题(label bias problem)。比如,有如下的概率分布(图来自于[7]):
根据上述递推式,则概率最大路径如下:
但是,从全局的角度分析:
- 无论观测值,State 1 总是更倾向于转移到State 2;
- 无论观测值,State 2 总是更倾向于转移到State 2.
可以看出MEMM所做的是本地归一化,导致有更少转移的状态拥有的转移概率普遍偏高,概率最大路径更容易出现转移少的状态。因MEMM存在着标注偏置问题,故全局归一化的CRF被提了出来[3]。欲知CRF如何,请看下一篇分解。
3. 参考资料
[1] McCallum, Andrew, Dayne Freitag, and Fernando CN Pereira. "Maximum Entropy Markov Models for Information Extraction and Segmentation." Icml. Vol. 17. 2000.
[2] Xue, Nianwen, and Libin Shen. "Chinese word segmentation as LMR tagging." Proceedings of the second SIGHAN workshop on Chinese language processing-Volume 17. Association for Computational Linguistics, 2003.
[3] Lafferty, John, Andrew McCallum, and Fernando Pereira. "Conditional random fields: Probabilistic models for segmenting and labeling sequence data." Proceedings of the eighteenth international conference on machine learning, ICML. Vol. 1. 2001.
[4] 李航,《统计学习方法》.
[5] 周志华,《机器学习》.
[6] Nikos Karampatziakis, Maximum Entropy Markov Models.
[7] Ramesh Nallapati, Conditional Random Fields.