Noise Contrastive Estimation

链接：http://www.cnblogs.com/ZJUT-jiangnan/p/5934647.html

Noise Contrastive Estimation
Notes from Notes on Noise Contrastive Estimation and Negative Sampling
one sample:
xi→[y0i,⋯,yki]

where y0i are true labeled words , and y1i,⋯,yki are noise samples word index, which is generated by unigram distribution q(w) of the dataset.
the probability of true data:
p(y0i=1|xi,θ)=exp(y0i,hθ)exp(y0ihθ)+k∗q(y0i)

the noise sample probability:
p(yti=0|xi,θ)=k∗q(yti)exp(ytihθ)+k∗q(yti),t=1,⋯,k

the cost function of this sample:
lnce=logp(y0i|xi,θ)+∑t=1klogp(yti|xi,θ)

the overall cost function of the dataset:
nce=1N∑iN{logp(y0i|xi,θ)+∑t=1klogp(yti|xi,θ)}

Related Paper
[Noise-Contrastive Estimation of Unnormalized Statistical Models with Applications to Natural Image Statistics]

[Word2vec Parameter Learning Explained]

[Efficient Estimation of Word Representation in Vector Space]

[Distributed Representations of Words and Phrases and their Compositionality]

[Notes on Noise Contrastive Estimation and Negative Sampling]

好文要顶 关注我 收藏该文
姜楠
关注 - 80
粉丝 - 59
+加关注
0 0
« 上一篇：vector - vector product
» 下一篇：theano .dimshuffle
posted @ 2016-10-06 19:59 姜楠 阅读(1440) 评论(0) 编辑 收藏