文本相似度指标-基于词汇的相似度量

Jaccard相似度

$J(A,B)$表示有限样本集之间的相似程度：

$J(A,B)=\frac{|A\cap B|}{|A\cup B|}=\frac{|A\cap B|}{|A|+|B|-|A\cap B|}$

Jaccard相似度:

$d_j(A,B)=1-J(A,B)=\frac{|A\cup B|-|A\cap B|}{|A\cup B|}=\frac{A\Delta B}{|A\cup B|}$

当A=B时，Jaccard相似度为1；当|A∩B|=0时，Jaccard相似度为0.

Jaccard相似度的取值范围为[0,1]，值越大表示越相似。代码如下：

def Jaccard(words1, words2):
    words1_cut, words2_cut = set(jieba.cut(words1)), set(jieba.cut(words2))
    interNum = 0
    for word in words1_cut:
        if word in words2_cut:
            interNum += 1
    return float(interNum/(len(set(words1_cut))+len(set(words2_cut))-interNum))

余弦相似度

$cos(X,Y)=\frac{X\cdot Y}{|X||Y|}$

Dice系数

$s=2\frac{|A\cap B|}{|A|+|B|}$

def Dice(words1, words2):
    words1_cut, words2_cut = set(jieba.cut(words1)), set(jieba.cut(words2))
    interNum = 0
    for word in words1_cut:
        if word in words2_cut:
            interNum += 1
    return float(2*interNum/(len(set(words1_cut))+len(set(words2_cut))))

匹配系数

$overlap(X,Y)=\frac{|X\cap Y|}{min(|X|,|Y|)}$

def overlap(words1, words2):
    words1_cut, words2_cut = set(jieba.cut(words1)), set(jieba.cut(words2))
    interNum = 0
    for word in words1_cut:
        if word in words2_cut:
            interNum += 1
    return float(2*interNum/min(len(set(words1_cut)),len(set(words2_cut))))