java tf值搜索_搜索引擎优化 TF_IDF之Java实现

实现之前，我们要事先说明一些问题：

我们用Redis对数据进行持久化，存两种形式的MAP:

key值为term，value值为含有该term的url

key值为url，value值为map，记录term及在文章中出现的次数

总的计算公式如下：

1.计算词频TF

这里通过给出url地址，获取搜索词term在此url中的数量，计算出TF

获取url中的词汇总数

/**

* @Author Ragty

* @Description 获取url中的词汇总数

* @Date 11:18 2019/6/4

**/

public Integer getWordCount(String url) {

String redisKey = urlSetKey(url);

Map map = jedis.hgetAll(redisKey);

Integer count = 0;

for(Map.Entry entry: map.entrySet()) {

count += Integer.valueOf(entry.getValue());

}

return count;

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

返回搜索项在url中出现的次数

/**

* @Author Ragty

* @Description 返回搜索项在url中出现的次数

* @Date 22:12 2019/5/14

**/

public Integer getTermCount(String url,String term) {

String redisKey = urlSetKey(url);

String count = jedis.hget(redisKey,term);

return new Integer(count);

}

1

2

3

4

5

6

7

8

9

10

获取搜索词的词频

/**

* @Author Ragty

* @Description 获取搜索词的词频(Term Frequency)

* @Date 11:25 2019/6/4

**/

public BigDecimal getTermFrequency(String url,String term) {

if (!isIndexed(url)) {

System.out.println("Doesn‘t indexed.");

return null;

}

Integer documentCount = getWordCount(url);

Integer termCount = getTermCount(url,term);

return documentCount==0 ? new BigDecimal(0) : new BigDecimal(termCount).divide(new BigDecimal(documentCount),6,BigDecimal.ROUND_HALF_UP);

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

2.计算逆文档频率

计算逆文档频率，需要计算文档总数，以及包含该搜索词的文章数

获取redis索引文章的总数

/**

* @Author Ragty

* @Description 获取redis索引文章的总数

* @Date 19:46 2019/6/5

**/

public Integer getUrlCount() {

Integer count = 0;

count = urlSetKeys().size();

return count;

}

1

2

3

4

5

6

7

8

9

10

获取含有搜索词的文章数

/**

* @Author Ragty

* @Description 获取含有搜索词的文章数

* @Date 22:42 2019/6/5

**/

public Integer getUrlTermCount(String term) {

Integer count = 0;

count = getUrls(term).size();

return count;

}

1

2

3

4

5

6

7

8

9

10

计算逆文档频率IDF(InverseDocumnetFrequency)

/**

* @Author Ragty

* @Description 计算逆文档频率IDF(InverseDocumnetFrequency)

* @Date 23:32 2019/6/5

**/

public BigDecimal getInverseDocumentFrequency(String term) {

Integer totalUrl = getUrlCount();

Integer urlTermCount = getUrlTermCount(term);

Double xx = new BigDecimal(totalUrl).divide(new BigDecimal(urlTermCount),6,BigDecimal.ROUND_HALF_UP).doubleValue();

BigDecimal idf = new BigDecimal(Math.log10(xx));

return idf;

}

1

2

3

4

5

6

7

8

9

10

11

12

3.获取TF-IDF

/**

* @Author Ragty

* @Description 获取tf-idf值

* @Date 23:34 2019/6/5

**/

public BigDecimal getTFIDF(String url,String term) {

BigDecimal tf = getTermFrequency(url, term);

BigDecimal idf = getInverseDocumentFrequency(term);

BigDecimal tfidf =tf.multiply(idf);

return tfidf;

}

1

2

3

4

5

6

7

8

9

10

11

4.数据测试

这里我采用我自己爬取的部分数据，进行一下简单的测试(可能因为数据集的原因导致部分结果不准确)

测试类方法：

/**

* @Author Ragty

* @Description 获取tfidf下的相关性

* @Date 8:47 2019/6/6

**/

private static BigDecimal getRelevance(String url,String term,JedisIndex index) {

BigDecimal tfidf = index.getTFIDF(url,term);

return tfidf;

}

/**

* @Author Ragty

* @Description 执行搜索

* @Date 23:49 2019/5/30

**/

public static WikiSearch search(String term,JedisIndex index) {

Map map = new HashMap();

Set urls = index.getUrls(term);

for (String url: urls) {

BigDecimal tfidf = getRelevance(url,term,index).setScale(6,BigDecimal.ROUND_HALF_UP);

map.put(url,tfidf);

}

return new WikiSearch(map);

}

/**

* @Author Ragty

* @Description 按搜索项频率顺序打印内容

* @Date 13:46 2019/5/30

**/

private void print() {

List> entries = sort();

for(Entry entry: entries) {

System.out.println(entry.getKey()+" "+entry.getValue());

}

}

/**

* @Author Ragty

* @Description 根据相关性对数据排序

* @Date 13:54 2019/5/30

**/

public List> sort(){

List> entries = new LinkedList>(map.entrySet());

Comparator> comparator = new Comparator>() {

@Override

public int compare(Entry o1, Entry o2) {

return o2.getValue().compareTo(o1.getValue());

}

};

Collections.sort(entries,comparator);

return entries;

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

测试代码：

public static void main(String[] args) throws IOException {

Jedis jedis = JedisMaker.make();

JedisIndex index = new JedisIndex(jedis);

// search for the first term

String term1 = "java";

System.out.println("Query: " + term1);

WikiSearch search1 = search(term1, index);

search1.print();

// search for the second term

String term2 = "programming";

System.out.println("Query: " + term2);

WikiSearch search2 = search(term2, index);

search2.print();

}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

测试结果：

Query: java

https://baike.baidu.com/item/LiveScript 0.029956

https://baike.baidu.com/item/Java/85979 0.019986

https://baike.baidu.com/item/Brendan%20Eich 0.017188

https://baike.baidu.com/item/%E7%94%B2%E9%AA%A8%E6%96%87/471435 0.013163

https://baike.baidu.com/item/Sun/69463 0.005504

https://baike.baidu.com/item/Rhino 0.004401

https://baike.baidu.com/item/%E6%8E%92%E7%89%88%E5%BC%95%E6%93%8E 0.003452

https://baike.baidu.com/item/javascript 0.002212

https://baike.baidu.com/item/js/10687961 0.002212

https://baike.baidu.com/item/%E6%BA%90%E7%A0%81 0.002205

https://baike.baidu.com/item/%E6%BA%90%E7%A0%81/344212 0.002205

https://baike.baidu.com/item/%E8%84%9A%E6%9C%AC%E8%AF%AD%E8%A8%80 0.001989

https://baike.baidu.com/item/SQL 0.001779

https://baike.baidu.com/item/PHP/9337 0.001503

https://baike.baidu.com/item/iOS/45705 0.001499

https://baike.baidu.com/item/Netscape 0.000863

https://baike.baidu.com/item/%E6%93%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F 0.000835

https://baike.baidu.com/item/Mac%20OS%20X 0.000521

https://baike.baidu.com/item/C%E8%AF%AD%E8%A8%80 0.000318

Query: programming

https://baike.baidu.com/item/C%E8%AF%AD%E8%A8%80 0.004854

https://baike.baidu.com/item/%E8%84%9A%E6%9C%AC%E8%AF%AD%E8%A8%80 0.002529

---------------------

原文：https://www.cnblogs.com/ly570/p/11106215.html