Python 文本词频统计

Hamlet文本词频统计

The Tragedy of Hamlet, Prince of Denmark
Shakespeare homepage | Hamlet | Entire play
ACT I

SCENE I. Elsinore. A platform before the castle.

FRANCISCO at his post. Enter to him BERNARDO
BERNARDO
Who's there?
FRANCISCO
Nay, answer me: stand, and unfold yourself.
BERNARDO
Long live the king!
FRANCISCO
Bernardo?
BERNARDO
He.
FRANCISCO
You come most carefully upon your hour.
...
...
...

代码：

def getText():
    txt = open("hamlet.txt", "r").read()
    txt = txt.lower()
    for ch in '!"#$%&()*+,-./;:<=>?@[\\]^‘_{|}~':
        txt = txt.replace(ch, " ")
    return txt

hamletTxt = getText()
words = hamletTxt.split()
counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1
items = list(counts.items())
items.sort(key=lambda x:x[1], reverse=True)
for i in range(10):
    word, count = items[i]
    print("{0:<10}{1:>5}".format(word, count))

输出频次最高的前十个单词：

the        1138
and         965
to          754
of          669
you         550
i           542
a           542
my          514
hamlet      462
in          436

输出三国演义出现次数最多的十个词汇

代码：

# -*- coding: UTF-8 -*-
import jieba
txt = open("threekingdoms.txt", "r", encoding="utf-8").read()
words = jieba.lcut(txt)
counts = {}
for word in words:
    if len(word) == 1:
        continue
    else:
        counts[word] = counts.get(word, 0) + 1
items = list(counts.items())
items.sort(key=lambda x:x[1], reverse=True)
for i in range(15):
    word, count = items[i]
    print("{0:<10}{1:<5}".format(word, count))

输出：

曹操        953  
孔明        836  
将军        772  
却说        656  
玄德        585  
关公        510  
丞相        491  
二人        469  
不可        440  
荆州        425  
玄德曰       390  
孔明曰       390  
不能        384  
如此        378  
张飞        358  

通过测试，统计三国演义出现次数最多的前十个人物名称

代码：

# -*- coding: UTF-8 -*-
import jieba
txt = open("threekingdoms.txt", "r", encoding="utf-8").read()
excludes = {"将军", "却说", "荆州", "二人", "不可", "不能", "如此", "主公",\
            "军士", "商议", "如何", "左右", "军马", "引兵", "次日", "大喜",\
            "天下", "东吴", "于是", "今日", "不敢", "魏兵", "陛下", "一人",\
            "都督", "人马", "不知"}
words = jieba.lcut(txt)
counts = {}
for word in words:
    if len(word) == 1:
        continue
    elif word == "诸葛亮" or word == "孔明曰":
        rword = "孔明"
    elif word == "关公" or word == "云长":
        rword = "关羽"
    elif word == "玄德" or word == "玄德曰":
        rword = "刘备"
    elif word == "孟德" or word == "丞相":
        rword = "曹操"
    else:
        rword = word
    counts[rword] = counts.get(rword, 0) + 1
for word in excludes:
    del counts[word]
items = list(counts.items())
items.sort(key=lambda x:x[1], reverse=True)
for i in range(10):
    word, count = items[i]
    print("{0:<10}{1:<5}".format(word, count))

输出：

曹操        1451 
孔明        1383 
刘备        1252 
关羽        784  
张飞        358  
吕布        300  
赵云        278  
孙权        264  
司马懿       221  
周瑜        217