Building prefix dictfrom the default dictionary ...
Loading model from cache
Default mode:我 在 学习 自然语言 处理
All mode:我 在 学习 自然 自然语言 语言 处理
Loading model cost 0.578 seconds.
Prefix dict has been built successfully.
import jieba
seg_list = jieba.lcut_for_search("如果放到旧字典中将会出错")print(seg_list)
Building prefix dictfrom the default dictionary ...
Loading model from cache
['如果','放到','旧','字典','中将','会','出错']
Loading model cost 0.590 seconds.
Prefix dict has been built successfully.
这种情况下在进行词语分割时容易混淆词语,实际“中”和“将”分
别是两个不同的词。为此可以用suggest_freq(segment,
tune=False)方法
Parameter:- segment : The segments that the word is expected
to be cut into.If the word should be treated as a
whole,use a str.- tune : If True, tune the word frequency.
对于参数tune,如果设置True,我们就增大segment中词的出现
频率。
import jieba
jieba.suggest_freq(('中','将'),True)
seg_list = jieba.lcut_for_search("如果放到旧字典中将会出错")print(seg_list)
Building prefix dictfrom the default dictionary ...
Loading model from cache
Loading model cost 0.569 seconds.
Prefix dict has been built successfully.['如果','放到','旧','字典','中','将','会','出错']
jieba.analyse.extract_tags(sentence,
topK=20, withWeight=False, allowPOS=(),
withFlag=False) 函数可以用来做关键词提取,topK用来设定
函数返回前topK个出现频率最高的词语,allowPOS()用来指定
词性(参数可传'ns','n','vn','v','nr',分别代表“地名”
“名词”“名动词”“动词”“人名”,除了上述五个词性以外还有很多
这里不一一列举了),withWeight可以用来设置是否返回词语的
权重,如果为True,则最后返回元组形式,即(词语,权重)的元组。
import jieba.analyse as analyse
lines =open("白夜行.txt").read()
seg_list = analyse.extract_tags(lines,topK=20,withWeight=False,allowPOS=())print(seg_list)
Building prefix dictfrom the default dictionary ...
Loading model from cache
Loading model cost 0.657 seconds.
Prefix dict has been built successfully.['笹垣','桐原','雪穗','今枝','友彦','利子','什么','没有','典子','知道','男子','唐泽雪穗','警察','菊池','筱冢','一成','这么','松浦','不是','千都']import jieba.analyse as analyse
lines =open("白夜行.txt").read()
seg_list = analyse.extract_tags(lines,topK=20,withWeight=True,allowPOS=())print(seg_list)
Building prefix dictfrom the default dictionary ...
Loading model from cache
Loading model cost 0.759 seconds.
Prefix dict has been built successfully.[('笹垣',0.0822914672578027),('桐原',0.07917333917389914),('雪穗',0.07619078187625225),('今枝',0.060328999884221086),('友彦',0.05992228752545106),('利子',0.041188814819915855),('什么',0.028355297812861044),('没有',0.0282050104733996),('典子',0.025758449388768555),('知道',0.021181785348317664),('男子',0.021159435305523867),('唐泽雪穗',0.01992890557973146),('警察',0.018198774503613253),('菊池',0.01816537645564464),('筱冢',0.01803091457213799),('一成',0.01796642218172475),('这么',0.016991657412780303),('松浦',0.016132923564544516),('不是',0.015944699687736586),('千都',0.015726211205774485)]
我们也可以加入停用词,从而避免切割出来的词语包含一些无用
词,例如“的”“了”“什么”“这么”“是”“不是”等。
import jieba.analyse as analyse
lines =open("白夜行.txt").read()
analyse.set_stop_words("stopwords.txt")
seg_list = analyse.extract_tags(lines,topK=20,withWeight=False,allowPOS=())print(seg_list)
Building prefix dictfrom the default dictionary ...
Loading model from cache
Loading model cost 0.743 seconds.
Prefix dict has been built successfully.['笹垣','桐原','雪穗','今枝','友彦','利子','典子','唐泽雪穗','警察','菊池','筱冢','一成','松浦','千都','高宫','奈美江','正晴','美佳','雄一','康晴']
set_stop_words(stop_words_path)函数传入停用词文件路径
最终结果显然不包含像“什么”“没有”“知道”“不是”等非关键性词语。
jieba.analyse.textrank(sentence, topK=20, withWeight=False, allowPOS=('ns','n','vn','v'), withFlag=False)
参数同以上几个函数,textrank算法的基本思想:
(1)将待切割的文本进行分词操作
(2)按照词语之间的关系构建无向带权图(图论的相关知识)(3)计算图中结点的PageRank。
import jieba.analyse as analyse
lines =open("白夜行.txt").read()
analyse.set_stop_words("stopwords.txt")
seg_list = analyse.textrank(lines,topK=20,withWeight=True,allowPOS=('n','ns','vn','v'))for each in seg_list:print("Key words:"+each[0]+"\t"+"weights:"+str(each[1]))
Building prefix dictfrom the default dictionary ...
Loading model from cache
Loading model cost 0.586 seconds.
Prefix dict has been built successfully.
Key words:桐原 weights:1.0
Key words:可能 weights:0.42444673809670785
Key words:利子 weights:0.36847047979727526
Key words:应该 weights:0.3365721550231226
Key words:时候 weights:0.3278732402186668
Key words:警察 weights:0.3120355440367427
Key words:东西 weights:0.3068897401798211
Key words:开始 weights:0.3015519959941887
Key words:调查 weights:0.29838940592155194
Key words:典子 weights:0.29588671242198666
Key words:公司 weights:0.2939855813808517
Key words:电话 weights:0.27845709742538927
Key words:不会 weights:0.27350278630982
Key words:看到 weights:0.27028179300492206
Key words:发现 weights:0.2681890733271942
Key words:房间 weights:0.2672507877051219
Key words:工作 weights:0.2661521099652389
Key words:声音 weights:0.24633054809460497
Key words:露出 weights:0.22866657032979934
Key words:认为 weights:0.21634682021764443