nltk中文分句_如何改进NLTK的分句技术？

最新推荐文章于 2024-04-22 02:39:54 发布

小麦的天空

最新推荐文章于 2024-04-22 02:39:54 发布

阅读量347

点赞数

文章标签： nltk中文分句

本文链接：https://blog.csdn.net/weixin_35918198/article/details/113957617

版权

本文介绍了如何利用nltk的PunktSentenceTokenizer无监督算法，针对新的文本进行训练，以改善英文句子分割。通过示例展示了训练过程，并且提到了训练后的模型能够自动学习缩写类型。

摘要由CSDN通过智能技术生成

Kiss和Strunk(2006)Punkt算法的可怕之处在于它是无监督的。所以给一个新的文本，你应该重新训练这个模型并将它应用到你的文本中，例如>>> from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters

>>> text = "An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952. Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law. It was during the tenure of F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action."

# Training a new model with the text.

>>> tokenizer = PunktSentenceTokenizer()

>>> tokenizer.train(text)

# It automatically learns the abbreviations.

>>> tokenizer._params.abbrev_types

{'f', 'fr', 'j'}

# Use the customized tokenizer.

>>> tokenizer.tokenize(text)

['An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952.', 'Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law.', "It was during the tenure of F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action."]