NLP---文本前期预处理的几个步骤

1、读取文本

text1 ="""
Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. 
Unqualified, the word football is understood to refer to whichever form of football is the most popular 
in the regional context in which the word appears. Sports commonly called football in certain places 
include association football (known as soccer in some countries); gridiron football (specifically American 
football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); 
and Gaelic football. These different variations of football are known as football codes.
"""
print("原文:\n", text1)

2、去除换行符

text = text1.replace("\n", "")
print("去除原文中的换行符:\n", text)

3、分句

import nltk
sents = nltk.sent_tokenize(text)
print("将文本进行分句:\n", sents)

4、分词

import string
punctuation_tokens = []
for sent in sents:
    for word in nltk.word_tokenize(sent):
        punctuation_tokens.append(word)
print("将每个句子进行分词:\n", punctuation_tokens)

5、过滤标点符号

tokens = []
for word in punctuation_tokens:
    if word not in string.punctuation:
        tokens.append(word)
print("将分词结果去除标点符号:\n", tokens)

6、过滤停用词

from nltk.corpus import stopwords
fltered = [w for w in tokens if w not in stopwords.words("english")]
print("过滤完停用词之后:\n", fltered)

7、剩下有用的单词进行计数

from collections import Counter
count = Counter(fltered)
print("对最终清洗好的单词进行计数:\n", count)
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

饿了就干饭

你的鼓励将是我创作的最大动力~

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值