自然语言处理 n-gram处理

自然语言处理 n-gram处理

今天学习网络爬虫时，学习了n-gram模型，其大概思路就是

1. 提取对应文本
2. 对文本进行清洗，保证后续建立文本单词之间联系
3. 对文本进行n-gram建模
4. 清理掉常见的无用字段
5. 对保留的字段，取出现频率高的，从而完成模型建立

一下程序使用的是2-gram模型，通过改变n的取值，可以实现任意n-gram模型

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
import string
import operator
def cleanInput(input):
    input = re.sub('\n+', " ", input).lower()
    input = re.sub('\[[0-9]*\]', "", input)
    input = re.sub(' +', " ", input)
    input = bytes(input, "UTF-8")
    input = input.decode("ascii", "ignore")
    cleanInput = []
    input = input.split(' ')

    for item in input:
        #item.strip(string.punctuation),是找出文中所有标点并移除
        item = item.strip(string.punctuation)
        if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'):
            cleanInput.append(item)
    return cleanInput

def ngrams(input, n):
    input = cleanInput(input)
    output = {}
    for i in range(len(input) - n + 1):
        if not (isCommon(input[i]) or isCommon(input[i+n-1])) :
            ngramTemp = " ".join(input[i:i + n])
            if ngramTemp not in output:
                output[ngramTemp] = 0
            output[ngramTemp] += 1
    return output

 #一下是常见的一些”无用单词“，在筛选时可以剔除
def isCommon(ngram):
    commonWords = ["the", "be", "and", "of", "a", "in", "to", "have", "it",
 "i", "that", "for", "you", "he", "with", "on", "do", "say", "this",
 "they", "is", "an", "at", "but","we", "his", "from", "that", "not",
 "by", "she", "or", "as", "what", "go", "their","can", "who", "get",
 "if", "would", "her", "all", "my", "make", "about", "know", "will",
 "as", "up", "one", "time", "has", "been", "there", "year", "so",
 "think", "when", "which", "them", "some", "me", "people", "take",
 "out", "into", "just", "see", "him", "your", "come", "could", "now",
 "than", "like", "other", "how", "then", "its", "our", "two", "more",
 "these", "want", "way", "look", "first", "also", "new", "because",
 "day", "more", "use", "no", "man", "find", "here", "thing", "give",
 "many", "well"]

    if ngram in commonWords:
        return True
    return False

#从目标网站爬取信息，然后进行处理
content = str(urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(),'utf-8')
ngrams = ngrams(content, 2)
sortedNGrams = sorted(ngrams.items(), key=operator.itemgetter(1), reverse=True)
#print(sortedNGrams)

#选取出现频率大于等于3的词组
for term in sortedNGrams:
    if term[1] >= 3:
        print(term)

最后结果：