python怎么读单词_字符串-在python中读取文本文件并将其拆分为单个单词

字符串-在python中读取文本文件并将其拆分为单个单词

我有一个由数字和单词组成的文本文件，例如09807754 18 n 03 aristocrat 0 blue_blood 0 patrician，我想将其拆分，以便每个单词或数字都以新行出现。

空格分隔符将是理想的，因为我希望带有破折号的单词保持连接。

这是我到目前为止所拥有的：

f = open('words.txt', 'r')

for word in f:

print(word)

不太确定如何从这里开始，我希望将其作为输出：

09807754

18

n

3

aristocrat

...

Johnnerz asked 2020-02-17T11:20:51Z

6个解决方案

122 votes

如果您的数据没有引号，并且一次只想要一个字（忽略文件中空格和换行符的含义）：

with open('words.txt','r') as f:

for line in f:

for word in line.split():

print(word)

如果要在文件的每一行中嵌套单词列表（例如，从文件创建行和列的矩阵）：

with open("words.txt") as f:

[line.split() for line in f]

或者，如果要将文件展平为文件中单个单词的平坦列表，则可以执行以下操作：

with open('words.txt') as f:

[word for line in f for word in line.split()]

如果您想使用正则表达式解决方案：

import re

with open("words.txt") as f:

for line in f:

for word in re.findall(r'\w+', line):

# word by word

或者，如果您希望将其作为带有正则表达式的逐行生成器：

with open("words.txt") as f:

(word for line in f for word in re.findall(r'\w+', line))

dawg answered 2020-02-17T11:21:22Z

18 votes

f = open('words.txt')

for word in f.read().split():

print(word)

dugres answered 2020-02-17T11:21:38Z

11 votes

作为补充，如果您正在读取vvvvery大文件，并且不想一次将所有内容读入内存，则可以考虑使用缓冲区，然后按yield返回每个单词：

def read_words(inputfile):

with open(inputfile, 'r') as f:

while True:

buf = f.read(10240)

if not buf:

break

# make sure we end on a space (word boundary)

while not str.isspace(buf[-1]):

ch = f.read(1)

if not ch:

break

buf += ch

words = buf.split()

for word in words:

yield word

yield '' #handle the scene that the file is empty

if __name__ == "__main__":

for word in read_words('./very_large_file.txt'):

process(word)

pambda answered 2020-02-17T11:21:58Z

3 votes

您可以做的是使用nltk标记单词，然后将所有单词存储在列表中，这就是我所做的。如果您不懂nltk， 它代表自然语言工具包，用于处理自然语言。 如果您想开始，这里有一些资源[[http://www.nltk.org/book/]]

import nltk

from nltk.tokenize import word_tokenize

file = open("abc.txt",newline='')

result = file.read()

words = word_tokenize(result)

for i in words:

print(i)

输出将是这样的：

09807754

18

n

03

aristocrat

0

blue_blood

0

patrician

Gaurav answered 2020-02-17T11:22:23Z

1 votes

with open(filename) as file:

words = file.read().split()

它是文件中所有单词的列表。

import re

with open(filename) as file:

words = re.findall(r"([a-zA-Z\-]+)", file.read())

mujad answered 2020-02-17T11:22:42Z

0 votes

这是我完全实用的方法，可以避免读取和拆分行。 它使用itertools.imap模块：

python 3的注释，将itertools.imap替换为map

import itertools

def readwords(mfile):

byte_stream = itertools.groupby(

itertools.takewhile(lambda c: bool(c),

itertools.imap(mfile.read,

itertools.repeat(1))), str.isspace)

return ("".join(group) for pred, group in byte_stream if not pred)

用法示例：

>>> import sys

>>> for w in readwords(sys.stdin):

... print (w)

...

I really love this new method of reading words in python

I

really

love

this

new

method

of

reading

words

in

python

It's soo very Functional!

It's

soo

very

Functional!

>>>

我想在您的情况下，这将是使用该功能的方式：

with open('words.txt', 'r') as f:

for word in readwords(f):

print(word)

smac89 answered 2020-02-17T11:23:16Z