python 读取文本文件_如何在Python中读取大文本文件

python 读取文本文件

Python File object provides various ways to read a text file. The popular way is to use the readlines() method that returns a list of all the lines in the file. However, it’s not suitable to read a large text file because the whole file content will be loaded into the memory.

Python File对象提供了多种读取文本文件的方法。 流行的方法是使用readlines（）方法，该方法返回文件中所有行的列表。 但是，不适合读取大文本文件，因为整个文件内容都将加载到内存中。

用Python读取大文本文件 (Reading Large Text Files in Python)

We can use the file object as an iterator. The iterator will return each line one by one, which can be processed. This will not read the whole file into memory and it’s suitable to read large files in Python.

我们可以使用文件对象作为迭代器。 迭代器将逐行返回每一行，可以对其进行处理。 这不会将整个文件读入内存，并且适合用Python读取大文件。

Here is the code snippet to read large file in Python by treating it as an iterator.

这是通过将Python视为迭代器来读取大型文件的代码段。

import resource
import os

file_name = "/Users/pankaj/abcdef.txt"

print(f'File Size is {os.stat(file_name).st_size / (1024 * 1024)} MB')

txt_file = open(file_name)

count = 0

for line in txt_file:
    # we can process file line by line here, for simplicity I am taking count of lines
    count += 1

txt_file.close()

print(f'Number of Lines in the file is {count}')

print('Peak Memory Usage =', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
print('User Mode Time =', resource.getrusage(resource.RUSAGE_SELF).ru_utime)
print('System Mode Time =', resource.getrusage(resource.RUSAGE_SELF).ru_stime)

When we run this program, the output produced is:

当我们运行该程序时，产生的输出为：

File Size is 257.4920654296875 MB
Number of Lines in the file is 60000000
Peak Memory Usage = 5840896
User Mode Time = 11.46692
System Mode Time = 0.09655899999999999

Python Read Large Text File

Python读取大文本文件

• I am using os module to print the size of the file.
  我正在使用os模块来打印文件的大小。
• The resource module is used to check the memory and CPU time usage of the program.
  资源模块用于检查程序的内存和CPU时间使用情况。

We can also use with statement to open the file. In this case, we don’t have to explicitly close the file object.

我们还可以使用with语句打开文件。 在这种情况下，我们不必显式关闭文件对象。

with open(file_name) as txt_file:
    for line in txt_file:
        # process the line
        pass

如果大文件没有行怎么办？ (What if the Large File doesn’t have lines?)

The above code will work great when the large file content is divided into many lines. But, if there is a large amount of data in a single line then it will use a lot of memory. In that case, we can read the file content into a buffer and process it.

当将大文件内容分为多行时，以上代码将非常有用。 但是，如果一行中有大量数据，那么它将占用大量内存。 在这种情况下，我们可以将文件内容读入缓冲区并进行处理。

with open(file_name) as f:
    while True:
        data = f.read(1024)
        if not data:
            break
        print(data)

The above code will read file data into a buffer of 1024 bytes. Then we are printing it to the console.

上面的代码会将文件数据读取到1024字节的缓冲区中。 然后我们将其打印到控制台。

When the whole file is read, the data will become empty and the break statement will terminate the while loop.

当读取整个文件时，数据将变为空，并且break语句将终止while循环。

This method is also useful in reading a binary file such as images, PDF, word documents, etc.

此方法在读取二进制文件（例如图像，PDF，Word文档等）时也很有用。

Here is a simple code snippet to make a copy of the file.

这是制作文件副本的简单代码段。

with open(destination_file_name, 'w') as out_file:
    with open(source_file_name) as in_file:
        for line in in_file:
            out_file.write(line)

Reference: StackOverflow Question

参考 ： StackOverflow问题

翻译自: https://www.journaldev.com/32059/read-large-text-files-in-python

python 读取文本文件