python下一行减上一行,阅读文件python中的上一行

本文介绍了解决在处理大型文件时,如何在内存限制下逐行读取并比较前后行,避免一次性加载所有内容。方法包括分块存储文件位置和逐行读取策略,以减少内存消耗。阅读者将学到如何实现从后向前读取文件,同时保持前一行的值以便于对比。
摘要由CSDN通过智能技术生成

I need to get the value of the previous line in a file and compare it with the current line as I'm iterating through the file. The file is HUGE so I can't read it whole or randomly accessing a line number with linecache because the library function still reads the whole file into memory anyway.

EDIT I'm so sorry I forgot the mention that I have to read the file backwardly.

EDIT2

I have tried the following:

f = open("filename", "r")

for line in reversed(f.readlines()): # this doesn't work because there are too many lines to read into memory

line = linecache.getline("filename", num_line) # this also doesn't work due to the same problem above.

解决方案

Just save the previous when you iterate to the next

prevLine = ""

for line in file:

# do some work here

prevLine = line

This will store the previous line in prevLine while you are looping

edit apparently OP needs to read this file backwards:

aaand after like an hour of research I failed multiple times to do it within memory constraints

Here you go Lim, that guy knows what he's doing, here is his best Idea:

General approach #2: Read the entire file, store position of lines

With this approach, you also read through the entire file once, but

instead of storing the entire file (all the text) in memory, you only

store the binary positions inside the file where each line started.

You can store these positions in a similar data structure as the one

storing the lines in the first approach.

Whever you want to read line X, you have to re-read the line from the

file, starting at the position you stored for the start of that line.

Pros: Almost as easy to implement as the first approach Cons: can take

a while to read large files

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值