【python】读取大规模训练集

可以先遍历一遍训练集，记录下每行训练数据的开始位置；读取训练集时，临时打开文件，定位到开始位置，读一行。

outList = ["早安", "希望世界和平", "快乐开心每一天"]

with open("tmp.txt", "w", encoding="utf8") as fout:
    for line in outList:
        fout.write(line+"\n")

with open("tmp.txt", "r", encoding="utf8") as fin:
    pos = [fin.tell()]
    while True:
        line = fin.readline()
        if not line:
            break
        pos.append(fin.tell())

with open("tmp.txt", "r", encoding="utf8") as fin:
    for p in pos:
        fin.seek(p)
        line = fin.readline()
        print(line)

运行结果：

早安

希望世界和平

快乐开心每一天