python读取数据时出现错误_Python Pandas：错误标记数据 . C错误：读取1GB CSV文件时字符串内的EOF...

我正在读取10,000行的1 GB CSV文件 . 该文件有1106012行和171列，其他较小的文件没有显示任何错误并成功完成但是当我读取这个1 GB文件时，它每次都在正确的行号1106011上显示错误，这是第二行文件，我可以手动删除该行，但这不是解决方案，因为我有数百个相同大小的其他文件，我无法手动修复所有行 . 请有人帮我 .

def extract_csv_to_sql(input_file_name, header_row, size_of_chunk, eachRow):

df = pd.read_csv(input_file_name,

header=None,

nrows=size_of_chunk,

skiprows=eachRow,

low_memory=False,

error_bad_lines=False,

sep=',')

# engine='python'

# quoting=csv.QUOTE_NONE

# encoding='utf-8'

df.columns = header_row

df = df.drop_duplicates(keep='first')

df = df.apply(lambda x: x.astype(str).str.lower())

return df

然后我在循环中调用此函数并且工作得很好 .

huge_chunk_return = extract_csv_to_sql(huge_input_filename, huge_header_row, the_size_of_chunk_H, each_Row_H)

engine ='python' quoting = csv.QUOTE_NONE //挂起甚至是python shell，不知道为什么encoding ='utf-8'

但没有一个工作，它仍然抛出以下错误

Error:

Traceback (most recent call last):

File "C:\Users\WCan\Desktop\wcan_new_python\pandas_test_3.py", line 115, in

huge_chunk_return = extract_csv_to_sql(huge_input_filename, huge_header_row, the_size_of_chunk_H, each_Row_H)

File "C:\Users\WCan\Desktop\wcan_new_python\pandas_test_3.py", line 24, in extract_csv_to_sql

sep=',')

File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f

return _read(filepath_or_buffer, kwds)

File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 411, in _read

data = parser.read(nrows)

File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 1005, in read

ret = self._engine.read(nrows)

File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 1748, in read

data = self._reader.read(nrows)

File "pandas\_libs\parsers.pyx", line 893, in pandas._libs.parsers.TextReader.read (pandas\_libs\parsers.c:10885)

File "pandas\_libs\parsers.pyx", line 966, in pandas._libs.parsers.TextReader._read_rows (pandas\_libs\parsers.c:11884)

File "pandas\_libs\parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas\_libs\parsers.c:11755)

File "pandas\_libs\parsers.pyx", line 2184, in pandas._libs.parsers.raise_parser_error (pandas\_libs\parsers.c:28765)

pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at line 1106011

>>>