Pandas read_csv导致pandas.errors.ParserError: Error tokenizing data

What you will learn？

• pandas read_csv escape commas and double qoutes

Prepare datas

# test.csv or test.txt
"1","123","4","\"data\""
"test","123","4","if(\"data\" = \"<test>\", (10*24))"

Wrong way

import pandas as pd

datas = pd.read_csv('test.txt', header=None, skip_blank_lines=True)

You got

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._tokenize_rows()

pandas\_libs\parsers.pyx in pandas._libs.parsers.raise_parser_error()

ParserError: Error tokenizing data. C error: Expected 4 fields in line 2, saw 5

Right way

import pandas as pd

datas = pd.read_csv('test.txt', header=None, skip_blank_lines=True, escapechar='\\')

题外话

• 网上有很多人遇到这个问题加参数：error_bad_lines=False(测试过，针对上面的数据会丢掉第二行)
• 如果数据量不是很大，查看指定行的方法： cat -n filename | head -n end_line_no| tail -n +start_line_no