ValueError: could not convert string to float: '\ufeff6'错误解决（UTF-8和UTF-8-sig的区别）

 

1 解决方案

载入csv文件，代码如下

def loadCsv(filename):
    lines = csv.reader(open(filename, "rt", encoding="UTF-8"))
    dataset = list(lines)
    for i in range(len(dataset)):
        dataset[i] = [float(x) for x in dataset[i]]
    return dataset

处理时出现如下错误：

Traceback (most recent call last):
  File ".\bayes.py", line 120, in <module>
    main()
  File ".\bayes.py", line 107, in main
    dataset = loadCsv(filename)
  File ".\bayes.py", line 11, in loadCsv
    dataset[i] = [float(x) for x in dataset[i]]
  File ".\bayes.py", line 11, in <listcomp>
    dataset[i] = [float(x) for x in dataset[i]]
ValueError: could not convert string to float: '\ufeff6'

实际数据是以下形式：

6,148,72,35,0,33.6,0.627,50,1
1,85,66,29,0,26.6,0.351,31,0
8,183,64,0,0,23.3,0.672,32,1
1,89,66,23,94,28.1,0.167,21,0
0,137,40,35,168,43.1,2.288,33,1

开头多出了\ufeff，utf-8改为utf-8-sig可解决

def loadCsv(filename):
    lines = csv.reader(open(filename, "rt", encoding="UTF-8-sig"))
    dataset = list(lines)
    for i in range(len(dataset)):
        dataset[i] = [float(x) for x in dataset[i]]
    return dataset

2 utf-8 和 uft-8-sig

”utf-8“ 是以字节为编码单元,它的字节顺序在所有系统中都是一样的,没有字节序问题,因此它不需要BOM,所以当用"utf-8"编码方式读取带有BOM的文件时,它会把BOM当做是文件内容来处理, 也就会发生类似上边的错误.

"uft-8-sig"中sig全拼为 signature 也就是"带有签名的utf-8", 因此"utf-8-sig"读取带有BOM的"utf-8文件时"会把BOM单独处理,与文本内容隔离开,也是我们期望的结果.