python gzip seek慢,Python gzip拒绝读取未压缩的文件

I seem to remember that the Python gzip module previously allowed you to read non-gzipped files transparently. This was really useful, as it allowed to read an input file whether or not it was gzipped. You simply didn't have to worry about it.

Now,I get an IOError exception (in Python 2.7.5):

Traceback (most recent call last):

File "tst.py", line 14, in

rec = fd.readline()

File "/sw/lib/python2.7/gzip.py", line 455, in readline

c = self.read(readsize)

File "/sw/lib/python2.7/gzip.py", line 261, in read

self._read(readsize)

File "/sw/lib/python2.7/gzip.py", line 296, in _read

self._read_gzip_header()

File "/sw/lib/python2.7/gzip.py", line 190, in _read_gzip_header

raise IOError, 'Not a gzipped file'

IOError: Not a gzipped file

If anyone has a neat trick, I'd like to hear about it. Yes, I know how to catch the exception, but I find it rather clunky to first read a line, then close the file and open it again.

解决方案

The best solution for this would be to use something like https://github.com/ahupp/python-magic with libmagic. You simply cannot avoid at least reading a header to identify a file (unless you implicitly trust file extensions)

If you're feeling spartan the magic number for identifying gzip(1) files is the first two bytes being 0x1f 0x8b.

In [1]: f = open('foo.html.gz')

In [2]: print `f.read(2)`

'\x1f\x8b'

gzip.open is just a wrapper around GzipFile, you could have a function like this that just returns the correct type of object depending on what the source is without having to open the file twice:

#!/usr/bin/python

import gzip

def opener(filename):

f = open(filename,'rb')

if (f.read(2) == '\x1f\x8b'):

f.seek(0)

return gzip.GzipFile(fileobj=f)

else:

f.seek(0)

return f