python3 json,包含python3中JSON数据的不同压缩方法

So, I want to compress my JSON data using different compressor. I used this to compress the JSON.

import gzip

import JSON

with gzip.GzipFile('2.json', 'r') as isfile:

for line in isfile:

obj = json.loads(line)

which raises error.

raise OSError('Not a gzipped file (%r)' % magic)

OSError: Not a gzipped file (b'[\n')

I also tried direct compressing using.

zlib_data= zlib.compress(data)

which raises an error.

return lz4.block.compress(*args, **kwargs)

TypeError: a bytes-like object is required, not 'list'

So, Basically i want to compress a JSON using all the methods and to compute the time taken for the compression in different methods.

解决方案

On python2.7

it seems to be a problem of the type of your data

the data to compress should be a 'str' type

import gzip

import json

import lz4

import time

with gzip.GzipFile('data.gz','w') as fid_gz:

with open('data.json','r') as fid_json:

# get json as type dict

json_dict = json.load(fid_json)

# convert dict to str

json_str = str(json_dict)

# write string

fid_gz.write(json_str)

# check well maded

with gzip.GzipFile('data.gz','r') as fid_gz :

print(fid_gz.read())

even if gzip compression

gzip.zlib.compress(json_str,9)

even if lz4 compression

lz4.block.compress(json_str)

and time checking would be

# set start time

st = time.time()

# calculate elasped time

print(time.time() - st)

On python3.5

the difference between python2.7 and python 3 is the type of your data to compress

the data to compress should be a 'byte' type via bytes()

when making a .gz file

with gzip.GzipFile('data.gz','w') as fid_gz:

with open('data.json','r') as fid_json:

json_dict = json.load(fid_json)

json_str = str(json_dict)

# bytes(string, encoding)

json_bytes = bytes(json_str,'utf8')

fid_gz.write(json_bytes)

or just compress with gzip.compress(data, compresslevel=9)

# 'data' takes bytes

gzip.compress(json_bytes)

or just compress with zlib.compress(bytes, level=-1, /)

gzip.zlib.compress(json_bytes,9)

or just compress with lz4.bloc.compress(source, compression=0)

# 'source' takes both 'str' and 'byte'

lz4.block.compress(json_str)

lz4.block.compress(json_bytes)

the measuring time is on your intention.

cheers