linux lz4 lzo,Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO

最新推荐文章于 2023-09-14 19:27:37 发布

Jane Gao

最新推荐文章于 2023-09-14 19:27:37 发布

阅读量363

点赞数

文章标签： linux lz4 lzo

Selected archives

I have selected:

Source of the kernel to test source compression

Stream protocol with flush

Test conditions

Tests were run on a desktop:

Intel Core i5 CPU 750 at 2.67GHz

8GB of DDR3 memory

tmpfs as ram disk is used

Linux kernel 3.3.2, gentoo amd64

CFLAGS: -pipe -O2 -g -floop-block -floop-interchange -fgraphite

bzip2-1.0.6-r3, xz-utils-5.0.3, gzip-1.4

Only normal mode will be tested firstly.

The file test results

Note: The first column with numbers 1..9 indicates the compression setting passed to gzip, bzip2 and lzmash (e.g. "gzip -9").

Tarball mode from linux-3.3, original size: 466083840 (445M)

Compressed file size in bytes

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

124875819 (120M)

93997047 (90M)

85618192 (82M)

72042179 (69M)

85630688 (82M)

72069084 (69M)

165844264 (159M)

168012430 (161M)

119040249 (114M)

87741348 (84M)

81480824 (78M)

70264395 (68M)

81492504 (78M)

70282944 (68M)

165844264 (159M)

166987891 (160M)

114931686 (110M)

84816957 (81MB)

79575087 (76MB)

69015118 (66MB)

79586568 (76MB)

69029204 (66MB)

165844264 (159M)

166987891 (160M)

102328357 (98M)

81837328 (79M)

69557610 (67M)

67879362 (65M)

69583428 (67M)

67875988 (65M)

166987891 (160M)

100128597 (96M)

80197758 (77M)

67276420 (65M)

66868212 (64M)

67294092 (65M)

66852780 (64M)

116205578 (111M)

99740486 (96M)

78963640 (76M)

65841213 (63M)

65362226 (63M)

65859432 (63M)

65372696 (63M)

114824102 (110M)

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

Compression ratio

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

26.8%

20.2%

18.4%

15.5%

18.4%

15.5%

35.6%

36.0%

25.5%

18.8%

17.5%

15.1%

17.5%

15.1%

35.6%

35.8%

24.7%

18.2%

17.1%

14.8%

17.1%

14.8%

35.6%

35.8%

22.0%

17.6%

14.9%

14.6%

14.9%

14.6%

35.8%

21.5%

17.2%

14.4%

14.3%

14.4%

14.3%

24.9%

21.4%

16.9%

14.1%

14.0%

14.1%

14.0%

24.6%

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

Compression time

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

8.1s

58.3s

31.7s

4m37s

32.2s

4m40s

1.3s

1.6s

8.5s

58.4s

40.7s

4m49s

41.9s

4m53s

1.4s

1.6s

9.6s

59.1s

1m2s

4m36s

1m1s

4m39s

1.3s

1.5s

14s

1m1s

3m5s

3m6s

4m53s

1.5s

21s

1m2s

4m14s

5m52s

4m13s

5m57s

35s

33s

1m3s

4m48s

6m40s

4m51s

6m40s

1m5s

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

gzip -1 vs lz4 -1 on x86: lz4 6.2x more fast

gzip -1 vs lz4 -1 on ARM: lz4 3.6x more fast

Decompression time

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

3.5s

3.4s

6.7s

5.9s

7.2s

6.5s

0.4s

1.5s

15.7

6.3s

5.6s

6.8s

6.3s

0.3s

1.4s

3.2s

15.9s

5.6s

6.7s

6.2s

0.4s

1.4s

3.2s

16s

5.5s

5.4s

6.2s

1.5s

15s

5.3s

5.9s

5.8s

1.3s

15s

5.1s

5.6s

1.2s

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

Memory requirements on compression

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

0.4MB

1.1MB

8.3MB

12.6MB

8.3MB

12.5MB

12MB

0.7MB

0.4MB

1.9MB

15.8MB

24MB

15.8MB

24MB

12MB

0.7MB

0.4MB

2.7MB

30.7MB

46.9MB

30.8MB

47M

13.2MB

0.7MB

0.4MB

4.2MB

93MB

0.7MB

0.4MB

5.7MB

185MB

0.9MB

0.4MB

7.2MB

672MB

673MB

0.9MB

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

Note: lz4 it's the program using this size, the code for internal lz4 use very less memory

Memory requirements on decompression

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

0.1MB

0.3MB

1MB

13MB

0.2MB

0.1MB

0.9MB

2.1MB

12MB

0.2MB

0.1MB

1.1MB

4.1MB

4.0MB

13MB

0.2MB

0.1MB

1.9MB

8.1MB

0.2MB

0.1MB

2.6MB

16MB

0.2MB

0.1MB

3.4MB

64MB

0.2MB

gzip

bzip2

lzma

lzma -e

xz -e

lz4

lzop

Note: lz4 it's the program using this size, the code for internal lz4 use very less memory

Cli used for -1 compression:

time gzip -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;time gunzip linux-3.3.tar*

time bzip2 -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;time bunzip2 linux-3.3.tar.bz2

time lzma -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unlzma linux-3.3.tar*

time lzma -e -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unlzma linux-3.3.tar*

time xz -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unxz linux-3.3.tar*

time xz -e -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unxz linux-3.3.tar*

time /home/user/lz4-read-only/lz4demo64 -c0 linux-3.3.tar linux-3.3.tar.lz4;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time /home/user/lz4-read-only/lz4demo64 -d linux-3.3.tar.lz4 linux-3.3.tar;rm linux-3.3.tar.*

time lzop -1 linux-3.3.tar;rm linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time lzop -d linux-3.3.tar*;rm linux-3.3.tar.*

Stream test

Time to compress

gzip

zlib

lz4

lz4hc

none

376

368

130

311

371

391

378

362

390

392

389

387

393

384

gzip

zlib

lz4

lz4hc

none

Time to decompress

gzip

zlib

lz4

lz4hc

none

1797

1778

809

796

1841

1831

1771

1780

1806

1960

1861

1851

1835

1806

gzip

zlib

lz4

lz4hc

none

Messured with:

timespec tstart,tstop;

clock_gettime(CLOCK_REALTIME, &tstart);

//compression/decompression

clock_gettime(CLOCK_REALTIME, &tstop);

compressionTime+=((quint64)tstop.tv_sec * 1000000LL + (quint64)tstop.tv_nsec / 1000LL)-((quint64)tstart.tv_sec * 1000000LL + (quint64)tstart.tv_nsec / 1000LL);

Memory used

gzip

zlib

lz4

lz4hc

none

0.5MB

0MB

gzip

zlib

lz4

lz4hc

none

The socket is always used as full duplex, then it's compression + decompression memory.

Compression ratio

It's transferred size/raw size (lower is better)

gzip

zlib

lz4

lz4hc

none

0.99

0.97

1.59

gzip

zlib

lz4

lz4hc

none

The most over-head of small packet (3Bytes) is drop by high compression with zlib/gzip for the big packet.

Graphics

Quick benchmark on ARM64

A quick benchmark on ARM64 (odroid, Cortex A53), on kernel Image (12MB), use default compression level (-6) because no way to configure the compression level of btrfs

The speed is on compressed stream, mean the hdd.

gzip

lz4

zstd

Size

4.2M

5.5M

7.8M

5.8M

Compression speed

0.2 MB/s

1.8 MB/s

21.4 MB/s

2.8 MB/s

Decompression speed

4.8 MB/s

13.6 MB/s

48.4 MB/s

19.1 MB/s

Jane Gao

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
linux lz4 lzo,Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO

Selected archivesI have selected:Source of the kernel to test source compressionStream protocol with flushTest conditionsTests were run on a desktop:Intel Core i5 CPU 750 at 2.67GHz8GB of DDR3 memoryt...
复制链接

扫一扫