Snappy is a compression/decompression library. It aims for very high speeds and reasonable compression, rather than maximum compression or compatibility with other compression libraries.
Snappy Installation
Snappy is provided in the native package along with the other native libraries (such as native gzipcompression). If you are already using this package there is no additional installation to do. Otherwise follow these installation instructions:
To install Snappy on Ubuntu systems:
To install Snappy on Red Hat systems:
To install Snappy on SUSE systems:
To take advantage of Snappy compression you need to set certain configuration properties, which are explained in the following sections.
Using Snappy for MapReduce Compression
It's very common to enable MapReduce intermediate compression, since this can make jobs run faster without you having to make any application changes. Only the temporary intermediate files created by Hadoop for the shuffle phase are compressed (the final output may or may not be compressed). Snappy is ideal in this case because it compresses and decompresses very fast compared to other compression algorithms, such as Gzip.
To enable Snappy for MapReduce intermediate compression for the whole cluster, set the following properties in mapred-site.xml:
You can also set these properties on a per-job basis.
Use the properties in the following table to compress the final output of a MapReduce job. These are usually set on a per-job basis.
Property | Description |
---|---|
mapred.output.compress | Whether to compress the final job outputs (true or false) |
mapred.output.compression.codec | If the final job outputs are to be compressed, which codec should be used. Set to org.apache.hadoop.io.compress.SnappyCodec for Snappy compression. |
mapred.output.compression.type | For SequenceFile outputs, what type of compression should be used (NONE, RECORD, or BLOCK). BLOCK is recommended. |
Using Snappy for Pig Compression
Set the same properties for Pig as for MapReduce (see the table in the previous section).
Using Snappy for Hive Compression
To enable Snappy compression for Hive output when creating SequenceFile outputs, use the following settings:
Configuring Flume to use Snappy Compression
Depending on the architecture of the machine you are installing on, add one of the following lines to/usr/lib/flume/bin/flume-env.sh:
- For 32-bit platforms:
- For 64-bit platforms:
The following section explains how to take advantage of Snappy compression.
Using Snappy compression in Flume Sinks
You can specify Snappy as a compression codec in Flume's configuration language. For example, the following specifies a Snappy-compressed SequenceFile sink on HDFS:
Using Snappy compression in Sqoop Imports
On the command line, use the following option to enable Snappy compression:
It is a good idea to use the --as-sequencefile option with this compression option.
Configuring HBase to use Snappy Compression
Depending on the architecture of the machine you are installing on, add one of the following lines to/etc/hbase/conf/hbase-env.sh:
- For 32-bit platforms:
- For 64-bit platforms:
To use Snappy compression in HBase Tables, specify the column family compression as snappy. For example, in the shell: