Hadoop/Hbase的Snappy安装

一、Snappy安装环境：

gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3

其中gcc一定要是4.4的版本。

二、下载Snappy 1.1.1

下载地址：http://code.google.com/p/snappy/downloads/list

三、编译Snappy

./configure
make
make install

四、Hadoop Snappy 源码编译

svn checkout http://hadoop-snappy.googlecode.com/svn/trunk hadoop-snappy

mvn package [-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR]

注：如果第三步snappy安装路径是默认的话，即/usr/local/lib，则此处 [-Dsnappy.prefix=SNAPPY_INSTALLATION_DIR] 可以不写，或者 -Dsnappy.prefix=/usr/local/

五、拷贝文件

解压第四步target下hadoop-snappy-0.0.1-SNAPSHOT.tar.gz，解压后，复制lib文件

cp -r /home/hadoopuser/snappy-hadoop/target/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/Linux-amd64-64/

将第四步target下的hadoop-snappy-0.0.1-SNAPSHOT.jar复制到$HADOOP_HOME/lib 下。

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

六、配置mapred-site.xml，这个文件中，所有跟压缩有关的配置选项有：

<property>
  <name>mapred.output.compress</name>
  <value>false</value>
  <description>Should the job outputs be compressed?
  </description>
</property>
 
<property>
  <name>mapred.output.compression.type</name>
  <value>RECORD</value>
  <description>If the job outputs are to compressed as SequenceFiles, how should
               they be compressed? Should be one of NONE, RECORD or BLOCK.
  </description>
</property>
 
<property>
  <name>mapred.output.compression.codec</name>
  <value>org.apache.hadoop.io.compress.DefaultCodec</value>
  <description>If the job outputs are compressed, how should they be compressed?
  </description>
</property>
 
<property>
  <name>mapred.compress.map.output</name>
  <value>false</value>
  <description>Should the outputs of the maps be compressed before being
               sent across the network. Uses SequenceFile compression.
  </description>
</property>
 
<property>
  <name>mapred.map.output.compression.codec</name>
  <value>org.apache.hadoop.io.compress.DefaultCodec</value>
  <description>If the map outputs are compressed, how should they be
               compressed?
  </description>
</property>

根据自己的需要，配置进去即可。我们为了验证方便，仅配置map部分：

<property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
        </property>
        <property>
        <name>mapred.map.output.compression.codec</name>
        <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

重新启动hadoop。为了验证是否成功，往hdfs上传一个文本文件，敲入一些词组，运行wordcount程序。如果map部分100%完成，即说明我们hadoop snappy安装成功。因为hadoop没有像HBase一样提供util.CompressionTest类(或者是我没有找到)，所以只能按照这种方法来测试。接下来，将详细列出HBase使用Snappy的配置过程。

bin/hadoop jar hadoop-examples-1.2.1.jar wordcount /input /output1

七、HBase配置Snappy

拷贝文件：

cp -r $HADOOP_HOME/lib/native/Linux-amd64-64/ $HBASE_HOME/lib/native/

hbase-env.sh配置：

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

重启HBase

验证安装是否成功：

hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://10.200.102.239:9000/output1/part-r-00000 snappy

执行命令后结果为：

14/01/15 16:41:55 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
14/01/15 16:41:55 DEBUG util.FSUtils: Creating file=hdfs://10.200.102.239:9000/output1/part-r-00000 with permission=rwxrwxrwx
14/01/15 16:41:55 INFO util.FSUtils: FileSystem doesn't support getDefaultReplication
14/01/15 16:41:55 INFO util.FSUtils: FileSystem doesn't support getDefaultBlockSize
14/01/15 16:41:55 ERROR metrics.SchemaMetrics: Inconsistent configuration. Previous configuration for using table name in metrics: true, new configuration: false
14/01/15 16:41:55 WARN snappy.LoadSnappy: Snappy native library is available
14/01/15 16:41:55 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/01/15 16:41:55 INFO snappy.LoadSnappy: Snappy native library loaded
14/01/15 16:41:55 INFO compress.CodecPool: Got brand-new compressor
14/01/15 16:41:55 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:disabled
14/01/15 16:41:56 INFO compress.CodecPool: Got brand-new decompressor
SUCCESS

接下来创建并操作Snappy压缩格式的表：

[hadoopuser@RDCMaster ~]# hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.94.2, r1395367, Sun Oct  7 19:11:01 UTC 2012
//创建表
hbase(main):001:0> create 'tsnappy', { NAME => 'f', COMPRESSION => 'snappy'}
0 row(s) in 10.6590 seconds
//describe表
hbase(main):002:0> describe 'tsnappy'
DESCRIPTION                                                                                                        ENABLED                                                      
 {NAME => 'tsnappy', FAMILIES => [{NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_ true                                                         
 SCOPE => '0', VERSIONS => '3', COMPRESSION => 'SNAPPY', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CE                                                              
 LLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}]}                                                                  
1 row(s) in 0.2140 seconds
//put数据
hbase(main):003:0> put 'tsnappy', 'row1', 'f:col1', 'value'
0 row(s) in 0.5190 seconds
//scan数据
hbase(main):004:0> scan 'tsnappy'
ROW                                           COLUMN+CELL                                                                                                                       
 row1                                         column=f:col1, timestamp=1358143780950, value=value                                                                               
1 row(s) in 0.0860 seconds
 
hbase(main):005:0>

以上所有过程均成功执行，说明Snappy 在Hadoop及HBase上配置成功~