脱离Hadoop的环境下使用Lzo

脱离Hadoop的环境下使用Lzo  转自http://heipark.iteye.com/blog/1605721

 

hadoop lzo

动机：希望通过Java API将日志行压缩到lzo文件，然后上传到HDFS。因为kevinweil-hadoop-lzo提供了LzopOutpuStream，所以这一切很容易实现，需要注意的是该jar包依赖native code和lzo本身。本人试过Karmasphere的lzo-java项目，该项目用java重写了lzo压缩算法，但是该项目好久不更新，提问没人回，性能也只有原生lzo压缩性能一半，最难以忍受的是会有压缩后文件损坏的情况，果断放弃。

工具：lzo

地址：http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz

         或者到 http://rpmfind.net 查找rpm包，下面两个是32bit redhat4和5的rpm包

          ftp://rpmfind.net/linux/dag/redhat/el4/en/i386/dag/RPMS/lzo-2.06-1.el4.rf.i386.rpm

  ftp://rpmfind.net/linux/dag/redhat/el5/en/i386/dag/RPMS/lzo-2.06-1.el5.rf.i386.rpm

工具：kevinweil-hadoop-lzo

地址：https://github.com/kevinweil/hadoop-lzo

1. 安装lzo和kevinweil-hadoop-lzo

   详见：http://heipark.iteye.com/blog/1172759

2. 设置环境变量

export JAVA_HOME=/usr/java/latest/
export LD_LIBRARY_PATH=/xxx/build/native/Linux-i386-32/lib

3. 写java代码

Java代码

1. public class TestLzo {  
2.  
3.     @Test 
4.     public void test(String input, String output) throws IOException {  
5.  
6.         BufferedReader textBr = new BufferedReader(new InputStreamReader(new FileInputStream(input)));  
7.         int lzoBufferSize = 256 * 1024;  
8.         LzoCompressor.CompressionStrategy strategy = LzoCompressor.CompressionStrategy.LZO1X_1;  
9.         LzoCompressor lzoCompressor = new LzoCompressor(strategy, lzoBufferSize);  
10.  
11.         LzopOutputStream lzopOut = new LzopOutputStream(new FileOutputStream(output), lzoCompressor, lzoBufferSize, strategy);  
12.         String textLine;  
13.         while ((textLine = textBr.readLine()) != null) {  
14.             textLine += "\n";  
15.             byte[] bytes = textLine.getBytes();  
16.             lzopOut.write(bytes, 0, bytes.length);  
17.         }  
18.         textBr.close();  
19.         lzopOut.close();  
20.     }  
21.  
22.     public static void main(String[] args) throws IOException {  
23.         TestLzo lzo = new TestLzo();  
24.         lzo.test(args[0], args[1]);  
25.     }  
26. } 

public class TestLzo {

	@Test
	public void test(String input, String output) throws IOException {

		BufferedReader textBr = new BufferedReader(new InputStreamReader(new FileInputStream(input)));
		int lzoBufferSize = 256 * 1024;
		LzoCompressor.CompressionStrategy strategy = LzoCompressor.CompressionStrategy.LZO1X_1;
		LzoCompressor lzoCompressor = new LzoCompressor(strategy, lzoBufferSize);

		LzopOutputStream lzopOut = new LzopOutputStream(new FileOutputStream(output), lzoCompressor, lzoBufferSize, strategy);
		String textLine;
		while ((textLine = textBr.readLine()) != null) {
			textLine += "\n";
			byte[] bytes = textLine.getBytes();
			lzopOut.write(bytes, 0, bytes.length);
		}
		textBr.close();
		lzopOut.close();
	}

	public static void main(String[] args) throws IOException {
		TestLzo lzo = new TestLzo();
		lzo.test(args[0], args[1]);
	}
}

 

4. 运行

/usr/java/latest/bin/java -cp .:hadoop-lzo-0.4.15.jar:hadoop-core-0.20.2-cdh3u4.jar:commons-logging-1.1.1.jar com/hadoop/compression/lzo/TestLzo abc.txt abc.txt.lzo 

--heipark