Java API 写 Hive Orc文件

接前面的文章 《Java API 读取Hive Orc文件》,本文中介绍使用Java API写Orc格式的文件。

下面的代码将三行数据：

张三,20

李四,22

王五,30

写入HDFS上的/tmp/lxw1234/orcoutput/lxw1234.com.orc文件中。

1. package com.lxw1234.test;
2.  
3. import java.io.DataInput;
4. import java.io.DataOutput;
5. import java.io.IOException;
6.  
7. import org.apache.hadoop.fs.FileSystem;
8. import org.apache.hadoop.fs.Path;
9. import org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat;
10. import org.apache.hadoop.hive.ql.io.orc.OrcSerde;
11. import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
12. import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
13. import org.apache.hadoop.io.NullWritable;
14. import org.apache.hadoop.io.Writable;
15. import org.apache.hadoop.mapred.JobConf;
16. import org.apache.hadoop.mapred.OutputFormat;
17. import org.apache.hadoop.mapred.RecordWriter;
18. import org.apache.hadoop.mapred.Reporter;
19.  
20. /**
21. * lxw的大数据田地 -- http://lxw1234.com
22. * @author lxw.com
23. *
24. */
25. public class TestOrcWriter {
26.  
27. public static void main(String[] args) throws Exception {
28. JobConf conf = new JobConf();
29. FileSystem fs = FileSystem.get(conf);
30. Path outputPath = new Path("/tmp/lxw1234/orcoutput/lxw1234.com.orc");
31. StructObjectInspector inspector =
32. (StructObjectInspector) ObjectInspectorFactory
33. .getReflectionObjectInspector(MyRow.class,
34. ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
35. OrcSerde serde = new OrcSerde();
36. OutputFormat outFormat = new OrcOutputFormat();
37. RecordWriter writer = outFormat.getRecordWriter(fs, conf,
38. outputPath.toString(), Reporter.NULL);
39. writer.write(NullWritable.get(),
40. serde.serialize(new MyRow("张三",20), inspector));
41. writer.write(NullWritable.get(),
42. serde.serialize(new MyRow("李四",22), inspector));
43. writer.write(NullWritable.get(),
44. serde.serialize(new MyRow("王五",30), inspector));
45. writer.close(Reporter.NULL);
46. fs.close();
47. System.out.println("write success .");
48. }
49.  
50. static class MyRow implements Writable {
51. String name;
52. int age;
53.  
54. MyRow(String name,int age){
55. this.name = name;
56. this.age = age;
57. }
58. @Override
59. public void readFields(DataInput arg0) throws IOException {
60. throw new UnsupportedOperationException("no write");
61. }
62. @Override
63. public void write(DataOutput arg0) throws IOException {
64. throw new UnsupportedOperationException("no read");
65. }
66.  
67. }
68.  
69. }
70.  

将上面的程序打包成orc.jar,上传至Hadoop客户端机器，

执行命令：

export HADOOP_CLASSPATH=/usr/local/apache-hive-0.13.1-bin/lib/hive-exec-0.13.1.jar:$HADOOP_CLASSPATH

hadoop jar orc.jar com.lxw1234.test.TestOrcWriter

执行成功后，在HDFS上查看该文件：

1. [liuxiaowen@dev tmp]$ hadoop fs -ls /tmp/lxw1234/orcoutput/
2. Found 1 items
3. -rw-r--r-- 2 liuxiaowen supergroup 312 2015-08-18 18:09 /tmp/lxw1234/orcoutput/lxw1234.com.orc

接下来在Hive中建立外部表，路径指向该目录，并设置文件格式为ORC：

1. CREATE EXTERNAL TABLE lxw1234(
2. name STRING,
3. age INT
4. ) stored AS ORC
5. location '/tmp/lxw1234/orcoutput/';

在Hive中查询该表数据：

1. hive> desc lxw1234;
2. OK
3. name string
4. age int
5. Time taken: 0.148 seconds, Fetched: 2 row(s)
6. hive> select * from lxw1234;
7. OK
8. 张三 20
9. 李四 22
10. 王五 30
11. Time taken: 0.1 seconds, Fetched: 3 row(s)
12. hive>

OK，数据正常显示。

注意：该程序只做可行性测试，如果Orc数据量太大，则需要改进，或者使用MapReduce；

后续将介绍使用MapReduce读写Hive Orc文件。