ORC (Optimized Row Columnar) is a columnar file format optimized to improve performance of Hive. Through the Hive metastore and HCatalog reading, writing, and processing can also be accomplished by MapReduce, Pig, Cascading, and so on. It is very similar to Parquet which is being developed by Cloudera and Twitter. Both are part of the most current Hive release and available to be used immediately. In this post I would like to describe some of the key concepts of ORC and demonstrate how to get started quickly using HCatalog.
The key concepts of ORC files are:
- Split the Rows Into Strips (default size 250MB)
Splitting the data into independent Strips allows to access the data concurrently using separate RecordReaders. Strips can be split without scanning for markers. - Use Lightweight Indexes
Strips are surrounded be an lightweight index and footer. The index, so called strip index, is applied to the columns contained within the Strip. This makes for really fast filtering as records can be skipped not contained in the query. - Compress Column Values
Column values are compressed so that not the complete row or Strip has to be decompressed prior to reading but only the values that are potential of interest. Supported codecs are Snappy, Zlib, or none. - Support for Hive’s Data Types
ORC supports Hive types like datetime, decimals, and so on. For Hive data types you should check which are supported by you current version and also if they are supported by HCatalog if you plan to use it outside of Hive!
There are some more aspects to ORC files, most notable a custom serialization that helps on speed and space. This includes ZigZag, dictionary, and variable-length encoding. All of that is very similar to what Parquet uses and the most notable differences seems to be the Parquet’s emphasis on deeply nested data as well as the use of Thrift in Parquet overProtocol Buffers in ORC.
Here is a visiualization provided be Hortonworks which shows some of the compression benefits you could gain from using ORC files.
Source: http://hortonworks.com/blog/orcfile-in-hdp-2-better-compression-better-performance/
In general to achieve a good compression using Zlib, which is the default for ORC, over Snappy is a good choice. This comes likely at sacrificing speed.
Creating a first ORC Backed Table in Hive
As ORC (Ser/Deser) is included with the most recent release of Hive using it is very simple. As a quick example we are going to model crawled Blog posts to be stored in our database web. This could be modeled for example like this:
This creates us a ORC backed table in Hive which uses Snappy compression instead of Zlib. The table will be partitioned by the day each blog post was crawled (Y-m-d). HCatalog only supports partitions being of type string. Partitions are basically folders in HDFS and and the column equals the partition values. That’s why it needs to be a string here and we use two columns to store the crawl date/day. There are some other table options (TBLEPROPERTIES) specific to ORC that might interest you:
orc.compress (default: zlib)orc.compress.size (default: 262144 Bytes)orc.stripe.size (default: 256 MB)orc.row.index.stride (default: 10,000)orc.create.index (default: true)
A blog post as consists of a URI, text, author and so on as well as the comments to the each post. We use a STRUCT to include the comments within each post. To create the table in our database we can execute the following command using hcat :
We include here the data definition from above to achieve the desired result of creating our table in Hive/HCatalog by using -e or we put the above statement into a file and use -f . Refer to hcat --help for more information.
From here we can use either Pig, Sqoop, Hive, MapReduce, or Cascading to update and integrate data to our table. Let’s have a quick look at how we can use either Pig to insert and query some data. For this we’ll use a small set of data we can experiment with. Here are the blog posts and comments we are going to use for this example.
Posts (post.tsv):
Comments (comments.tsv):
We have posts and comments in separate files. For simplicity the values are tab separated so we can use the default PigLoader. Please keep in mind that the defaultPigLoader does not supports proper escaping and would in such a case be a bad choice as title and text could likely contain tabs as well as line breaks.
Since posts and comments are in separated files we have to group them together. This sometimes does not very look nice in Pig and requires some re-naming. Pig appends in some cases the name of a tuple to the values to make them unique. This is often ignored so that posts::uri and uri are the same, but here unfortunately not. But a re-naming doesn’t hurt either as it will be part of any appropriate map or reduce step and doesn’t require a separated step. Let’s have a look on how we could load the posts and comments and then store them into Hive:
To execute the above you have to run Pig with HCatalog support. It does not by default include the required libraries. In addition depending on your setup it might be required to point to the metastore of Hive. You can do that with the following command:
With -useHCatolog Pig can make use of HCatalog and here in addition we also point to the location of the Hive metastore.
That’s it! And you should be able to execute the following:
Further Readings
- Hadoop: The Definitive Guide (Amazon)
2065
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
