RegexSerDe是hive自带的一种序列化/反序列化的方式,主要用来处理正则表达式。
举个实际的例子:
CREATE TABLE test_serde(
c0 string,
c1 string,
c2 string)
ROW FORMAT
SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES
( 'input.regex' = '([^ ]*) ([^ ]*) ([^ ]*)',
'output.format.string' = '%1$s %2$s %3$s')
STORED AS TEXTFILE;
这个创建表test_serde,使用RegexSerDe,正则表达式为([^ ]*) ([^ ]*) ([^ ]*),实际上就是按空格分割。
输出的格式为%1$s %2$s %3$s,分别对应表上的三个字段。
$ cat test03.txt
101 tom 90
102 kate 100
103 mary 101
$ hive
Hive history file=/tmp/hjl/hive_job_log_hjl_201105122213_1206014869.txt
hive> add jar /home/hjl/hive/lib/hive_contrib.jar;
Added /home/hjl/hive/lib/hive_contrib.jar to class path
为了避免每次执行hive时add jar,可以将下面的配置语句,加在配置文件: $HIVE_INSTALL/conf/hive-site.xml中,value中hive-contrib-*.jar的路径为你机器上实际的放置,在$HIVE_INSTALL/lib目录下寻找。
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/develop/hive-0.8.0/lib/hive-contrib-0.8.0.jar</value>
<description>Added by tiger.zeng on 20120202.These JAR file are available to all users for all jobs</description>
</property>
退出hive shell, 重新启动hive shell, 可运行正常。
hive> CREATE TABLE test_serde(
> c0 string,
> c1 string,
> c2 string)
> ROW FORMAT
> SERDE ‘org.apache.hadoop.hive.contrib.serde2.RegexSerDe’
> WITH SERDEPROPERTIES
> ( ‘input.regex’ = ‘([^ ]*) ([^ ]*) ([^ ]*)’,
> ‘output.format.string’ = ‘%1$s %2$s %3$s’)
> STORED AS TEXTFILE;
OK
Time taken: 0.085 seconds
hive> load data local inpath ‘/root/test03.txt’ overwrite into table test_serde;
Copying data from file:/root/test03.txt
Loading data to table test_serde
OK
Time taken: 0.226 seconds
hive> select * from test_serde;
OK
101 tom 90
102 kate 100
103 mary 101
Time taken: 0.246 seconds
RegexSerDe最麻烦的地方就是要写正则表达式。
RegexSerDe主要下面三个参数:
input.regex
output.format.string
input.regex.case.insensitive