hive的序列化表（avro）

   hive数据内的avro表也是可以是外部表和内部表两种形式，如下为创建avro外部表的语句：

 1 CREATE EXTERNAL TABLE tweets
 2  COMMENT "A table backed by Avro data with the
 3        Avro schema embedded in the CREATE TABLE statement"
 4  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
 5  STORED AS
 6  INPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
 7 OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
 8  LOCATION '/data/WM/warehouse/tweets'
 9  TBLPROPERTIES (
10     'avro.schema.literal'='{
11         "type": "record",
12         "name": "Tweet",
13         "namespace": "com.miguno.avro",
14         "fields": [
15             { "name":"username",  "type":"string"},
16            { "name":"tweet",     "type":"string"},
17             { "name":"timestamp", "type":"long"}
18         ]
19    }'
20  );

上边hsql语句为创建了一张tweets的外部表，这张表的地段就是fields内的几个字段内容。

怎么将json格式的数据转换成avro格式的数据？

 1、需要一个avsc格式的的文件  ， 文件的内容如下：

{
   "type" : "record",
   "name" : "twitter_schema",
   "namespace" : "com.miguno.avro",
   "fields" : [
        {     "name" : "username",
               "type" : "string",
              "doc"  : "Name of the user account on Twitter.com"   },
         {
             "name" : "tweet",
             "type" : "string",
             "doc"  : "The content of the user's Twitter message"   },
         {
             "name" : "timestamp",
             "type" : "long",
             "doc"  : "Unix epoch time in seconds"   } 
    ],
   "doc:" : "A basic schema for storing Twitter messages" 
}

描述你要转换数据的格式，还需要一个数据文件 twitter.json 中有一些数据其内容如下：

{"username":"miguno","tweet":"Rock: Nerf paper, scissors is fine.","timestamp":1366150681}{"username":"BlizzardCS","tweet":"Works as intended.  Terran is IMBA.","timestamp":1366154481}

在执行avro的jar即可生成avro格式的数据，具体的命令如下：

java -jar ~/avro-tools-1.7.7.jar fromjson --schema-file twitter.avsc twitter.json > twitter.avro

讲上述生成的avro格式的文件导入到hive 存在hdfs的目录下即可使用hsql语句查看起内容啦。

CREATE EXTERNAL TABLE tweets

 COMMENT "A table backed by Avro data with the

       Avro schema embedded in the CREATE TABLE statement"

 ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'

 STORED AS

 INPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'

OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'

 LOCATION '/data/WM/warehouse/tweets'

 TBLPROPERTIES (

    'avro.schema.literal'='{

        "type": "record",

        "name": "Tweet",

        "namespace": "com.miguno.avro",

        "fields": [

            { "name":"username",  "type":"string"},

           { "name":"tweet",     "type":"string"},

            { "name":"timestamp", "type":"long"}

        ]

   }'

 );

转载于:https://www.cnblogs.com/bigdatatechnology/p/7279684.html