全景图:
1. 创建hive表
CREATE TABLE IF NOT EXISTS newsinfo.test(
name STRING
)
CLUSTERED BY (name)INTO 3 BUCKETS
ROW FORMAT DELIMITED
STORED AS ORC
TBLPROPERTIES('transactional'='true');
2. 这里用了 ReplaceText 生成 json 数据 (正式环境可以直接从hfs里取到)
3. 用ConvertJSONToAvro 转换json 到avro
{ "name": "dtu", "type": "record", "fields":[ { "name":"name","type": "string" } ] }
4. PutHiveStreaming