avrorecord.java,失败,但发生异常java.io.IOException:org.apache.avro.AvroTypeException:发现的很长,期望在配置单元中实现联合...

需要帮忙!!!

我正在使用flume将Twitter提要流式传输到hdfs中,并将其加载hive进行分析。

步骤如下:

我已经avro schema在avsc文件中描述了并将其放入hadoop:

{"type":"record",

"name":"Doc",

"doc":"adoc",

"fields":[{"name":"id","type":"string"},

{"name":"user_friends_count","type":["int","null"]},

{"name":"user_location","type":["string","null"]},

{"name":"user_description","type":["string","null"]},

{"name":"user_statuses_count","type":["int","null"]},

{"name":"user_followers_count","type":["int","null"]},

{"name":"user_name","type":["string","null"]},

{"name":"user_screen_name","type":["string","null"]},

{"name":"created_at","type":["string","null"]},

{"name":"text","type":["string","null"]},

{"name":"retweet_count","type":["boolean","null"]},

{"name":"retweeted","type":["boolean","null"]},

{"name":"in_reply_to_user_id","type":["long","null"]},

{"name":"source","type":["string","null"]},

{"name":"in_reply_to_status_id","type":["long","null"]},

{"name":"media_url_https","type":["string","null"]},

{"name":"expanded_url","type":["string","null"]}]}

我写了一个.hql文件来创建表并在其中加载数据:

create table tweetsavro

row format serde

'org.apache.hadoop.hive.serde2.avro.AvroSerDe'

stored as inputformat

'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'

outputformat

'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'

tblproperties ('avro.schema.url'='hdfs:///avro_schema/AvroSchemaFile.avsc');

load data inpath '/test/twitter_data/FlumeData.*' overwrite into table tweetsavro;

我已经成功运行.hql文件,但是当我select *from 在蜂巢中运行命令时,它显示以下错误:

tweetsavro的输出为:

hive> desc tweetsavro;

OK

id string

user_friends_count int

user_location string

user_description string

user_statuses_count int

user_followers_count int

user_name string

user_screen_name string

created_at string

text string

retweet_count boolean

retweeted boolean

in_reply_to_user_id bigint

source string

in_reply_to_status_id bigint

media_url_https string

expanded_url string

Time taken: 0.697 seconds, Fetched: 17 row(s)

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值