hive字段拼接json格式

最新推荐文章于 2024-08-02 00:27:30 发布

我是60岁程序员

最新推荐文章于 2024-08-02 00:27:30 发布

阅读量1.5w

点赞数 2

分类专栏： hive 文章标签： hive字段拼成json格式

本文链接：https://blog.csdn.net/weixin_38799368/article/details/83450008

版权

hive 专栏收录该内容

17 篇文章 1 订阅

订阅专栏

hive用named_struct(‘字段1’，字段1，‘字段2’，字段2) 可以拼接json，并且特别方便

用最原始的方法拼接json：

startdate=`date -d '-1 days' +%Y-%m-%d`
date=`date '+%Y-%m-%d %H:%M:%S'`

sql="
select concat('{\"date\":\"','$date','\",\"actions\":[',concat('{\"itemId\":\"',itemId,
'\",\"actionTime\":',actionTime,
',\"action\":\"',action,
'\",\"sceneId\":\"',sceneId,
'\",\"userId\":\"',userId,
'\",\"blogId\":\"',nvl(blogId, '-'),
'\",\"uuid_tt_dd\":\"',uuid_tt_dd,
'\"}'),']}') as value
from(
select split(regexp_replace(curl,'\\\\?','/'),'/')[4] itemId,
cast(concat(unix_timestamp(concat(pv_dt,' ',pv_time)),'000') as bigint) actionTime,
'csdnStatistics' action,
case when split(curl,'=')[1]='blogt0' then '420' else '34' end sceneId,
uid userId,
split(regexp_replace(ref,'\\\\?','/'),'/')[6] blogId,
cid uuid_tt_dd
from csdnbi.fact_pv_data
where (curl = 'https://www.baidu.com/topic/python115?utm_source=blogt0'
or curl = 'https://www.baidu.com/topic/ai20?utm_source=blogt3'
or curl = 'https://www.baidu.com/topic/blockchain10?utm_source=blogt3')
and pdate = '$startdate') t
"

hive -e "set mapred.job.name=zoujcTask;set mapred.job.queue.name=super;$sql" > logdata.txt
cat logdata.txt | while read line
do
curl -X POST 'https://www.baidu.com/action/api/log?clientToken=1difushafhsiufh88888' -H 'Content-Type: application/json' -d "$line"
done