考虑使用linux自带的crontab将文本文件导入到hive中
shell脚本如下:
1.入hive表的shell脚本 repeatoffer.sh
set hive.execution.engine=spark;
set spark.yarn.queue=xxtg045;
load data local inpath '${hivevar:oripath}' into table ${hivevar:targettable} partition(month_partition_id='${hivevar:monthid}')
2.实现自动化入hive的shell脚本 auto.sh
#!bin/bash
period=$(date -d last-month +%Y%m)
oripathcomm="/data/xxtg045/origindata/repeatoffer/${period}"
targettablecomm="xxtg045.tf_f_repeat_offer"
monthidcomm="${period}"
hive --hivevar oripath=${oripathcomm} --hivevar targettable=${targettablecomm} --hivevar monthid=${monthidcomm} -f /data/xxtg045/shell/repeatoffer/repeatoffer.sh
#echo "${oripathcomm}入库状态:$?">./logs.txt
将auto.sh写入到crontab -e中,实现定时的数据提取
# 博学谷IT 技术支持