shell获取hive查询数据结果_如何使用Shell脚本将HiveQL查询的结果输出到CSV？

I would like to run multiple Hive queries, preferably in parallel rather than sequentially, and store the output of each query into a csv file. For example, query1 output in csv1, query2 output in csv2, etc. I would be running these queries after leaving work with the goal of having output to analyze during the next business day. I am interested in using a bash shell script because then I'd be able to set-up a cron task to run it at a specific time of day.

I know how to store the results of a HiveQL query in a CSV file, one query at a time. I do that with something like the following:

hive -e

"SELECT * FROM db.table;"

" | tr "\t" "," > example.csv;

The problem with the above is that I have to monitor when the process finishes and manually start the next query. I also know how to run multiple queries, in sequence, like so:

hive -f hivequeries.hql

Is there a way to combine these two methods? Is there a smarter way to achieve my goals?

Code answers are preferred since I do not know bash well enough to write it from scratch.

解决方案

You can run and monitor parallel jobs in a shell script:

#!/bin/bash

#Run parallel processes and wait for their completion

#Add loop here or add more calls

hive -e "SELECT * FROM db.table1;" | tr "\t" "," > example1.csv &

hive -e "SELECT * FROM db.table2;" | tr "\t" "," > example2.csv &

hive -e "SELECT * FROM db.table3;" | tr "\t" "," > example3.csv &

#Note the ampersand in above commands says to create parallel process

#You can wrap hive call in a function an do some logging in it, etc

#And call a function as parallel process in the same way

#Modify this script to fit your needs

#Now wait for all processes to complete

#Failed processes count

FAILED=0

for job in `jobs -p`

do

echo "job=$job"

wait $job || let "FAILED+=1"

done

#Final status check

if [ "$FAILED" != "0" ]; then

echo "Execution FAILED! ($FAILED)"

#Do something here, log or send messege, etc

exit 1

fi

#Normal exit

#Do something else here

exit 0