shell脚本实现把文本数据导入Redis

今年公司注重实时相关的业务，但是平台Hive入Redis的功能还没上线，只能通过脚本实现把Hive里的数据定时导入Redis。

前提：Hive表的数据已导出为竖线分割的文本格式。

文本数据:

Shell脚本如下：

#!/bin/bash


function Clean(){
incre=0
while read row
do
  #echo "${row}";
  for i in ${row}
  do
    telnum=$(echo "${i}" |awk -F "|" '{print $1}')
    activeType=$(echo "${i}" |awk -F "|" '{print $2}')
    region=$(echo "${i}" |awk -F "|" '{print $3}')
    crowd=$(echo "${i}" |awk -F "|" '{print $4}')
    product=$(echo "${i}" |awk -F "|" '{print $5}')
    department_id=$(echo "${i}" |awk -F "|" '{print $6}')
    #echo "${telnum}"
    #echo "${activeType}"
    #echo "${region}"
    #echo "${crowd}"
    #echo "${product}"
    #echo "${department_id}"
    echo "hmset asiainfo:${incre} telnum ${telnum} activeType ${activeType} region ${region} crowd ${crowd} product ${product} department_id ${department_id}" >> clean1.txt
    echo "EXPIRE asiainfo:${incre} 20" >> clean1.txt
    let incre++
  done
done < hive1.txt
}

function Load_redis(){
todos /root/clean1.txt #转码
time cat /root/clean1.txt | redis-cli --pipe
}


function start(){
Clean
Load_redis
}


start

效果如下：

root@hadoop:~# ./load_redis.sh 
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 15824

real    0m0.076s
user    0m0.003s
sys     0m0.001s

20秒到期后消失：