Linux shell中解决08 value too long问题

1.使用10#

#SHELL脚本中变量以“0”开头，则会被认为是八进制数字，而在八进制数字中出现8则为不合理，遂报该错误。
#先将变量转换为十进制，然后再做运算即可，比如 echo $((10#$2+2))可以运行 。

hour_format=''
for ((i=0;i<=23;i++))
do
if [[ $i -le 9 ]];then
  number="0$i"
  hour_format= $(echo printf "%02d" $((10#$number)))
else 
  hour_format=$i
fi
echo $hour_format
done


[root@iZbp10e8oyd5hz4v6ta7c2Z conf]# sh test.sh 
00
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23

2.使用datetime的格式化 获取08 09数字

#!/bin/bash 

timeone=`date -d "+1 day" +%Y-%m-%d`

for i in {0..47}
do
days=`date -d "$timeone $i hour" +%Y%m%d`
hours=`date -d "$timeone $i hour" +%H`
echo $days --- $hours
done 


# 结果:
20200910 --- 00
20200910 --- 01
20200910 --- 02
20200910 --- 03
20200910 --- 04
20200910 --- 05
20200910 --- 06
20200910 --- 07
20200910 --- 08
20200910 --- 09
20200910 --- 10
20200910 --- 11
20200910 --- 12

3. 应用:flume搜集数据到hive小时分区表,用shell提前一天创建好分区

   #!/bin/bash

today=`date +%Y%m%d`
tomorrow=`date -d "+0 day" +%Y%m%d`

set -u
function add_partition(){
# flume搜集kafka数据到hive分区表
hour_format=''
for ((i=0;i<=23;i++))
do
 
if [[ $i -le 9 ]];then
  number="0$i"
  hour_format=`printf "%02d" $((10#$number))`
else
  hour_format=$i
fi
#echo $hour_format

/hive/apache-hive-XXX-bin/bin/hive -e "
set hive.execution.engine=tez;
alter table ods.XXX add if not exists partition (day='$tomorrow',hour='$hour_format');

" > /flume/flumeFlu/logs/ngnixLogAddpartition.log 2>&1

done

}

function send_email(){
if grep -q "FAILED" /flume/flumeFlu/logs/nginxLogAddpartition.log
then
echo "nginxLogAddpartition $tomorrow failed !!!" | mail -s '[ERROR] Flume nginxLogAddpartition' XX@pentaKill.com
else
echo "nginxLogAddpartition $tomorrow success !!!" | mail -s '[INFO] Flume nginxLogAddpartition' XX@pentaKill.com
fi
}

add_partition
send_email

# 删除分区
# alter table ods.xxx drop if exists partition(day='2020907');

4.重复启动hive JVM 优化

上面的方式没建立一个分区都要启动hive客户端,即启动一个JVM,非常耗时,如果将所有建分区语句放入到一个add_partition.sql中,使用hive -f执行,速度将快几倍.

  #!/bin/bash 

timeone=`date -d "+1 day" +%Y-%m-%d`

#提前创建2天ods.XXX与odsXXX分区
for i in {0..47}
do
days=`date -d "$timeone $i hour" +%Y%m%d`
hours=`date -d "$timeone $i hour" +%H`

echo "alter table ods.XXX add if not exists partition (day='$days',hour='$hours');" >> /opt/add_partition.sql
echo "alter table ods.XXX add if not exists partition (day='$days',hour='$hours');" >> /opt/add_partition.sql

done 

/hive/apache-hive-XXX-bin/bin/hive -f /opt/add_partition.sql

rm -f /opt/add_partition.sql

5.shell中年月日去除月日前的0

year=`date +%Y`
tmonth=`date +%m`
tday=`date -d "-1 day" +%d`
month=`expr $tmonth + 0`   #去除月份前面的零
day=`expr $tday + 0`       #去除日期前面的零

statis_time=`printf "%04d%d%d" $year $month $day`

echo $statis_time