[Linux]结合awk删除hdfs指定日期前的数据

业务背景

约定五天前的HDFS数据为过期版本号数据。写一个脚本自己主动删除过期版本号数据

$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data
Found 9 items
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-01
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-02
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-03
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-04
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-05
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09

脚本实现

# ---------------------------------------------------------
#
# 删除历史版本号（五天前的为过期版本号数据）
#
# ---------------------------------------------------------

old_version=$(hadoop fs -ls /user/pms/workspace/ouyangyewei/data | awk 'BEGIN{ five_days_ago=strftime("%F", systime()-5*24*3600) }{ split($8,arr,"/"); if(arr[7]<five_days_ago){printf "%s\n", $8} }')
arr=(${old_version// / })
for version in ${arr[@]} 
do 
    hadoop fs -rmr $version
done

运行以后

$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data
Found 4 items
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08
drwxr-xr-x   - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09

转载于:https://www.cnblogs.com/jzdwajue/p/7218361.html