linux删除数据用的时间,Linux基础教程：结合awk删除hdfs指定日期前的数据

业务背景

约定五天前的HDFS数据为过期版本数据，写一个awk脚本自动删除过期版本数据

$ Hadoop fs -ls /user/pms/workspace/ouyangyewei/data

Found 9 items

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-01

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-02

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-03

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-04

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-05

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09

脚本实现

# ---------------------------------------------------------

#

# 删除历史版本(五天前的为过期版本数据)

#

# ---------------------------------------------------------

old_version=$(hadoop fs -ls /user/pms/workspace/ouyangyewei/data | awk 'BEGIN{ five_days_ago=strftime("%F", systime()-5*24*3600) }{ split($8,arr,"/"); if(arr[7]

arr=(${old_version// / })

for version in ${arr[@]}

do

hadoop fs -rmr $version

done

执行以后

$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data

Found 4 items

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08

drwxr-xr-x  - pms pms          0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09