【linux 上批量下载amazon s3数据】

linux 上批量下载amazon s3数据

最近导师喊我下载amazon s3上的数据传到服务器上，没用过这玩意，网上那些批量下载的方法又看不懂,就自己用python和shell写了个小的批量下载的脚本
在运行之前要确认环境中有python3，然后在运行路劲底下放个conf.txt配置文件已经用pip install awscli下载所需要的awscli包，

conf.txt内容如下 在什么路劲下运行conf.txt就放到那个路劲底下
第一行为协议，第二行为Access Key，第三行为Secret，第四行为远程路径

s3.cn-northwest-1.amazonaws.com.cn
AKIA6VFASDFGREWEGSZAB25X
AL1pZubEyiQW123FDQ9rHQWEqGLvFMOUFSDFE2
partrsonbaio-release/P132531231/

python脚本aws_download内容如下

#!/home/anaconda3/bin/python    ###你自己的python解释器路劲
import os
import subprocess


configfname = os.getcwd() + '/' + "conf.txt"
conf = open(configfname)
content = conf.readlines()
region = content[0].split('.')[1]
key = content[1].strip()
secret = content[2].strip()
remote_path = content[3].strip()
remote_path = 's3://' + remote_path
aws_path = "aws"

def for_conf():
    script_path = "/".join(__file__.split('/')[:-1])
    os.system(f'sh {script_path}/aws_configure.sh {key} {secret} {region}')


def download(path):
    p = subprocess.Popen(f'{aws_path} s3 ls {path}', shell=True, stdout=subprocess.PIPE)
    out,err = p.communicate()
    for line in out.splitlines():
        line = line.decode()
        parts = line.strip().split()
        if parts[-2].isdecimal():
            remote_f = path + parts[-1]
            local_path = os.getcwd() + '/' + path.split(remote_path)[1]
            os.system(f"{aws_path} s3 cp {remote_f} {local_path}")
        else:
            temp_path = path + parts[-1]
            local_path = temp_path.split(remote_path)[1]
            os.system(f'mkdir -p {local_path}')
            download(temp_path)


def main():
    download(remote_path)


if __name__ == '__main__':
    for_conf()
    main()

shell脚本aws_configure.sh内容如下

key=$1
secret=$2
region=$3


/usr/bin/expect <<-EOF
spawn aws configure

expect "AWS Access Key ID *"
send "$key\r"
expect "AWS Secret Access Key*"
send "$secret\r"
expect "Default region name*"
send "$region\r"
expect "Default output forma*:"
send "\r"
expect eof
EOF

将python脚本aws_download以及shell脚本aws_configure.sh放在同一路劲下（例如我放在了/home/users/a/software/aws_download这个路劲下）然后将用chmod +x aws_download给python脚本执行权限，再在~/.bashrc中添加一句export PATH="/home/users/a/software/aws_download:$PATH"将写好的脚本放在环境变量中

运行效果如下
![在这里插入图片描述](https://img-blog.csdnimg.cn/f8e90342eebe4f0bb45343f10861515b.png

脚本还会自动在本地创建跟远程一样的路劲进行下载