linux 上批量下载amazon s3数据
linux 上批量下载amazon s3数据
最近导师喊我下载amazon s3上的数据传到服务器上,没用过这玩意,网上那些批量下载的方法又看不懂,就自己用python和shell写了个小的批量下载的脚本
在运行之前要确认环境中有python3,然后在运行路劲底下放个conf.txt配置文件已经用pip install awscli下载所需要的awscli包,
conf.txt内容如下 在什么路劲下运行conf.txt就放到那个路劲底下
第一行为协议,第二行为Access Key,第三行为Secret,第四行为远程路径
s3.cn-northwest-1.amazonaws.com.cn
AKIA6VFASDFGREWEGSZAB25X
AL1pZubEyiQW123FDQ9rHQWEqGLvFMOUFSDFE2
partrsonbaio-release/P132531231/
python脚本aws_download内容如下
#!/home/anaconda3/bin/python ###你自己的python解释器路劲
import os
import subprocess
configfname = os.getcwd() + '/' + "conf.txt"
conf = open(configfname)
content = conf.readlines()
region = content[0].split('.')[1]
key = content[1].strip()
secret = content[2].strip()
remote_path = content[3].strip()
remote_path = 's3://' + remote_path
aws_path = "aws"
def for_conf():
script_path = "/".join(__file__.split('/')[:-1])
os.system(f'sh {script_path}/aws_configure.sh {key} {secret} {region}')
def download(path):
p = subprocess.Popen(f'{aws_path} s3 ls {path}', shell=True, stdout=subprocess.PIPE)
out,err = p.communicate()
for line in out.splitlines():
line = line.decode()
parts = line.strip().split()
if parts[-2].isdecimal():
remote_f = path + parts[-1]
local_path = os.getcwd() + '/' + path.split(remote_path)[1]
os.system(f"{aws_path} s3 cp {remote_f} {local_path}")
else:
temp_path = path + parts[-1]
local_path = temp_path.split(remote_path)[1]
os.system(f'mkdir -p {local_path}')
download(temp_path)
def main():
download(remote_path)
if __name__ == '__main__':
for_conf()
main()
shell脚本aws_configure.sh内容如下
key=$1
secret=$2
region=$3
/usr/bin/expect <<-EOF
spawn aws configure
expect "AWS Access Key ID *"
send "$key\r"
expect "AWS Secret Access Key*"
send "$secret\r"
expect "Default region name*"
send "$region\r"
expect "Default output forma*:"
send "\r"
expect eof
EOF
将python脚本aws_download以及shell脚本aws_configure.sh放在同一路劲下(例如我放在了/home/users/a/software/aws_download这个路劲下)然后将用chmod +x aws_download给python脚本执行权限,再在~/.bashrc中添加一句export PATH="/home/users/a/software/aws_download:$PATH"将写好的脚本放在环境变量中
运行效果如下
![在这里插入图片描述](https://img-blog.csdnimg.cn/f8e90342eebe4f0bb45343f10861515b.png
脚本还会自动在本地创建跟远程一样的路劲进行下载