AWS工具boto3使用笔记

通过适用于 Python 的 AWS 开发工具包 boto3 快速开始使用 AWS。Boto3 可以支持您轻松将 Python 应用程序、库或脚本与 AWS 服务进行集成，包括 Amazon S3、Amazon EC2 和 Amazon DynamoDB 等。

1. 上传和下载数据

class CephBOTO3:
    def __init__(self):
        self.access_key = [your access key]
        self.secret_access_key = [your secret access key]
        self.bucket = [your bucket]
        self.endpoint = [your endpoint]
        self.service_name = [your service_name]

        self.session = boto3.Session(aws_access_key_id=self.access_key, aws_secret_access_key=self.secret_access_key)
        self.client = self.session.client(service_name=self.service_name, use_ssl=False, endpoint_url=self.endpoint)
        self.resource = boto3.resource(self.service_name, endpoint_url=self.endpoint,
                                       aws_access_key_id=self.access_key,
                                       aws_secret_access_key=self.secret_access_key)

    def upload(self, directory):
        file_paths = []
        for root, dirs, filenames in os.walk(directory):
            for name in filenames:
                file_path = os.path.join(root, name)
                file_paths.append((name, file_path))

        for file_name, file_path in file_paths:
            print("Uploading {}".format(file_name))
            self.resource.meta.client.upload_file(file_path, self.bucket, file_name)

    def download(self, directory):
        bucket = self.resource.Bucket(self.bucket)
        keys = self.get_buckets()
        for key in keys:
            file_name = directory + '/' + key
            bucket.download_file(Filename=file_name, Key=key)

2. 上传和下载用进度条

有时上传或者下载的数据较大，用户需要一个进度条，只需要定义回调类

class ProgressPercentage(object):
    def __init__(self, file_name, file_path):
        self.file_name = file_name
        self.file_path = file_path
        self._size = float(os.path.getsize(file_path))
        self._seen_so_far = 0
        self._lock = threading.Lock()

    def __call__(self, bytes_amount):
        # To simplify, assume this is hooked up to a single filename
        with self._lock:
            self._seen_so_far += bytes_amount
            percentage = (self._seen_so_far / self._size) * 100
            sys.stdout.write("\r正在上传 %s (%.2f%%)" % (self.file_name, percentage))
            sys.stdout.flush()

 然后在将其加入到上传下载函数中作为回调

bucket.upload_file(Filename=file_path, Key=file_name, Callback=ProgressPercentage(file_name, file_path))

3. 把数据存放在远端指定目录下

在传参时使用key来指定目录

key = "A/B/C/{}".format(file_name)
bucket.upload_file(Filename=file_path, Key=key, Callback=ProgressPercentage(file_name, file_path))

上面的代码运行后，文件会存放在远端的A/B/C目录下

参考文献

适用于python的AWS开发工具包Boto3_AWS SDK工具包-AWS云服务