Amazon Simple Storage Service (Amazon S3) 是一种对象存储服务,提供行业领先的可扩展性、数据可用性、安全性和性能。这意味着各种规模和行业的客户都可以使用 S3 来存储并保护各种用例(如数据湖、网站、移动应用程序、备份和还原、存档、企业应用程序、IoT 设备和大数据分析)的数据,容量不限, S3 可达到 99.999999999%(11 个 9)的持久性.
需求背景
原amazon的s3上传比较繁琐,对于需要经常上传并且获取s3url的需求(国际产品开发常见)不太友好,因此希望使用代码上传
代码实现
关键信息可以自己登陆amonzon获取。地址为:
https://console.aws.amazon.com/iam/home?region=ap-east-1#/security_credentials
关于如何获取amazon的accesskey这里不详细介绍...
话不多说,代码如下:
import os
import datetime
import bson
import boto3
def s3back(res):
print("【aws回调函数】",res)
def s3Upload(_path, _type):
'''
上传指定文件到aws s3
:param _path: 文件的绝对路径
:param _type: 文件的content-type,例如image/png这种,可以通过filetype包获取
:return: aws s3的url地址,可以直接访问并获取到文件
'''
#[获取地址](https://console.aws.amazon.com/iam/home?region=ap-east-1#/security_credentials)
if not os.path.exists(_path): #文件不存在
raise FileNotFoundError("this file not exists!")
print("s3完整文件路径:%s, 文件类型类型:%s" % (_path, _type))
AwsAccessKeyId = "AAAAAAAAAAAAAAAAAAAA" #【换成自己的】20位长度大写字母与数字的混合
AwsSecretAccessKey = "00000000000000000000000000000000000000000000000000" #【换成自己的】40位长度大小写字母 + 特殊符号 + 数字的混合
S3Region = "ap-southeast-1"
S3Bucket = "webtest-ios-upload-pro"
try:
session = boto3.Session(
aws_access_key_id = AwsAccessKeyId,
aws_secret_access_key = AwsSecretAccessKey,
region_name = S3Region
)
s3 = session.client("s3")
#ACL参数可选: 'private'|'public-read'|'public-read-write'|'authenticated-read' (私有|公共读取|公共读写|认证读取 )
extraData = {
'ACL': 'public-read',
"ContentType": _type,
"ServerSideEncryption":"AES256",
"StorageClass": "INTELLIGENT_TIERING"
}
#bson.ObjectId() 创建一个24位的随机ID,类似UUID那种
#key用于唯一标识aws s3存储桶中的文件,防止文件冲突,会使用uuid与时间的方式标识key。也可以自定义命名
key = datetime.datetime.now().strftime("%Y%m%d") + "/" + str(bson.ObjectId()) + "." + _path.split(".")[-1]
#upload没有返回结果,但可以通过回调函数,返回一个任务的ID
s3.upload_file(Filename=_path, Bucket=S3Bucket, Key=key, ExtraArgs=extraData, Callback=s3back)
uploadURL = "https://%s.s3.%s.amazonaws.com/%s" % (S3Bucket, S3Region, key)
return uploadURL
except Exception as e:
raise e
if __name__=="__main__":
s3Upload("./test.png", "image/png")
方法源码
因为一开始也是参考网上的用例,和实际实现有些出入,比如s3.upload_file方法在pydoc与官网说明中都是extra_args参数,但实际运行会报错:
TypeError: upload_file() got an unexpected keyword argument 'extra_args'
然后通过help查看方法的源码,才发现这个参数实际是ExtraArgs。
>>> help(s3.upload_file)
Help on method upload_file in module boto3.s3.inject:
upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None) method of botocore.client.S3 instance
Upload a file to an S3 object.
Usage::
import boto3
s3 = boto3.resource('s3')
s3.meta.client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Similar behavior as S3Transfer's upload_file() method,
except that parameters are capitalized. Detailed examples can be found at
:ref:`S3Transfer's Usage <ref_s3transfer_usage>`.
:type Filename: str
:param Filename: The path to the file to upload.
:type Bucket: str
:param Bucket: The name of the bucket to upload to.
:type Key: str
:param Key: The name of the key to upload to.
:type ExtraArgs: dict
:param ExtraArgs: Extra arguments that may be passed to the
client operation.
:type Callback: function
:param Callback: A method which takes a number of bytes transferred to
be periodically called during the upload.
:type Config: boto3.s3.transfer.TransferConfig
:param Config: The transfer configuration to be used when performing the
transfer.
结果论证
如果accesskey等键值存在问题,代码运行会直接报错。报错信息比较友好,翻译基本能看懂。
其他问题:
Access denied
与上传的权限有关,主要由ACL参数控制访问权限,如果登录的用户与上传的用户key不一致,可能导致访问拒绝。另外publiced-read是全部公开权限,所有人都可以访问