最近在做http使用POST请求上传图片到阿里云对象存储oss的时候发生一件很诡异的事情!
我的环境是python3.5+django11.6+gunicorn部署的web后台服务settings文件logging配置如下:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse'
}
},
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(module)s '
'%(process)d %(thread)d %(message)s'
},
'simple': {
'format': '%(message)s',
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
'report_data': {
'level': 'INFO',
'class': 'xxx.libs.logging.handlers.TimedRotatingFileHandlerMultiProcess',
'filename': '/var/log/xxx/info.log',
'formatter': 'simple'
},
},
'loggers': {
'django.security.DisallowedHost': {
'level': 'ERROR',
'handlers': ['console'],
'propagate': True
},
'info': {
'level': 'INFO',
'handlers': ['report_data'],
},
...
}
}
views.py日志记录部分代码如下:
@api_view(['POST'])
@permission_classes((TokenHasReadWriteScope, ))
def report_data_view(request, format=None):
report_header = request.META
if request.method == 'POST':
report_data = request.data
logger = logging.getLogger("info")
logger.info(report_data)
...
按道理是POST请求提交数据将会被记录到/var/log/xxx/info.log中,实际上也确实正常运行。(虽然这样做有一点另类),但是加上文件上传功能之后就出问题了:
客户端图片上传部分代码如下:
def upload_images(task):
if task.success and len(task.result):
token_string = _get_oauth_token(local=True)
headers = {'Authorization': token_string, 'Image': 'True'}
for item in task.result:
data = {"id": item['id']}
files = {}
for image_url, image_path in item['files'].items():
files[image_url] = (open(image_path, 'rb'))
res = requests.post(AGENT_UPLOAD_URI, headers=headers, data=data, files=files)
if res.status_code == 201:
status = 2 # 表示上传成功
else:
status = 1 # 表示上传失败
update_uploaded_status(item['table_name'], item['course_id'], item['url_path_str'], status)
服务器端接收图片以及上传oss部分代码如下:
@api_view(['POST'])
@permission_classes((TokenHasReadWriteScope, ))
def upload_image_view(request, format=None):
if request.method == 'POST':
if 'HTTP_IMAGE' in report_header.keys():
if not request.FILES:
return Response(None, status=status.HTTP_200_OK)
else:
id = request.data['id'] # 形如"201709168888"
date = '-'.join([id[0:4], id[4:6], id[6:8]])
try:
for url_path, fd in request.FILES.items():
tmp_dir = os.path.join('/tmp/', get_hash_val(os.path.dirname(url_path)))
if not os.path.exists(tmp_dir):
os.mkdir(tmp_dir)
file_path = os.path.join(tmp_dir, os.path.basename(url_path))
with open(file_path, 'wb+') as f:
for chrunk in fd.chunks():
f.write(chrunk)
upload_image(date, url_path, file_path)
return Response(None, status=status.HTTP_201_CREATED)
except Exception as e:
print("Unexpected error: {0}".format(e))
...
def upload_image(date, key, file_path):
auth = oss2.Auth(ALIYUN_ACCESS_KEY_ID, ALIYUN_ACCESS_KEY_SECRET)
bucket = oss2.Bucket(auth, ALIYUN_OSS_SZ_ENDPOINT, OSS_IMAGE_BUCKET)
try:
oss2.resumable_upload(bucket, key, file_path,
store=oss2.ResumableStore(root='/tmp'),
multipart_threshold=100 * 1024,
part_size=100 * 1024,
num_threads=5)
except oss2.exceptions.OssError as e:
print("Upload image[{0}] faild. Details:{1}.".format(url_path, str(e)))
两个几乎完全没有联系的功能模块凑一块就搞出了幺蛾子,gunicorn日志中竟然莫名其妙记录了POST请求发送的数据!虽然问题不是很严重,但是久而久之文件大小会越来越大占用磁盘,而且总感觉像埋了地雷一样不让人舒坦。没办法只能一段一段注释代码再运行测试!最后发现下载写到文件但不上传oss是没有问题的,大胆猜测是oss文件上传导致http请求中的header变化了?因为文件上传是“Content-Type: multipart/form-data”,其他请求是“Content-Type: application/json”。或者环境变量发生改变之类的?解决方案是新增一个进程调用upload_image函数:
def upload_image_view(request, format=None):
...
#upload_image(date, url_path, file_path)
p = multiprocessing.Process(target=upload_image, args=(date, url_path, file_path))
p.start()
...
最后也没能去验证这个想法,写第一篇博客提醒自己留心注意这个问题同时记录一下关于python文件上传和以及oss模块的使用方法。
新手上路,请多多指教!