python upload_module_nginx-upload-module模块实现文件上传(multipart/form-data和断点续传)

前言

有时候我们想简单实现文件上传的功能，又不想使用额外的语言(比如PHP、Java)，或者想实现文件的断点续传。这个时候Nginx的一个模块nginx-upload-module就能满足我们的需求。

模块安装

下载模块：

cd /tmp

wget https://codeload.github.com/vkholodkov/nginx-upload-module/zip/2.2

unzip 2.2

安装模块：

.configure --add-module=/tmp/nginx-upload-module-2.2/

multipart/form-data表单上传示例

nginx.conf配置：

server {

[...]

location /upload {

upload_pass @uploadHandler;

upload_store /usr/local/nginx/upload_temp 1;

upload_set_form_field $upload_field_name.path "$upload_tmp_path";

}

location @uploadHandler {

proxy_pass http://backend-host;

}

[...]

}

这里在server里定义了upload location，这个location是上传的接口，还有@uploadHandler location，是当文件上传完成后，nginx模块会对这个location发送一些必要的信息，如文件上传的路径。

这里涉及了几个指令：

upload_pass @uploadHandler： 上传完成后会发送必要的数据到@uploadHandler;

upload_store /usr/local/nginx/upload_temp 1: 文件上传的临时目录;

upload_set_form_field $upload_field_name.path “$upload_tmp_path”: 设置文件上传完成后，把文件临时路径发送给upload_pass指定的location。

断点续传示例

nginx.conf配置

server {

[...]

location /resumable_upload {

upload_resumable on;

upload_state_store /usr/local/nginx/upload_temp ;

upload_pass @drivers_upload_handler;

upload_store /usr/local/nginx/upload_temp;

upload_set_form_field $upload_field_name.path "$upload_tmp_path";

}

location @resumable_upload_handler {

proxy_pass http://localhost:8002;

}

[...]

}

与上一步multipart/form-data表单上传示例配置不同的地方有：

upload_resumable on: 开启断点续传功能;

upload_state_store /usr/local/nginx/upload_temp: 设置断点续传状态文件存储的目录。

上传文件第一个片段

POST /upload HTTP/1.1

Host: example.com

Content-Length: 51201

Content-Type: application/octet-stream

Content-Disposition: attachment; filename="big.TXT"

X-Content-Range: bytes 0-51200/511920

Session-ID: 1111215056

<0-51200的字节文件数据>

上传文件第一个片段服务器响应

HTTP/1.1 201 Created

Date: Thu, 02 Sep 2010 12:54:40 GMT

Content-Length: 14

Connection: close

Range: 0-51200/511920

0-51200/511920

上传文件最后一个片段

POST /upload HTTP/1.1

Host: example.com

Content-Length: 51111

Content-Type: application/octet-stream

Content-Disposition: attachment; filename="big.TXT"

X-Content-Range: bytes 460809-511919/511920

Session-ID: 1111215056

<460809-511919字节文件数据>

上传文件最后一个片段服务器响应

HTTP/1.1 200 OK

Date: Thu, 02 Sep 2010 12:54:43 GMT

Content-Type: text/html

Connection: close

Content-Length: 2270

请求头说明

请求头

说明

Content-Disposition

attachment, filename=“上传的文件名”

Content-Type

待上传文件的mime type,如application/octet-stream(注:不能为multipart/form-data)

X-Content-Range

待上传文件字节范围,如第一片段bytes 0-51200/511920,最后一个片段bytes 460809-511919/511920(注:文件第一个字节标号为0,最后一个字节标号为n-1,其中n为文件字节大小)

上传文件的标识,由客户端随机指定.因为是断点续传,客户端必须确保同一个文件的所有片段上传标识一致

Content-Length

上传片段的大小

Python上传demo

#!/usr/bin/python

# -*- coding: utf-8 -*-

import os.path

import requests

import hashlib

# 待上传文件路径

FILE_UPLOAD = "/tmp/testfile"

# 上传接口地址

UPLOAD_URL = "http://host/drivers_upload"

# 单个片段上传的字节数

SEGMENT_SIZE = 1048576

def upload(fp, file_pos, size, file_size):

session_id = get_session_id()

fp.seek(file_pos)

payload = fp.read(size)

content_range = "bytes {file_pos}-{pos_end}/{file_size}".format(file_pos=file_pos,

pos_end=file_pos+size-1,file_size=file_size)

headers = {'Content-Disposition': 'attachment; filename="big.TXT"','Content-Type': 'application/octet-stream',

'X-Content-Range':content_range,'Session-ID': session_id,'Content-Length': size}

res = requests.post(UPLOAD_URL, data=payload, headers=headers)

print(res.text)

# 根据文件名hash获得session id

def get_session_id():

m = hashlib.md5()

file_name = os.path.basename(FILE_UPLOAD)

m.update(file_name)

return m.hexdigest()

def main():

file_pos = 0

file_size = os.path.getsize(FILE_UPLOAD)

fp = open(FILE_UPLOAD,"r")

while True:

if file_pos + SEGMENT_SIZE >= file_size:

upload(fp, file_pos, file_size - file_pos, file_size)

fp.close()

break

else:

upload(fp, file_pos, SEGMENT_SIZE, file_size)

file_pos = file_pos + SEGMENT_SIZE

if __name__ == "__main__":

main()

参考