1.log.conf,python的日志模块,详见:http://www.red-dove.com/python_logging.html
[loggers]
keys=root,upload,resize
[handlers]
keys=consoleHandler,uploadFileHandler,resizeFileHandler
[formatters]
keys=simpleFormatter
[formatter_simpleFormatter]
format=%(asctime)s %(levelname)s %(message)s
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_upload]
level=DEBUG
handlers=uploadFileHandler
qualname=upload
[logger_resize]
level=DEBUG
handlers=resizeFileHandler
qualname=resize
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)
[handler_uploadFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('upload.log',"midnight", 1)
[handler_resizeFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('resize.log',"midnight", 1)
跟java中的log4j很类似,定义三个logger对象:root,upload,resize,其中root是StreamHandler类型,使用sys.out;upload和resize是TimedRotatingFileHandler,每天生成一个日志文件的方式。
2.conf.py配置文件
FILE_BIG_PATH = '/data/hiluofilev2/big/'
FILE_SMALL_PATH = '/data/hiluofilev2/small/'
HEAD_BIG_PATH = '/data/hiluohead/big/'
HEAD_SMALL_PATH = '/data/hiluohead/small/'
GEARMAN_SERVER = '127.0.0.1:4730'
RESIZE_WORKER = 'resize'
FILE_RESIZE = '140x160!'
HEAD_RESIZE = '86x86!'
IMAGE_FILE = ['jpg','png','gif','jpeg']
AUDIO_FILE = ['amr']
HTTP_FILE = 'http://192.168.0.181:9999/file/'
HTTP_HEAD = 'http://192.168.0.181:9999/head/'
import logging
import logging.config
import os
logging.config.fileConfig(os.path.join(os.path.dirname(os.path.abspath(__file__)), "log.conf"))
FILE_BIG_PATH:这几个PATH表示原图和缩略图保存路径,根据业务区分,用户头像和用户上传的文件分开存放。
GEARMAN_SERVER:gearman Server端的地址。
RESIZE_WORKER:执行图片缩放的worker的名字。
FILE_RESIZE和HEAD_RESIZE:普通图片和头像的缩放尺寸。
IMAGE_FILE和AUDIO_FILE:系统支持的图片格式和音频文件格式,业务上仅支持两种类型。
HTTP_FILE和HTTP_HEAD:文件下载的HTTP前缀。
然后是初始化logging的功能。
3.worker_resize.py 生成缩略图的worker
#!/usr/bin/env python
import os
import gearman
import math
import conf
import logging
import pickle
from pgmagick import Image, FilterTypes
log_resize = logging.getLogger("resize")
class CustomGearmanWorker(gearman.GearmanWorker):
def on_job_execute(self, current_job):
log_resize.info("Resize Worker started")
return super(CustomGearmanWorker, self).on_job_execute(current_job)
def resize_task_callback(gearman_worker, job):
filein = job.data.split('-')[0]
fileout = job.data.split('-')[1]
fileresize = job.data.split('-')[2]
log_resize.info('start filein:%s -> fileout:%s' % (filein,fileout))
im = Image(filein)
im.quality(80)
im.filterType(FilterTypes.SincFilter)
im.scale(fileresize)
im.sharpen(1.0)
im.write(fileout)
log_resize.info('end filein:%s -> fileout:%s' % (filein,fileout))
return job.data
new_worker = CustomGearmanWorker([conf.GEARMAN_SERVER])
new_worker.register_task("resize", resize_task_callback)
new_worker.work()
jobdata是string,filein代表原图路径,fileout代表缩略图路径,fileresize代表缩放尺寸
4.upload.py 用于上传文件和测试上传文件,使用web.py实现
#!/usr/bin/env python
# coding: utf-8
import web
import conf
import logging
import gearman
urls = (
'/', 'Index',
'/upload', 'FileUpload'
)
app = web.application(urls, globals())
log_upload = logging.getLogger("upload")
# get some directories in conf file
filebigdir = conf.FILE_BIG_PATH
filesmalldir = conf.FILE_SMALL_PATH
headbigdir = conf.HEAD_BIG_PATH
headsmalldir = conf.HEAD_SMALL_PATH
resize_client = gearman.GearmanClient([conf.GEARMAN_SERVER])
class Index:
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return """<html><head></head><body>this hiluofileV2</body></html>"""
class FileUpload:
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return """<html><head></head><body>
<form method="POST" enctype="multipart/form-data" action="">
username:<input type="text" name="username" size="20" /><br/>
md5 or sha1:<input type="text" name="md5orsha1" size="80"/><br/>
<input type="file" name="myfile" />
<br/>
<input type="submit" />
</form>
</body></html>"""
def POST(self):
client = web.ctx.environ['REMOTE_ADDR']
x = web.input(myfile={},username="",md5orsha1="")
username = x.username.encode('utf-8')
intusername = 0
try:
intusername = int(username) # check if the username is right
except ValueError:
log_upload.error('ip:%s username:%s username not right!' % (client,username))
return 'error[username]'
md5orsha1 = x.md5orsha1.encode('utf-8') # check if the md5orsha1 is right
if len(md5orsha1)==32 or len(md5orsha1)==40:
if 'myfile' in x: # check if the file-object is created
filepath = x.myfile.filename.replace('\\','/')
filename = filepath.split('/')[-1]
filesuffix = filename.split('.')[-1].lower()
filename = md5orsha1.lower() + '.' + filesuffix # get saved filename(md5orsha1+‘.’+suffix)
filebig = filebigdir + filename
if filesuffix in conf.AUDIO_FILE: # audio file just save no need to resize
fout = open(filebig,'w') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
log_upload.info('ip:%s username: %d upload: %s ok!' % (client,intusername,filename))
fileurl = conf.HTTP_FILE + 'big/' + filename # file download url
return fileurl
elif filesuffix in conf.IMAGE_FILE:
filesmall = filesmalldir + filename # thumbnail file
fileresize = conf.FILE_RESIZE # default FILE_RESIZE
fileurl = conf.HTTP_FILE + 'small/' + filename
if len(md5orsha1)==40: # user upload head
filebig = headbigdir + filename
filesmall = headsmalldir + filename # thumbnail head
fileresize = conf.HEAD_RESIZE # thumbnail head HEAD_RESIZE
fileurl = conf.HTTP_HEAD + 'small/' + filename
fout = open(filebig,'w') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
log_upload.info('ip:%s username:%d upload:%s ok!' % (client,intusername,filename))
jobdata = ('%s-%s-%s' % (filebig,filesmall,fileresize))
resize_client.submit_job(conf.RESIZE_WORKER, jobdata) # submit_job resize and ignore job result
log_upload.info('submit resize:%s ok!' % (jobdata))
return fileurl
else:
log_upload.error('ip:%s username:%d upload:%s unsupport file type' % (client,intusername,filename))
return 'error[filetype]'
else:
log_upload.error('ip:%s username:%d upload no myfile' % (client,intusername))
return 'error[myfile]'
else:
log_upload.error('ip:%s username:%d upload:%s unsupport md5orsha1' % (client,intusername,md5orsha1))
return 'error[filemd5orsha1]'
if __name__ == "__main__":
#app = web.application(urls, globals())
web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)
app.run()
这里业务上定义传头像的key是40位的sha1编码,普通图像文件是32位的md5编码。
5.nginx的nginx.conf修改,使用fastCGI方式运行,
在http节点下增加:
proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; #获取真实IP client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k;
增加fastCGI静态图片的下载location,nginx开启8099端口代理fastCGI,9999端口用来下载文件
server{ listen 8099; server_name fast; charset utf-8; location / { fastcgi_param REQUEST_METHOD $request_method; fastcgi_param QUERY_STRING $query_string; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx/$nginx_version; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param SCRIPT_FILENAME $fastcgi_script_name; fastcgi_param PATH_INFO $fastcgi_script_name; fastcgi_pass 127.0.0.1:8091; } } server { listen 9999; server_name file.hiluo.cn; location /file/ { alias /data/hiluofilev2/; } location /head/ { alias /data/hiluohead/; } }
注意,在配置nginx的fastcgi时可能会报错如下:
5.1.child exited with 2
解决方法: insert #!/usr/bin/env python into header of main.py
5.2.spawn-fcgi child exited with 126
解决方法: chmod +x upload.py
5.3.child exited with1
这时可以先修改upload.py,屏蔽web.wsgi.runwsgi行,打开app = web.application(urls, globals())行,用python upload.py 8091方式启动测试。
6.hiluo-file-http.sh,runwsgi的启动停止管理脚本
#!/bin/sh # # hiluo-file-http control # $Date: 2012-07-04 willzhai $ # # If pid file path is not set elsewhere, set to /tmp/hiluo-file-http.pid [ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-http.pid" dir=/root/test-file-v2 start() { echo "Starting hiluo-file-http..." exec_result=`spawn-fcgi -d $dir -f $dir/upload.py -a 127.0.0.1 -p 8091` echo $exec_result if [ ! -z "$PIDFILE" ]; then echo $exec_result|awk -F ':' '{print $4}' > $PIDFILE fi } stop() { # Stop daemons. echo "Shutting down hiluo-file-http..." kill `pgrep -f "python $dir/upload.py"` rm -f $PIDFILE } restart() { stop sleep 3 # give it a few moments to shut down start } status() { pid=`cat $PIDFILE 2>&1` if [ "$?" = "1" ]; then echo "hiluo-file-http is not running" RETVAL=0 else ps -p $pid > /dev/null 2>&1 if [ "$?" = "0" ]; then echo "hiluo-file-http is running" RETVAL=0 else echo "hiluo-file-http is not running" RETVAL=0 fi fi } case "$1" in start) start ;; stop) stop ;; restart) restart ;; status) status ;; *) echo "Usage $0 {start|stop|restart|status}" RETVAL=1 esac exit $?
注意将dir改成实际的部署路径。这个可以再改进,自动识别当前路径。
7.hiluo-file-resize.sh ,resize worker的启动停止管理脚本
#!/bin/sh # # hiluo-file-resize control # $Date: 2012-07-04 willzhai $ # # If pid file path is not set elsewhere, set to /tmp/hiluo-file-resize.pid [ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-resize.pid" dir=/root/test-file-v2 start() { echo "Starting hiluo-file-resize..." exec_command="exec python $dir/worker_resize.py 2>&1 &" eval $exec_command RETVAL=$? if [ $RETVAL -eq 0 -a ! -z "$PIDFILE" ]; then echo $! > $PIDFILE fi echo "hiluo-file-resize new pid: "`cat $PIDFILE` } stop() { # Stop daemons. echo "Shutting down hiluo-file-resize..." kill `pgrep -f "python $dir/worker_resize.py"` rm -f $PIDFILE } restart() { stop sleep 3 # give it a few moments to shut down start } status() { pid=`cat $PIDFILE 2>&1` if [ "$?" = "1" ]; then echo "hiluo-file-resize is not running" RETVAL=0 else ps -p $pid > /dev/null 2>&1 if [ "$?" = "0" ]; then echo "hiluo-file-resize is running" RETVAL=0 else echo "hiluo-file-resize is not running" RETVAL=0 fi fi } case "$1" in start) start ;; stop) stop ;; restart) restart ;; status) status ;; *) echo "Usage $0 {start|stop|restart|status}" RETVAL=1 esac exit $?
注意将dir改成实际的部署路径。这个可以再改进,自动识别当前路径。
8.效果如下
8.1上传页面
8.2上传成功页面
8.3缩略图页面
文件打包下载见附件!
参考:
http://webpy.org/cookbook/storeupload/
http://webpy.org/cookbook/fastcgi-nginx
http://www.pythonclub.org/python-basic/string
http://hi.baidu.com/thinkinginlamp/blog/item/4b61e9241f08820f4c088d95.html