python运行时间长,我如何在Python的后台运行一个长时间运行的工作

I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.

What I am thinking of doing is to have the route which takes a long time to complete, call a function that creates a thread. The function will then return a guid back to the route, and the route will return a url (using the guid) that the user can use to check progress. I am making the thread a daemon (thread.daemon = True) so that the thread exits if my calling code exits (unexpectedly).

Is this the correct approach to use? It works, but that doesn't mean that it is correct.

my_thread = threading.Thread(target=self._run_audit, args=())

my_thread.daemon = True

my_thread.start()

解决方案

The more regular approch to handle such issue is extract the action from the base application and call it outside, using a task manager system like Celery.

Using this tutorial you can create your task and trigger it from your web application.

from flask import Flask

app = Flask(__name__)

app.config.update(

CELERY_BROKER_URL='redis://localhost:6379',

CELERY_RESULT_BACKEND='redis://localhost:6379'

)

celery = make_celery(app)

@celery.task()

def add_together(a, b):

return a + b

Then you can run:

>>> result = add_together.delay(23, 42)

>>> result.wait()

65

Just remember you need to run worker separately:

celery -A your_application worker