celery 连接 mysql,Celery Worker数据库连接池

I am using Celery standalone (not within Django). I am planning to have one worker task type running on multiple physical machines. The task does the following

Accept an XML document.

Transform it.

Make multiple database reads and writes.

I'm using PostgreSQL, but this would apply equally to other store types that use connections. In the past, I've used a database connection pool to avoid creating a new database connection on every request or avoid keeping the connection open too long. However, since each Celery worker runs in a separate process, I'm not sure how they would actually be able to share the pool. Am I missing something? I know that Celery allows you to persist a result returned from a Celery worker, but that is not what I'm trying to do here. Each task can do several different updates or inserts depending on the data processed.

What is the right way to access a database from within a Celery worker?

Is it possible to share a pool across multiple workers/tasks or is there some other way to do this?

解决方案

I like tigeronk2's idea of one connection per worker. As he says, Celery maintains its own pool of workers so there really isn't a need for a separate database connection pool. The Celery Signal docs explain how to do custom initialization when a worker is created so I added the following code to my tasks.py and it seems to work exactly like you would expect. I was even able to close the connections when the workers are shutdown:

from celery.signals import worker_process_init, worker_process_shutdown

db_conn = None

@worker_process_init.connect

def init_worker(**kwargs):

global db_conn

print('Initializing database connection for worker.')

db_conn = db.connect(DB_CONNECT_STRING)

@worker_process_shutdown.connect

def shutdown_worker(**kwargs):

global db_conn

if db_conn:

print('Closing database connectionn for worker.')

db_conn.close()