开发项目中采用Django作为后端框架,Celery作为异步任务队列,Mysql作为django的数据库,redis作为celery的broker和backed。在View中Django接收用户上传的文件,通过ORM将相关数据入库后,将Models的id传送给celery的task。Celery的task利用id通过
FileModels.objects.get(id=file_id)
来获取相关数据,进行处理后再利用ORM保存处理结果。
任务线很清晰,但是在使用过程中却遇到了问题。django采用如下方法开启:
python /Project/Path/manage.py runserver
celery 采用下面的方法开启:
celery -A Project worker -l info -P enventlet
当任务上传后,报如下的错误:
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id xxxxx and this is thread id xxxxxxx.
最终跟踪的出错点为“site-packages\django\db\backends\base\base.py”中如下位置:
def validate_thread_sharing(self):
"""
Validate that the connection isn't accessed by another thread than the
one which originally created it, unless the connection was explicitly
authorized to be shared between threads (via the `inc_thread_sharing()`
method). Raise an exception if the validation fails.
"""
if not (self.allow_thread_sharing or self._thread_ident == _thread.get_ident()):
raise DatabaseError(
"DatabaseWrapper objects created in a "
"thread can only be used in that same thread. The object "
"with alias '%s' was created in thread id %s and this is "
"thread id %s."
% (self.alias, self._thread_ident, _thread.get_ident())
)
验证是否是同一个线程在操作该DatabaseWrapper。通过将抛出错误的代码注释掉,即可消除该错误,系统运行正常。
但总感觉这不是合理的解决方案,我觉得有两个方向,但不知道能否实现:
一是在celery运行中,是否可以和django的models进行切割,使其采用不同实例的databasewrapper;
二是我发现“site-packages\django\db\backends\base\base.py”中有如下两个函数定义:
@property
def allow_thread_sharing(self):
with self._thread_sharing_lock:
return self._thread_sharing_count > 0
def inc_thread_sharing(self):
with self._thread_sharing_lock:
self._thread_sharing_count += 1
也就是说如果能够调用“inc_thread_sharing”,则“allow_thread_sharing”即可返回true了,但我一时没有找到Django的view中调用的方式。
不知道哪位大师了解相关的知识,可以为在下答疑解惑,不胜感激!