python 管道 非阻塞_python – 非阻塞Scrapy管道到数据库

我在Scrapy中有一个Web抓取器,可以获取数据项.我想异步地将它们插入到数据库中.

例如,我有一个事务,使用SQLAlchemy Core将一些项目插入到我的数据库中：

def process_item(self, item, spider):

with self.connection.begin() as conn:

conn.execute(insert(table1).values(item['part1'])

conn.execute(insert(table2).values(item['part2'])

我知道可以与Twisted with alchimia异步使用SQLAlchemy Core.alchimia的文档代码示例如下.

我不明白的是如何在alchimia框架中使用我的上述代码.如何设置process_item以使用反应器？

我可以这样做吗？

@inlineCallbacks

def process_item(self, item, spider):

with self.connection.begin() as conn:

yield conn.execute(insert(table1).values(item['part1'])

yield conn.execute(insert(table2).values(item['part2'])

我怎么写反应堆部分？

或者是否有更简单的方法在Scrapy管道中进行非阻塞数据库插入？

供参考,以下是alchimia文档中的代码示例：

from alchimia import TWISTED_STRATEGY

from sqlalchemy import (

create_engine, MetaData, Table, Column, Integer, String

)

from sqlalchemy.schema import CreateTable

from twisted.internet.defer import inlineCallbacks

from twisted.internet.task import react

@inlineCallbacks

def main(reactor):

engine = create_engine(

"sqlite://", reactor=reactor, strategy=TWISTED_STRATEGY

)

metadata = MetaData()

users = Table("users", metadata,

Column("id", Integer(), primary_key=True),

Column("name", String()),

)

# Create the table

yield engine.execute(CreateTable(users))

# Insert some users

yield engine.execute(users.insert().values(name="Jeremy Goodwin"))

yield engine.execute(users.insert().values(name="Natalie Hurley"))

yield engine.execute(users.insert().values(name="Dan Rydell"))

yield engine.execute(users.insert().values(name="Casey McCall"))

yield engine.execute(users.insert().values(name="Dana Whitaker"))

result = yield engine.execute(users.select(users.c.name.startswith("D")))

d_users = yield result.fetchall()

# Print out the users

for user in d_users:

print "Username: %s" % user[users.c.name]

if __name__ == "__main__":

react(main, [])