Scrapy startproject 创建一个项目
怎么创建它 cd 进入要TLs(原有的包里面可以创建多个,一起使用)创建的包里面去
Scrapy genspider db db.com 显示 Created创建成功即可
#窗口化执行
from scrapy import cmdline
cmdline.execute('scrapy crawl db'.split())
Scrapy 数据去重操作
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class TlsPipeline:
def __init__(self):
self.ids_seen = set()
def process_item(self, item, spider):
adapter = ItemAdapter(item)
if adapter['id'] in self.ids_seen:
raise DropItem(f"Duplicate item found: {item!r}")
else:
self.ids_seen.add(adapter['id'])
return item