在scrapy中,如果想批量运行爬虫文件,常见的两种方法 :
1)使用CrawProcess实习
2)使用修改craw源码 + 自定义命令的方式实现
接下来我们将重点通过实战讲解第二种方法。
首先,创建一个Scrapy爬虫项目,如下所示:
#(1)
D:\Python35\myweb\part12>scrapy startproject mymultispd
New Scrapy project 'mymultispd', using template directory 'd:\\python35\\lib\\site-packages\\scrapy\\templates\\project', created in:
D:\Python35\myweb\part12\mymultispd
You can start your first spider with:
cd mymultispd
scrapy genspider example example.com
然后,进入该目录爬虫项目,并在项目中创建爬虫文件,如下所示:
#(2)
D:\Python35\myweb\part12>cd mymultispd
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd1 sina.com.cn
Created spider 'myspd1' using template 'basic' in module:
mymultispd.spiders.myspd1
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd2 sina.com.cn
Created spider 'myspd2' using template 'basic' in module:
mymultispd.spiders.myspd2
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd3 sina.com.cn
Created spider 'myspd3' using template 'basic' in module
1)使用CrawProcess实习
2)使用修改craw源码 + 自定义命令的方式实现
接下来我们将重点通过实战讲解第二种方法。
首先,创建一个Scrapy爬虫项目,如下所示:
#(1)
D:\Python35\myweb\part12>scrapy startproject mymultispd
New Scrapy project 'mymultispd', using template directory 'd:\\python35\\lib\\site-packages\\scrapy\\templates\\project', created in:
D:\Python35\myweb\part12\mymultispd
You can start your first spider with:
cd mymultispd
scrapy genspider example example.com
然后,进入该目录爬虫项目,并在项目中创建爬虫文件,如下所示:
#(2)
D:\Python35\myweb\part12>cd mymultispd
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd1 sina.com.cn
Created spider 'myspd1' using template 'basic' in module:
mymultispd.spiders.myspd1
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd2 sina.com.cn
Created spider 'myspd2' using template 'basic' in module:
mymultispd.spiders.myspd2
D:\Python35\myweb\part12\mymultispd>scrapy genspider -t basic myspd3 sina.com.cn
Created spider 'myspd3' using template 'basic' in module