Scrapy

启动爬虫
python -m venv venv

(venv) D:\book>pip install C:\Users\Administrator\Desktop\Twisted-18.4.0-cp36-cp
36m-win32.whl

pip install scrapy

scrapy startproject book .

scrapy genspider books www.dushu.com –template=crawl

pip install pypiwin32

scrapy crawl movie -o result.json

scrapy shell 网址

http://www.waanwang.com/index-cate-cid-160-p-2.html
http://www.waanwang.com/index-cate-cid-160.html

return None: continue processing this request 走到下载器
or return a Response object 走到蜘蛛
or return a Request object 走到调度器

启动爬虫 python -m venv venv (venv) 

D:\book>pip install C:\Users\Administrator\Desktop\Twisted-18.4.0-cp36-cp 36m-win32.whl

pip install scrapy

scrapy startproject book .

scrapy genspider books www.dushu.com –template=crawl

shouhuo