Scrapy框架利用CrawlSpider创建自动爬虫

 一、适用条件

   可以对有规律或者无规律的网站进行自动爬取

 二、代码讲解 

 (1)创健scrapy项目

E:myweb>scrapy startproject mycwpjt
New Scrapy project 'mycwpjt', using template directory 'd:\\python35\\lib\\site-packages\\scrapy\\templates\\project', created in:
    D:\Python35\myweb\part16\mycwpjt
You can start your first spider with:
    cd mycwpjt
    scrapy genspider example example.com

 (2) 创健爬虫

E:\myweb>scrapy genspider -t crawl weisuen sohu.com
Created spider 'weisuen' using template 'crawl' in module:
  Mycwpjt.spiders.weisuen

(3)item编写