python scrapy框架制作爬虫

使用pip安装:

pip install Scrapy

进入http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted ，下载对应的版本

查看相应版本

import pip
import pip._internal.pep425tags
print(pip)
print(pip._internal.pep425tags.get_supported())

添加新的 scrapy 项目

scrapy startproject douban
    New Scrapy project 'douban', using template directory 'f:\python\lib\site-packages\scrapy\templates\project', created in:
        F:\PythonWenjian\project\douban
    
    You can start your first spider with:
        cd douban
        scrapy genspider example example.com

创建爬虫

scrapy genspider douban_spider movie.douban.com

允许访问 (启动爬虫)

scrapy crawl douban_spider（爬虫名称）

安装PyMySQL

python -m pip install PyMySQL