Scrapy第一个实例
演示HTML地址
http://python123.io/ws/demo.html
产生步骤:
- 步骤1:建立爬虫工厂
scrapy startproject python123demo
生成的工程目录
python123demo/ → 外层目录
scrapy.cfg → 部署Scrapy爬虫的配置文件
python123demo/ → Scrapy框架的用户自定义Python代码
__init__.py → 初始化脚本
items.py → Items代码模板(继承类)
middlewares.py → Middlewares代码模板(继承类)
pipelines.py → Pipelines代码模板(继承类)
settings.py → Scrapy爬虫的配置文件
spiders/ → Spiders代码模板目录(继承类)
__init__.py → 初始文件,无需修改
__pycache__/ → 缓存目录,无需修改
- 步骤2:在工程中产生一个Scrapy爬虫
cd python123demo
scrapy genspider demo python123.io
demo.py
# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider);
name = "demo"
#allowed_domains = ["python123.io"]
start_url = ['http://python123.io/']
def parse(self, response):
fname = response.url.split('/')[-1]
with open(fname,'wb') as f:
f.write(response.body)
self.log('Saved file %s.' % fname)
pass
- 步骤3:配置产生的爬虫
- 步骤4:运行爬虫,获取网页。
scrapy crawl demo