scrapy爬虫的简单实例

最新推荐文章于 2022-07-03 17:41:10 发布

王阿文要努力啊

最新推荐文章于 2022-07-03 17:41:10 发布

阅读量406

点赞数

本文链接：https://blog.csdn.net/WangaWen1229/article/details/105990304

版权

在这里插入图片描述

创建文件D:/pycodes

D:\pycodes>scrapy startproject python123demo
New Scrapy project 'python123demo', using template directory 'c:\users\administrator\appdata\local\programs\python\python37\lib\site-packages\scrapy\templates\project', created in:
    D:\pycodes\python123demo

You can start your first spider with:
    cd python123demo
    scrapy genspider example example.com

在这里插入图片描述

# -*- coding: utf-8 -*-
import scrapy


class DemoSpider(scrapy.Spider):#继承scrapy.Spider的子类
	name = 'demo'
	start_urls = ['http://python123.io/ws/demo.html']#爬取的初始页面

	def parse(self, response):#parse()用于处理响应，解析内容形成字典，发现新的URL爬取请求。
		fname=response.url.split('/')[-1]
		with open(fname,'wb') as f:
			f.write(response.body)
		self.log('Saved file %s.' % fname)

D:\pycodes\python123demo>scrapy crawl demo
2020-05-08 11:57:02 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: python123demo)
2020-05-08 11:57:02 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2020-05-08 11:57:02 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-05-08 11:57:02 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'python123demo',