大家好,我是天空之城,今天给大家带来小福利,用scrapy框架中的CrawlSpider类构建爬虫获取信息
Scrapy框架中分两类爬虫
Spider类和CrawlSpider类。
crawlspider是Spider的派生类(一个子类),Spider类的设计原则是只爬取start_url列表中的网页,而CrawlSpider类定义了一些规则(rule)来提供跟进link的方便的机制,从爬取的网页中获取link并继续爬取的工作更适合。
spider有parse函数
crawl spider没有parse函数
crawl spider生成了一个rules,内含一个元祖或者列表,包含rule对象
rule标识规则,包含linkextractor,callback,follow等参数.
linkextractor连接提取器,可以通过正则,或者xpath或者css规则提取.
callback标识经过提取器取出来的url地址响应的回调函数,
重点是follow=true/falase 标识是否在当前页面中继续使用该规则进行深层提取.
如果一个被提取的url满足多个Rule,那么会从rules中选择一个满足匹配条件的Rule执行
.
下面项目的目录结构图
最后获取的数据截图
首先是cyg爬虫文件里面的代码
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import re
'''
1.创建CrawlSpider 创建方式 scrapy genspider -t crawl 爬虫的名字 域名
2.CrawlSpider 需要定义回调函数的时候最好找个函数名字不要以parse命名
3.Rule对象 什么时候你要follow callback的实现方式
'''
class CygSpider(CrawlSpider):
name = 'cyg'
allowed_domains = ['wz.sun0769.com']
start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1']
# 定义提取url地址的规则
rules = (
# LinkExtractor 链接提取器 需要提取的url地址
# callback 提取Url地址的response会交给回调函数处理
# follow=True 就是请求连续不断新的url地址
# 列表页
Rule(LinkExtractor(allow=r'http://wz.sun0769.com/political/index/politicsNewest\?id=\d+'),follow=True),
# 详情页
Rule(LinkExtractor(allow=r'http://wz.sun0769.com/political/politics/index\?id=\d+'), callback='parse_item'),
)
def parse_item(self, response):
item = {}
#item['domain_id'] = response.xpath('//input[@id="sid"]/@value').get()
#item['name'] = response.xpath('//div[@id="name"]').get()
#item['description'] = response.xpath('//div[@id="description"]').get()
# 详情页的数据
item['title']=response.xpath("//div[@class='mr-three']/p/text()").extract_first()
# item['author']=response.xpath("//html/body/div[3]/div[2]/div[2]/div[1]/span[1]/text()")
item['author'] = response.xpath("//div[@class='mr-three']/div[1]/span[1]/text()")[1]
item['content'] = response.xpath("//div[@class='details-box']/pre/text()").extract_first()
#item['date'] = response.xpath("//div[@class='mr-three']/div/span[@class='fl']/text()").extract_first()
print(item)
return item
然后是设置文件里面的代码
# -*- coding: utf-8 -*-
# Scrapy settings for yg project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'yg'
SPIDER_MODULES = ['yg.spiders']
NEWSPIDER_MODULE = 'yg.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'yg (+http://www.yourdomain.com)'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
# Obey robots.txt rules
# ROBOTSTXT_OBEY = True
# LOG_LEVEL = 'WARNING'
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.5
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'yg.middlewares.YgSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'yg.middlewares.YgDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'yg.pipelines.YgPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
FEED_URI='./storage/data/%(name)s.csv'
FEED_FORMAT='CSV'
FEED_EXPORT_ENCODING='ansi'
然后是items里面代码
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class YgItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
pass
然后是pipelines里面代码
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
class YgPipeline:
def process_item(self, item, spider):
return item
最后是启动爬虫的代码
from scrapy import cmdline
# cmdline.execute("scrapy crawl db".split())
cmdline.execute(['scrapy','crawl','cyg'])
这里还有个小问题,就是最后在获取author内容时,总是无法直接获得发帖人的姓名,换了好几个xpath语句,都不行,不知道怎么回事,有大牛知道的麻烦给解答下哈
{'title': '石碣华润广场到处黑漆漆,强烈要求亮化', 'author': <Selector xpath="//div[@class='mr-three']/div[1]/span[1]/text()" data='米拉多 '>, 'content': '随着梧桐里生活广场的马上开业,目前华润广场那里一到晚上到处都黑漆漆的,而且广场那边灯已经坏了几个月了,也没人去修,华润广场后面全部黑漆漆,没任何灯,很不安全,东风中路的墙体也没安装夜景灯,目前到处是黑漆漆的\r\n♥强烈要求政府对整个华润广场安装夜景灯彩色灯亮化,然后把东风中路一侧的墙体进行亮化,提升石碣中心城区的夜景景观!随着梧桐里生活广场开业,除了合信广场,华润广场那边也会成为石碣另一个商业中心,强烈要求政府改善华润广场的环境,对华润广场安装好彩色灯,把东风中路的墙体跟东风南路一样,进行亮化!'}
就是这里 'author': <Selector xpath="//div[@class='mr-three']/div[1]/span[1]/text()" data='米拉多 '>,我只想要后的“米拉多 ”,但是总会出现前面的那些selector内容,不知道怎么回事
网页源码是这样的
其中“自由自在多好啊”就是发帖人的姓名,但是总是无法直接获取到。