平时写了很多小爬虫,最近找到一个小网页,页面特别简单,文章地址如下:
(闲来无事,写个小爬虫,爬取1800篇高中作文)可以先了解一下爬取原理https://blog.csdn.net/hq606/article/details/115215425,
可以实现几十页的批量爬取。具体文章我已经写到博客里面了,里面有详细的思路和代码。写完以后,我突发奇想,既然这个爬虫这么简单好实现,这么多页面,为啥不用scrapy试试呢?以前想学scrapy但是实在不懂人家的方法,但是凭着直觉,我感觉我的爬虫肯定可以比较简单的放入scrapy框架,同时也能进一步学习它的原理和用法。
爬取的网页域名 http://www.zuowen.com
页面结构http://www.zuowen.com/gaokaozw/manfen/index_+页码+.shtml
每一页提取的数据页面 结构如下: 很容易提取
http://www.zuowen.com/e/20201201/5fc644d87ed36.shtml
http://www.zuowen.com/e/20201201/5fc6453d1e41b.shtml
http://www.zuowen.com/e/20201201/5fc64653d9ea8.shtml
http://www.zuowen.com/e/20201201/5fc6451d5130f.shtml
http://www.zuowen.com/e/20201201/5fc646982fefb.shtml
第一步创建爬虫框架
# 命令行先输入 cd desktop 把目录切换到桌面 这是我的坏习惯 为了在桌面方便操作
# 命令行输入 scrapy startproject zuowen
上面是自动生成的文件
init #初始化设置 一般是空的
middlewares #中间件,个人理解为中间传递工具,一个框架工具,暂时不管他
pipelines
#管道 负责输出你的数据,并且设置写入文件,这里为了简单我只是让他print()输出
settings #爬取参数的设置,设置时间,线程,请求头,日志等等
items #这里定义你要爬取得到的所有数据,有几种,就写几个,注意这里的数据写好 spiders 里面的爬虫以后,才能知道你有哪些数据要获得,然后才放入这个item
下面放出代码:
init.py
#这个是自己生成的必须有 但暂时是空的文件,可能是逻辑需要,或者设置备用
#这里好空啊啥也没有
再看spiders文件夹里面的爬虫代码,
zuowen.py
import scrapy
import sys
#sys.path.append("..")
#import items
from ..items import ZuowenItem
import requests
import bs4
from bs4 import BeautifulSoup
import time
# 这个爬虫和我https://blog.csdn.net/hq606/article/details/115215425里面写的一样主要是用来产生网页地址然后用BeautifulSoup解析
class GaokmfSpider(scrapy.Spider):#自动生成的类
name = 'zuowen'#爬虫名字 自己写一个
allowed_domains = ['http://www.zuowen.com'] #爬取网页的域名 或者首页吧
def start_requests(self):# 这个函数就是用来产生需要爬取的地址
start_urls = [] #里面存放大量地址为了爬取,
#我这里因为下面重新定义了一个列表result,所以这里做备用
#例如地址http://www.zuowen.com/e/20201201/5fc644d87ed36.shtml
urlhead = 'http://www.zuowen.com/gaokaozw/manfen/index_'
for n in range(1,10):#自己设置需要爬取的页数,先弄10页试试效果
if n==1:
url =urlhead[:-1] +'.shtml' # 按照页码构造需要请求的网页第一页的链接
else:
url =urlhead+str(n)+'.shtml' # 按照页码构造需要请求第二页之后的网页的链接
html = requests.get(url) # get方式请求数据
# print(html.status_code) # 查看请求的状态码(200表示请求正常)
html.encoding = 'gbk'
content = html.text
# 解析器自带的是html.parser,也可以用lxml,兼容性更好,不过要单独安装第三方库
soup = BeautifulSoup(content, "lxml")
words=soup.find_all('div',class_="artbox_l_c")
#print(words)
#global result
#global drl
result = []
for div in words:
drl=div.find('a')
#print(type(drl))
result.append(drl['href']) # 获取每页的20篇文章地址放入列表
#print(drl['href'])
for i in result:
url =i
start_urls.append(i) #暂时放入列表,这里仅仅备用
#print(start_urls)
yield scrapy.Request(url=url,callback=self.parse)
# yield把每个地址传送给Request #这是我的理解
def parse(self, response):#response 我理解为,是框架已经自动执行了Request(url)的动作
item=ZuowenItem() #ZuowenItem()函数定义你要爬取的数据,感觉像是定义了一个字典
#里面放的两个东西一个是我将来要的标题title 一个是文章内容cont2
#title = scrapy.Field()
#cont2 = scrapy.Field()
#response.encoding = 'gbk'
soup = bs4.BeautifulSoup(response.text,'lxml') #暂时放入列表,这里仅仅备用
result2=[]
words=soup.find_all('p')#获取作文内但是这是一句一句的话,组成的列表
item['title']=soup.find('h1').text #获取作文标题就一个标签 赋值给 item['title']
#item['cont1']=titl.text
for th in words[1:-11]: #把作文内容文字部分,全部放入一个列表,切片方法去掉广告文字
result2.append(th.text) #把作文内容文字部分,全部放入一个列表,去掉广告文字
#items['titl']=result2.append(titl.text)
item['cont2'] = result2[:] #获取作文所有句子,赋值给 item['cont2']
#print({item['title']:item['cont2']})
yield item
#urlhead = 'http://www.zuowen.com/gaokaozw/manfen/index_'
#a=GaokmfSpider(scrapy.Spider)
#a.start_requests()
#print(a.start_requests())
#a.parse(i)
那个item我为了简单,就给出了两个title和cont2,这是每次爬取作文要得到的,一个是作文标题 一个是作文内容。
#Item 是保存爬取到的数据的容器;其使用方法和python字典类似, 并且提供了额外保护机制来避免拼写错误导致的未定义字段错误
下面是items.py
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class ZuowenItem(scrapy.Item):
# define the fields for your item here like:
# scrapy.Field() 我猜它似乎是一个字典
title = scrapy.Field() #像是定义了一个字典的键
cont2 = scrapy.Field() #像是又定义了一个字典的键
pass
pipelines.py 数据管道
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# useful for handling different item types with a single interface
#from itemadapter import ItemAdapter
#用于把爬取的数据存储到文件的管道
class ZuowenPipeline:
def process_item(self, item, spider): #把文字写入txt文件
with open('爬取作文.txt','a',encoding='gbk') as f:
f.write('\n'+str({item['title']:item['cont2']})+'\n\n') #按照字典形式写入
f.close()
print('写入成功')
return item
middlewares.py #这个是自己生成的,我不敢动它
# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter
class ZuowenSpiderMiddleware:
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(self, response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(self, response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, or item objects.
for i in result:
yield i
def process_spider_exception(self, response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Request or item objects.
pass
def process_start_requests(self, start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
class ZuowenDownloaderMiddleware:
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the downloader middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
# Called for each request that goes through the downloader
# middleware.
# Must either:
# - return None: continue processing this request
# - or return a Response object
# - or return a Request object
# - or raise IgnoreRequest: process_exception() methods of
# installed downloader middleware will be called
return None
def process_response(self, request, response, spider):
# Called with the response returned from the downloader.
# Must either;
# - return a Response object
# - return a Request object
# - or raise IgnoreRequest
return response
def process_exception(self, request, exception, spider):
# Called when a download handler or a process_request()
# (from other downloader middleware) raises an exception.
# Must either:
# - return None: continue processing this exception
# - return a Response object: stops process_exception() chain
# - return a Request object: stops process_exception() chain
pass
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
settings.py#设置爬取的参数,线程,延时,速度,日志等等
#里面去掉注释的那几行,是根据网上学的,设置了几个参数
# Scrapy settings for zuowen project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'zuowen'
SPIDER_MODULES = ['zuowen.spiders']
NEWSPIDER_MODULE = 'zuowen.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 QIHU 360SE'
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0
CONCURRENT_REQUESTS = 100
CONCURRENT_REQUESTS_PER_DOMAIN = 100
CONCURRENT_REQUESTS_PER_IP = 100
COOKIES_ENABLED = False
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'Accept': '*/*',
'Accept-Language': 'zh-CN,zh;q=0.9',
'User-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 QIHU 360SE'
}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'zuowen.middlewares.ZuowenSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'zuowen.middlewares.ZuowenDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'zuowen.pipelines.ZuowenPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
#最后在第一层文件夹里面,放入一个main.py 文件 为了启动运行爬虫
main.py #注意这个是我自己添加的,不是scrapy产生的啊
from scrapy import cmdline
cmdline.execute(['scrapy','crawl','zuowen','-s','LOG_FILE=all.log'])
这样写main,为了让运行的时候,日志保存到all.txt文件,避免一堆红色日志输出,显得很乱。’-s’, ‘LOG_FILE=all.log’ 这两个东西去掉也可以的。
下面看效果
愉快的爬虫框架启动了,爽啊!!