网络爬虫框架Scrapy详解之Request


文章转自: http://developer.51cto.com/art/201808/580814.htm

介绍

Request类是一个http请求的类,对于爬虫而言是一个很重要的类。通常在Spider中创建这样的一个请求,在Downloader中执行这样的一个请求。同时也有一个子类FormRequest继承于它,用于post请求。

源码介绍:

"""
This module implements the Request class which is used to represent HTTP
requests in Scrapy.

See documentation in docs/topics/request-response.rst
"""
import six
from w3lib.url import safe_url_string

from scrapy.http.headers import Headers
from scrapy.utils.python import to_bytes
from scrapy.utils.trackref import object_ref
from scrapy.utils.url import escape_ajax
from scrapy.http.common import obsolete_setter


class Request(object_ref):

    def __init__(self, url, callback=None, method='GET', headers=None, body=None,
                 cookies=None, meta=None, encoding='utf-8', priority=0,
                 dont_filter=False, errback=None, flags=None):

        self._encoding = encoding  # this one has to be set first
        self.method = str(method).upper()
        self._set_url(url)
        self._set_body(body)
        assert isinstance(priority, int), "Request priority not an integer: %r" % priority
        self.priority = priority

        if callback is not None and not callable(callback):
            raise TypeError('callback must be a callable, got %s' % type(callback).__name__)
        if errback is not None and not callable(errback):
            raise TypeError('errback must be a callable, got %s' % type(errback).__name__)
        assert callback or not errback, "Cannot use errback without a callback"
        self.callback = callback
        self.errback = errback

        self.cookies = cookies or {}
        self.headers = Headers(headers or {}, encoding=encoding)
        self.dont_filter = dont_filter

        self._meta = dict(meta) if meta else None
        self.flags = [] if flags is None else list(flags)

    @property
    def meta(self):
        if self._meta is None:
            self._meta = {}
        return self._meta

    def _get_url(self):
        return self._url

    def _set_url(self, url):
        if not isinstance(url, six.string_types):
            raise TypeError('Request url must be str or unicode, got %s:' % type(url).__name__)

        s = safe_url_string(url, self.encoding)
        self._url = escape_ajax(s)

        if ':' not in self._url:
            raise ValueError('Missing scheme in request url: %s' % self._url)

    url = property(_get_url, obsolete_setter(_set_url, 'url'))

    def _get_body(self):
        return self._body

    def _set_body(self, body):
        if body is None:
            self._body = b''
        else:
            self._body = to_bytes(body, self.encoding)

    body = property(_get_body, obsolete_setter(_set_body, 'body'))

    @property
    def encoding(self):
        return self._encoding

    def __str__(self):
        return "<%s %s>" % (self.method, self.url)

    __repr__ = __str__

    def copy(self):
        """Return a copy of this Request"""
        return self.replace()

    def replace(self, *args, **kwargs):
        """Create a new Request with the same attributes except for those
        given new values.
        """
        for x in ['url', 'method', 'headers', 'body', 'cookies', 'meta',
                  'encoding', 'priority', 'dont_filter', 'callback', 'errback']:
            kwargs.setdefault(x, getattr(self, x))
        cls = kwargs.pop('cls', self.__class__)
        return cls(*args, **kwargs)

参数说明:
url : str类型
请求的url
flags list类型
一般不会用到,发送请求的标志,一般用于日志记录
meta :dict类型
可用户自定义从Request到Response传递参数,这个参数一般也可在middlewares中处理   
callback : 指定函数
回调函数,用于接收请求后的返回信息,若没指定,则默认为parse()函数
method : str类型
http请求的方式,默认为GET请求,一般不需要指定。若需要POST请求,用FormRequest即可
headers : dict类型
请求头信息,一般在settings中设置即可,也可在middlewares中设置
body :str类型
为请求体,一般不需要设置(get和post其实都可以通过body来传递参数,不过一般不用)
cookies :dict或list类型
请求的cookie dict方式(name和value的键值对):   
dict方式:
cookies = {'name1' : 'value1' , 'name2' : 'value2'} 

list方式: 
cookies = [  
   {'name': 'Zarten', 'value': 'my name is Zarten', 'domain': 'example.com', 'path': '/currency'}  
   ] 
encoding : str类型
请求的编码方式,默认为'utf-8'
priority : int类型
int类型,指定请求的优先级,数字越大优先级越高,可以为负数,默认为0
dont_filter : bool 类型
默认为False,若设置为True,这次请求将不会过滤(不会加入到去重队列中),可以多次执行相同的请求
errback : 指定函数
抛出错误的回调函数,错误包括404,超时,DNS错误等,它接收Twisted Failure实例作为第一个参数
from scrapy.spidermiddlewares.httperror import HttpError  
   from twisted.internet.error import DNSLookupError  
   from twisted.internet.error import TimeoutError, TCPTimedOutError  
   class ToScrapeCSSSpider(scrapy.Spider):  
       name = "toscrape-css"  
       # start_urls = [  
       #     'http://quotes.toscrape.com/',  
       # ]  
       start_urls = [  
           "http://www.httpbin.org/",  # HTTP 200 expected  
           "http://www.httpbin.org/status/404",  # Not found error  
           "http://www.httpbin.org/status/500",  # server issue  
           "http://www.httpbin.org:12345/",  # non-responding host, timeout expected  
           "http://www.httphttpbinbin.org/",  # DNS error expected  
       ]  
       def start_requests(self):  
           for u in self.start_urls:  
               yield scrapy.Request(u, callback=self.parse_httpbin,  
                                    errback=self.errback_httpbin,  
                                    dont_filter=True)  
       def parse_httpbin(self, response):  
           self.logger.info('Got successful response from {}'.format(response.url))  
           # do something useful here...  
       def errback_httpbin(self, failure):  
           # log all failures  
           self.logger.info(repr(failure))  
           # in case you want to do something special for some errors,  
           # you may need the failure's type:  
           if failure.check(HttpError):  
               # these exceptions come from HttpError spider middleware  
               # you can get the non-200 response  
               response = failure.value.response  
               self.logger.info('HttpError错误 on %s', response.url)  
           elif failure.check(DNSLookupError):  
               # this is the original request  
               request = failure.request  
               self.logger.info('DNSLookupError错误 on %s', request.url)  
           elif failure.check(TimeoutError, TCPTimedOutError):  
               request = failure.request  
               self.logger.info('TimeoutError错误 on %s', request.url) 

meta参数在Response中:

my_name = response.meta[‘name’]
不过也有scrapy内置的特殊key,也非常有用,它们如下:

proxy:str类型
一般在middlewares中设置, 可以设置http或https代理, request.meta['proxy'] = 'https://' + 'ip:port' 
downloadtimeout
设置请求超时等待时间(秒),通常在settings中设置DOWNLOADTIMEOUT,默认是180秒(3分钟)
maxretrytimes
最大重试次数(除去第一次下载),默认为2次,通常在settings中 RETRY_TIMES设置
dont_redirect : bool 类型
设为True后,Request将不会重定向
dont_retry : bool 类型
设为True后,对于http链接错误或超时的请求将不再重试请求
handlehttpstatuslist : list 类型

http返回码200-300之间都是成功的返回,超出这个范围的都是失败返回,scrapy默认是过滤了这些返回,不会接收这些错误的返回进行处理。不过可以自定义处理哪些错误返回:

yield scrapy.Request(url= 'https://httpbin.org/get/zarten', meta= {'handle_httpstatus_list' : [404]}) 

在parse函数中可以看到处理404错误:

def parse(self, response):  
    print('返回信息为:',response.text) 
 handlehttpstatusall 设为True后,Response将接收处理任意状态码的返回信息
 dontmergecookies scrapy会自动保存返回的cookies,用于它的下次请求,当我们指定了自定义cookies时,如果我们不需要合并返回的cookies而使用自己指定的cookies,可以设为True
 cookiejar 可以在单个spider中追踪多个cookie,它不是粘性的,需要在每次请求时都带上       
def start_requests(self):  
           urls = ['http://quotes.toscrape.com/page/1',  
                   'http://quotes.toscrape.com/page/3',  
                   'http://quotes.toscrape.com/page/5',  
                   ]  
           for i ,url in enumerate(urls):  
               yield scrapy.Request(urlurl= url, meta= {'cookiejar' : i})  
       def parse(self, response):  
           next_page_url = response.css("li.next > a::attr(href)").extract_first()  
           if next_page_url is not None:  
               yield scrapy.Request(response.urljoin(next_page_url), meta= {'cookiejar' : response.meta['cookiejar']}, callback= self.parse_next)  
       def parse_next(self, response):  
           print('cookiejar:', response.meta['cookiejar']) 
dont_cache : bool 类型
 设为True后,不会缓存
redirect_urls : ? 类型
暂时还不清楚具体的作用,知道的小伙伴们欢迎在评论留言
bindaddress : str类型
绑定输出IP
dontobeyrobotstxt : bool 类型
设为True,不遵守robots协议,通常在settings中设置
downloadmaxsize : int 类型
设置下载器最大下载的大小(字节),通常在settings中设置DOWNLOADMAXSIZE,默认为1073741824 (1024MB=1G),若不设置最大的下载限制,设为0
download_latency : int
只读属性,获取请求的响应时间(秒)     
def start_requests(self):  
           headers = {  
               'user-agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'  
           }  
           yield scrapy.Request(url= 'https://www.amazon.com', headersheaders= headers)  
       def parse(self, response):  
           print('响应时间为:', response.meta['download_latency']) 
downloadfailon_dataloss :?类型
很少用到
referrer_policy : ? 类型
设置Referrer Policy

子类: FormRequest

FormRequest 类为Request的子类,用于POST请求

  • 这个类新增了一个参数 formdata,其他参数与Request一样,详细可参考上面的讲述

一般用法为:

yield scrapy.FormRequest(url="http://www.example.com/post/action",  
                       formdata={'name': 'Zarten', 'age': '27'},  
                       callback=self.after_post)  
  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值