scrapy框架_Scrapy框架进阶篇

1 Scrapy框架进阶篇

1.1 Request

Scrapy 使用 Request和 Request 对象爬取 web 站点。

一般来说，Request 对象在 spider 中被生成并且最终传递到下载器(Downloader)，下载器对其进行处理并返回一个 Response 对象，Response 对象还会返回到生成 request 的 spider 中。

22.1.1 Request 对象

一个 Request 对象代表一个 HTTP 请求，一般来讲，HTTP请求是由 Spider 产生并被 Downloader处理进而生成一个 Response。

22.1.2 Request 模块的位置

C:Python34Libsite-packagesscrapyhttpequest

22.1.3 Request 源码

class Request(object_ref):

def __init__(self, url, callback=None, method='GET', headers=None, body=None,

cookies=None, meta=None, encoding='utf-8', priority=0,

dont_filter=False, errback=None, flags=None):

self._encoding = encoding # this one has to be set first

self.method = str(method).upper()

self._set_url(url)

self._set_body(body)

assert isinstance(priority, int), "Request priority not an integer: %r" % priority

self.priority = priority

if callback is not None and not callable(callback):

raise TypeError('callback must be a callable, got %s' % type(callback).__name__)

if errback is not None and not callable(errback):

raise TypeError('errback must be a callable, got %s' % type(errback).__name__)

assert callback or not errback, "Cannot use errback without a callback"

self.callback = callback

self.errback = errback

self.cookies = cookies or {}

self.headers = Headers(headers or {}, encoding=encoding)

self.dont_filter = dont_filter

self._meta = dict(meta) if meta else None

self.flags = [] if flags is None else list(flags)

@property

def meta(self):

if self._meta is None:

self._meta = {}

return self._meta

def _get_url(self):

return self._url

def _set_url(self, url):

if not isinstance(url, six.string_types):

raise TypeError('Request url must be str or unicode, got %s:' % type(url).__name__)

s = safe_url_string(url, self.encoding)

self._url = escape_ajax(s)

if ':' not in self._url:

raise ValueError('Missing scheme in request url: %s' % self._url)

url = property(_get_url, obsolete_setter(_set_url, 'url'))

def _get_body(self):

return self._body

def _set_body(self, body):

if body is None:

self._body = b''

else:

self._body = to_bytes(body, self.encoding)

body = property(_get_body, obsolete_setter(_set_body, 'body'))

@property

def encoding(self):

return self._encoding

def __str__(self):

return "" % (self.method, self.url)

__repr__ = __str__

def copy(self):

"""Return a copy of this Request"""

return self.replace()

def replace(self, *args, **kwargs):

"""Create a new Request with the same attributes except for those

given new values.

"""

for x in ['url', 'method', 'headers', 'body', 'cookies', 'meta',

'encoding', 'priority', 'dont_filter', 'callback', 'errback']:

kwargs.setdefault(x, getattr(self, x))

cls = kwargs.pop('cls', self.__class__)

return cls(*args, **kwargs)

22.1.4 Request的初始化构造方法

def __init__(self, url, callback=None, method='GET', headers=None, body=None,

cookies=None, meta=None, encoding='utf-8', priority=0,

dont_filter=False, errback=None, flags=None)

项目中使用的例子：

scrapy.Request(self.url + str(self.offset), callback = self.parse)

常用的参数：

【url】: 参数类型 string，请求的 url。

【callback】 ：处理响应数据的回调方法(指定该请求返回的 Response，由那个函数来处理)，用来解析响应数据，如果没有指定，则 spider 的 parse 方法。如果在处理期间引发异常，则会调用 errback。

【method】: 参数类型 string ，HTTP 的请求方法，请求一般不需要指定，默认 GET 方法，可设置为"GET