Python爬虫：scrapy框架请求参数meta、headers、cookies一探究竟

最新推荐文章于 2024-07-17 15:15:27 发布
最新推荐文章于 2024-07-17 15:15:27 发布
阅读量1w 收藏 34
点赞数 10
分类专栏： Python scrapy
本文为博主原创文章，欢迎转载，请注明出处
本文链接：https://blog.csdn.net/mouday/article/details/81159416
版权
Python 同时被 2 个专栏收录
614 篇文章 36 订阅
订阅专栏
scrapy
35 篇文章 2 订阅
订阅专栏

对于scrapy请参数，会经常用到，不过没有深究

今天我就来探索下scrapy请求时所携带的3个重要参数headers, cookies, meta

原生参数

首先新建myscrapy项目，新建my_spider爬虫

通过访问：http://httpbin.org/get 来测试请求参数

将爬虫运行起来

# -*- coding: utf-8 -*-

from scrapy import Spider, Request
import logging


class MySpider(Spider):
    name = 'my_spider'
    allowed_domains = ['httpbin.org']
    start_urls = [
        'http://httpbin.org/get'
    ]

    def parse(self, response):
        self.write_to_file("*" * 40)
        self.write_to_file("response text: %s" % response.text)
        self.write_to_file("response headers: %s" % response.headers)
        self.write_to_file("response meta: %s" % response.meta)
        self.write_to_file("request headers: %s" % response.request.headers)
        self.write_to_file("request cookies: %s" % response.request.cookies)
        self.write_to_file("request meta: %s" % response.request.meta)

    def write_to_file(self, words):
        with open("logging.log", "a") as f:
            f.write(words)


if __name__ == '__main__':
    from scrapy import cmdline
    cmdline.execute("scrapy crawl my_spider".split())

保存到文件中的信息如下：

response text: 
{
    "args":{},
    "headers":{
        "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Encoding":"gzip,deflate",
        "Accept-Language":"en",
        "Connection":"close",
        "Host":"httpbin.org",
        "User-Agent":"Scrapy/1.5.1 (+https://scrapy.org)"
    },
    "origin":"223.72.90.254",
    "url":"http://httpbin.org/get"
}

response headers: 
{
    b'Server': [b'gunicorn/19.8.1'], 
    b'Date': [b'Sun, 22 Jul 2018 10:03:15 GMT'], 
    b'Content-Type': [b'application/json'], 
    b'Access-Control-Allow-Origin': [b'*'], 
    b'Access-Control-Allow-Credentials': [b'true'], 
    b'Via': [b'1.1 vegur']
}

response meta: 
{
    'download_timeout': 180.0, 
    'download_slot': 'httpbin.org', 
    'download_latency': 0.5500118732452393
}

request headers: 
{
    b'Accept': [b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], b'Accept-Language': [b'en'], 
    b'User-Agent': [b'Scrapy/1.5.1 (+https://scrapy.org)'], 
    b'Accept-Encoding': [b'gzip,deflate']
}

request cookies: 
{}

request meta: 
{
    'download_timeout': 180.0, 
    'download_slot': 'httpbin.org', 
    'download_latency': 0.5500118732452393
}

meta

通过上面的输出比较，发现 response 和 request 的meta参数是一样的，meta的功能就是从request携带信息，将其传递给response的

修改下代码，测试下传递效果

# -*- coding: utf-8 -*-

from scrapy import Spider, Request
import logging


class MySpider(Spider):
    name = 'my_spider'
    allowed_domains = ['httpbin.org']
    start_urls = [
        'http://httpbin.org/get'
    ]

    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, meta={"uid": "this is uid of meta"})

    def parse(self, response):
        print("request meta: %s" % response.request.meta.get("uid"))
        print("response meta: %s" % response.meta.get("uid"))

输出如下

request meta: this is uid of meta
response meta: this is uid of meta

看来获取request中meta这两种方式都可行，这里的meta类似字典，可以按照字典获取key-value的形式获取对应的值
当然代理设置也是通过meta的
以下是一个代理中间件的示例

import random


class ProxyMiddleware(object):       
    def process_request(self, request, spider):
        proxy=random.choice(proxies)
        request.meta["proxy"] = proxy

headers

按照如下路径，打开scrapy的default_settings文件

from scrapy.settings import default_settings

发现是这么写的

USER_AGENT = 'Scrapy/%s (+https://scrapy.org)' % import_module('scrapy').__version__

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
}

修改下请求头，看服务器返回的信息

# -*- coding: utf-8 -*-

from scrapy import Spider, Request
import logging


class MySpider(Spider):
    name = 'my_spider'
    allowed_domains = ['httpbin.org']
    start_urls = [
        'http://httpbin.org/get',
    ]

    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, headers={"User-Agent": "Chrome"})

    def parse(self, response):
        logging.debug("*" * 40)
        logging.debug("response text: %s" % response.text)
        logging.debug("response headers: %s" % response.headers)
        logging.debug("request headers: %s" % response.request.headers)


if __name__ == '__main__':
    from scrapy import cmdline
    cmdline.execute("scrapy crawl my_spider".split())

输出如下

response text: 
{
    "args":{},
    "headers":
    {
        "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Encoding":"gzip,deflate",
        "Accept-Language":"en",
        "Connection":"close",
        "Host":"httpbin.org",
        "User-Agent":"Chrome"
    },
    "origin":"122.71.64.121",
    "url":"http://httpbin.org/get"
}

response headers: 
{
    b'Server': [b'gunicorn/19.8.1'], 
    b'Date': [b'Sun, 22 Jul 2018 10:29:26 GMT'], 
    b'Content-Type': [b'application/json'], 
    b'Access-Control-Allow-Origin': [b'*'], 
    b'Access-Control-Allow-Credentials': [b'true'], 
    b'Via': [b'1.1 vegur']
}


request headers: 
{
    b'User-Agent': [b'Chrome'], 
    b'Accept': [b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], b'Accept-Language': [b'en'], 
    b'Accept-Encoding': [b'gzip,deflate']
}

看到 request 和 服务器接收到并返回的的 headers（User-Agent）变化了，说明已经把默认的User-Agent修改了

看到default_settings中默认使用了中间件UserAgentMiddleware

'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,

源码如下


class UserAgentMiddleware(object):
    """This middleware allows spiders to override the user_agent"""

    def __init__(self, user_agent='Scrapy'):
        self.user_agent = user_agent

    @classmethod
    def from_crawler(cls, crawler):
        o = cls(crawler.settings['USER_AGENT'])
        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
        return o

    def spider_opened(self, spider):
        self.user_agent = getattr(spider, 'user_agent', self.user_agent)

    def process_request(self, request, spider):
        if self.user_agent:
            request.headers.setdefault(b'User-Agent', self.user_agent)

仔细阅读源码，发现无非就是对User-Agent读取和设置操作，仿照源码写自己的中间件

这里使用fake_useragent库来随机获取请求头，详情可参看：
https://blog.csdn.net/mouday/article/details/80476409

middlewares.py 编写自己的中间件

from fake_useragent import UserAgent


class UserAgentMiddleware(object):
    def process_request(self, request, spider):
        ua = UserAgent()
        user_agent = ua.chrome
        request.headers.setdefault(b'User-Agent', user_agent)

settings.py 用自己的中间件替换默认中间件

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'myscrapy.middlewares.UserAgentMiddleware': 500
}

输出如下：

request headers: 
{
    b'Accept': [b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], b'Accept-Language': [b'en'], 
    b'User-Agent': [b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36'], 
    b'Accept-Encoding': [b'gzip,deflate']
}

关于scrapy请求头设置，可以参考我之前的文章：
https://blog.csdn.net/mouday/article/details/80776030

cookies

上面的信息中少了个response.cookies，如果添加上回报错：

AttributeError: 'TextResponse' object has no attribute 'cookies'

说明响应是不带cookies参数的

通过 http://httpbin.org/cookies 测试cookies

# -*- coding: utf-8 -*-

from scrapy import Spider, Request
import logging


class MySpider(Spider):
    name = 'my_spider'
    allowed_domains = ['httpbin.org']
    start_urls = [
        'http://httpbin.org/cookies'
    ]


    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, cookies={"username": "pengshiyu"})

    def parse(self, response):
        logging.debug("*" * 40)
        logging.debug("response text: %s" % response.text)
        logging.debug("request headers: %s" % response.request.headers)
        logging.debug("request cookies: %s" % response.request.cookies)


if __name__ == '__main__':
    from scrapy import cmdline
    cmdline.execute("scrapy crawl my_spider".split())

返回值如下：

response text: 
{
    "cookies":
        {
        "username":"pengshiyu"
        }
}

request headers: 
{
    b'Accept': [b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], b'Accept-Language': [b'en'], 
    b'User-Agent': [b'Scrapy/1.5.1 (+https://scrapy.org)'], 
    b'Accept-Encoding': [b'gzip,deflate'], 
    b'Cookie': [b'username=pengshiyu']
}

request cookies: 
{
    'username': 'pengshiyu'
}

服务器端已经接收到我的cookie值了，不过request的headers也包含了相同的cookie，保存到了键为Cookie下面

其实并没有什么cookie，浏览器请求的·cookies·被包装到了·headers·中发送给服务器端
既然这样，在headers中包含Cookie试试

 def start_requests(self):
        for url in self.start_urls:
            yield Request(url, headers={"Cookie": {"username": "pengshiyu"}})

返回结果

response text: 
{
    "cookies":{}
}

request headers: 
{
    b'Accept': [b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], b'Accept-Language': [b'en'], 
    b'User-Agent': [b'Scrapy/1.5.1 (+https://scrapy.org)'], 
    b'Accept-Encoding': [b'gzip,deflate']
}


request cookies: 
{}

cookies 是空的，设置失败

我们找到 default_settings 中的cookie中间件

'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700

class CookiesMiddleware(object):
    """This middleware enables working with sites that need cookies"""

    def __init__(self, debug=False):
        self.jars = defaultdict(CookieJar)
        self.debug = debug

    @classmethod
    def from_crawler(cls, crawler):
        if not crawler.settings.getbool('COOKIES_ENABLED'):
            raise NotConfigured
        return cls(crawler.settings.getbool('COOKIES_DEBUG'))

    def process_request(self, request, spider):
        if request.meta.get('dont_merge_cookies', False):
            return

        cookiejarkey = request.meta.get("cookiejar")
        jar = self.jars[cookiejarkey]
        cookies = self._get_request_cookies(jar, request)
        for cookie in cookies:
            jar.set_cookie_if_ok(cookie, request)

        # set Cookie header
        request.headers.pop('Cookie', None)
        jar.add_cookie_header(request)
        self._debug_cookie(request, spider)

    def process_response(self, request, response, spider):
        if request.meta.get('dont_merge_cookies', False):
            return response

        # extract cookies from Set-Cookie and drop invalid/expired cookies
        cookiejarkey = request.meta.get("cookiejar")
        jar = self.jars[cookiejarkey]
        jar.extract_cookies(response, request)
        self._debug_set_cookie(response, spider)

        return response

    def _debug_cookie(self, request, spider):
        if self.debug:
            cl = [to_native_str(c, errors='replace')
                  for c in request.headers.getlist('Cookie')]
            if cl:
                cookies = "\n".join("Cookie: {}\n".format(c) for c in cl)
                msg = "Sending cookies to: {}\n{}".format(request, cookies)
                logger.debug(msg, extra={'spider': spider})

    def _debug_set_cookie(self, response, spider):
        if self.debug:
            cl = [to_native_str(c, errors='replace')
                  for c in response.headers.getlist('Set-Cookie')]
            if cl:
                cookies = "\n".join("Set-Cookie: {}\n".format(c) for c in cl)
                msg = "Received cookies from: {}\n{}".format(response, cookies)
                logger.debug(msg, extra={'spider': spider})

    def _format_cookie(self, cookie):
        # build cookie string
        cookie_str = '%s=%s' % (cookie['name'], cookie['value'])

        if cookie.get('path', None):
            cookie_str += '; Path=%s' % cookie['path']
        if cookie.get('domain', None):
            cookie_str += '; Domain=%s' % cookie['domain']

        return cookie_str

    def _get_request_cookies(self, jar, request):
        if isinstance(request.cookies, dict):
            cookie_list = [{'name': k, 'value': v} for k, v in \
                    six.iteritems(request.cookies)]
        else:
            cookie_list = request.cookies

        cookies = [self._format_cookie(x) for x in cookie_list]
        headers = {'Set-Cookie': cookies}
        response = Response(request.url, headers=headers)

        return jar.make_cookies(response, request)

观察源码，发现以下几个方法

# process_request
jar.add_cookie_header(request)   # 添加cookie到headers

# process_response
jar.extract_cookies(response, request)  # 提取出cookie

# _debug_cookie 
request.headers.getlist('Cookie')  # 从headers获取cookie

# _debug_set_cookie
response.headers.getlist('Set-Cookie')  # 从headers获取Set-Cookie

几个参数：

# settings
COOKIES_ENABLED
COOKIES_DEBUG

# meta
dont_merge_cookies
cookiejar

# headers
Cookie
Set-Cookie

使用最开始cookie部分的代码，为了看的清晰，我删除了headers中其他参数，下面逐个做测试

1、COOKIES_ENABLED

COOKIES_ENABLED = True (默认)

response text: 
{
    "cookies":{"username":"pengshiyu"}
}

request headers: 
{
    b'Cookie': [b'username=pengshiyu']
}

request cookies: 
{
    'username': 'pengshiyu'
}

一切ok
COOKIES_ENABLED = False 

response text: 
{
    "cookies":{}
}

request headers: 
{}

request cookies: 
{
    'username': 'pengshiyu'
}

虽然request的cookies有内容，不过headers没有加进去，所以服务器端没有获取到cookie

注意：查看请求的真正cookie，应该在request的header中查看

2、COOKIES_DEBUG
COOKIES_DEBUG = False (默认)

DEBUG: Crawled (200) <GET http://httpbin.org/cookies> (referer: None)

COOKIES_DEBUG = True
多输出了下面一句，可以看到我设置的cookie

[scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET http://httpbin.org/cookies>
Cookie: username=pengshiyu

当然，debug模式下服务器肯定能正常接收我的cookie

3、dont_merge_cookies
设置meta={"dont_merge_cookies": True} 默认为 False

response text: 
{
    "cookies":{}
}

request headers: 
{}

request cookies: 
{
    'username': 'pengshiyu'
}

服务器并没有接收到我的cookie

4、cookiejar
直接通过response.request.meta.get("cookiejar")获取


response text: 
{"cookies":{"username":"pengshiyu"}}

request headers: 
{b'Cookie': [b'username=pengshiyu']}

request cookies: 
{'username': 'pengshiyu'}

request cookiejar: 
None

啥也没有

5、Cookie
直接获取：response.request.headers.get("Cookie"))

headers Cookie: 
b'username=pengshiyu'

看来这里已经被处理成字节串了

修改Request请求参数
cookies={"username": "pengshiyu", "password": "123456"}

# response.request.headers.get("Cookie"))
headers Cookie: 
b'username=pengshiyu; password=123456'

# request.headers.getlist('Cookie')
headers Cookies: 
[b'username=pengshiyu; password=123456']

很明显，两个获取方式，一个获取的是字符串，一个获取的是列表

6、Set-Cookie

同样，我通过以下

response.headers.get("Set-Cookie")
response.headers.getlist("Set-Cookie")

还是啥都没有

headers Set-Cookie: None
headers Set-Cookies: []

不过，到目前为止，cookie设置的大概流程应该如下：

request cookies: {'username': 'pengshiyu', 'password': '123456'}
request cookiejar: None
request Cookie: b'username=pengshiyu; password=123456'
response text: {"cookies":{"password":"123456","username":"pengshiyu"}}
response Set-Cookie: None
response Set-Cookies: []

7、接收服务器传递过来的cookie

将请求链接改为 ：’http://httpbin.org/cookies/set/key/value’
开启 COOKIES_DEBUG
在debug中看到如下变化

Sending cookies to: <GET http://httpbin.org/cookies/set/key/value>
Cookie: username=pengshiyu; password=123456

Received cookies from: <302 http://httpbin.org/cookies/set/key/value>
Set-Cookie: key=value; Path=/

Redirecting (302) to <GET http://httpbin.org/cookies> from <GET http://httpbin.org/cookies/set/key/value>

Sending cookies to: <GET http://httpbin.org/cookies>
Cookie: key=value; username=pengshiyu; password=123456

日志看出他进行了两次请求，看到中间的cookie变化：

发送 -> 接收 -> 发送

第二次发送的cookie包含了第一次请求时服务器端传递过来的cookie，说明scrapy对服务器端和客户端的cookie进行了管理

最后的cookie输出

request cookies: {'username': 'pengshiyu', 'password': '123456'}
request cookiejar: None
request Cookie: b'key=value; username=pengshiyu; password=123456'
response text: {"cookies":{"key":"value","password":"123456","username":"pengshiyu"}}
response Set-Cookie: None

request的cookies并没有变化，而request.headers.get(“Cookie”)已经发生了变化

8、接收服务器传递过来的 同key键cookie
将请求链接换为：httpbin.org/cookies/set/username/pengpeng

Sending cookies to: <GET http://httpbin.org/cookies/set/username/pengpeng>
Cookie: username=pengshiyu

Received cookies from: <302 http://httpbin.org/cookies/set/username/pengpeng>
Set-Cookie: username=pengpeng; Path=/

Redirecting (302) to <GET http://httpbin.org/cookies> from <GET http://httpbin.org/cookies/set/username/pengpeng>

Sending cookies to: <GET http://httpbin.org/cookies>
Cookie: username=pengshiyu

发现虽然收到了username=pengpeng但是，第二次发请求的时候，又发送了原来的的cookieusername=pengshiyu

这说明客户端设置的cookie优先级高于服务器端传递过来的cookie

9、取消使用中间件CookiesMiddleware

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None
}

请求链接：http://httpbin.org/cookies

request cookies: {'username': 'pengshiyu'}
request cookiejar: None
request Cookie: None
response text: {"cookies":{}}
response Set-Cookie: None
response Set-Cookies: []

这个效果类似COOKIES_ENABLED = False

10、自定义cookie池

class RandomCookiesMiddleware(object):
    def process_request(self, request, spider):
        cookies = []
        cookie = random.choice(cookies)
        request.cookies = cookie

同样需要设置

DOWNLOADER_MIDDLEWARES = {
    'myscrapy.middlewares.RandomCookiesMiddleware': 600
}

注意到scrapy的中间件CookiesMiddleware值是700,为了cookie设置生效，需要在这个中间件启用之前就设置好自定义的cookie，优先级按照从小到大的顺序执行，所以我们自己自定义的cookie中间件需要小于 < 700

'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,

总结

参数设置获取说明
metaRequest(url, meta={“uid”: “100”}) request.meta[“uid”] = “100”response.request.meta.get(“uid”) response.meta.get(“uid”)携带request参数给response,或设置代理
headersRequest(url, headers={“User-Agent”: “chrome”}) request.headers[“User-Agent”]=”chrome” USER_AGENT=”chrome”response.request.headers.get(“User-Agent”)设置客户端请求头参数
cookiesRequest(url, cookies={“username”: “pengshiyu”} ) request.cookies = {“username”: “pengshiyu”}response.request.cookies response.request.headers.get(“Cookie”) response.headers.get(‘Set-Cookie’)客户端请求头中的Cookie参数，管理客户端与服务器端之间的会话识别

常用的中间件如下

import random
from fake_useragent import UserAgent


class RandomUserAgentMiddleware(object):
    def process_request(self, request, spider):
        ua = UserAgent()
        user_agent = ua.chrome
        request.headers.setdefault(b'User-Agent', user_agent)


class RandomProxyMiddleware(object):
    def process_request(self, request, spider):
        proxies = []
        proxy = random.choice(proxies)
        request.meta["proxy"] = proxy


class RandomCookiesMiddleware(object):
    def process_request(self, request, spider):
        cookies = []
        cookie = random.choice(cookies)
        request.cookies = cookie

当然，cookies 和 proxies 需要结合自己的情况补全

彭世瑜
关注
  • 10
    点赞
  • 踩
  • 34
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
解析Python网络爬虫：核心技术、Scrapy框架、分布式爬虫全套教学资料
10-25
Scrapy是Python的一个强大爬虫框架，它提供了一整套解决方案，包括请求调度、数据提取、中间件处理、项目结构管理等。其主要特性包括： 1. 项目结构：Scrapy项目由多个组件构成，如Spider、Item、Pipeline、...
scrapy设置headers，cookies
池边的树的博客
02-23 1万+
scrapy中有三种方式设置headers，cookies setting中设置cookie middlewares中设置cookie sipder文件中重写start_requests方法 这里记录第三种，重写start_requests方法，这里以豆瓣网为例 一、设置请求头headers 在start_request中新增 headers = { 'User-Agent'...
在scrapy里实现自定义请求头的几种方法
阿坨的博客
10-23 4743
抽空整理的几种在scrapy框架里实现自定义请求头的方法 前言 scrapy爬虫框架示意图 scrapy项目里add_headers爬虫的任务是爬取新浪首页，并打印其默认headers信息 import scrapy class AddHeadersSpider(scrapy.Spider): name = 'add_headers' allowed_domains = [...
scrapy提交参数请求
最新发布
m0_46780416的博客
07-17 190
【代码】scrapy提交参数请求。
Python网络爬虫(二十一)——Request 和 Response
止步听风的博客
05-25 1651
Request class Request(object_ref): def __init__(self, url, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None, encoding='utf-8', priority=0, dont_filter=False, errback=None, flags=None, cb
Scrapy设置headers、cookies三种方法
江冷易水寒
08-15 1万+
方法一：start_requests方法中添加cookies与headers： 1、重写spider中的start_requests方法 浏览器中cookies是这样的 Cookie:_T_WM=98075578786; WEIBOCN_WM=3349; H5_wentry=H5; backURL=https%3A%2F%2Fm.weibo.cn%2Fdetail%2F4396824548...
在scrapy中设置headers
梅花14的博客
09-28 4385
现在的大部分网站都进行了反爬虫措施，比如知乎、猫眼和豆瓣等等。那么作为一个spider我们就要学会反反爬虫，而且这是反爬虫工程师和反反爬虫工程师之间的长期斗争，就像矛与盾谁更厉害一样。扯远了，下面介绍一种最简单最实用的反反爬虫措施，就是设置headers。 方式一： 在settings里面开启DEFAULT_REQUEST_HEADERS，根据需求设置相关内容即可，主要就是配置"User-Agen...
Python爬虫框架Scrapy教程 完整版PDF
04-06
《Python爬虫框架Scrapy教程》主要是针对学习python爬虫的课程，又基础的python爬虫框架scrapy开始，一步步学习到最后完整的爬虫完成，现在python爬虫应用的非常广泛，本文档详细介绍了scrapy爬虫和其他爬虫技术的...
精通Python爬虫框架Scrapy.pdf
12-20
精通Python爬虫框架Scrapy.pdf
Python爬虫进阶：Scrapy框架教程
06-11
掌握了python爬虫基本理论和操作，本课程主要以讲授Scarpy框架来实现python爬虫进阶 该课程主要分享Python爬虫框架Scarpy的安装、使用、selector选择器的使用 spider的使用、itempipeline的使用、scrapy爬虫框架案例...
【Python爬虫：Scrapy】 之 PyCharm 搭建Scrapy环境+创建Scrapy项目 实例
12-21
Scrapy是一个强大的Python爬虫框架，它提供了一整套解决方案，包括数据提取、处理、存储等，适用于各种网络爬虫的开发。 首先，确保你已经安装了Python 3.6以及集成开发环境PyCharm。接下来，我们需要安装Scrapy...
scrapy中的headers，Request，response
King
03-19 1826
headers，Request，response
python之scrapy框架中间件，请求头，IP代理及fake-useragent使用，及面试中正向代理、反向代理区别
风华浪浪的博客
04-14 6275
什么下载器中间件？ 引擎与下载器之间的一个组件，一个爬虫可以有很多下载器中间件I
Python爬虫：关于scrapy模块的请求头
热门推荐
彭世瑜的博客
06-22 2万+
内容简介 使用scrapy写爬虫的时候，会莫名其妙的被目标网站拒绝，很大部分是浏览器请求头的原因，现在一起来看看scrapy的请求头 工具准备 开发环境python2.7 + scrapy 1.1.2 测试请求头网站：https://httpbin.org/get?show_env=1 json在线解析：https://www.json.cn/ 浏览器请求头大全： http://w...
scrapy设置headers和cookies
nnjy_1993的博客
06-12 408
0. 在settings.py中设置。
Scrapy-请求和响应
奔跑的豆子的专栏
04-07 1029
Scrapy使用Request和Response对象来爬行网站。通常，Request对象是在爬虫中生成的，并在整个系统中传递，直到它们到达下载器，后者执行请求并返回一个Response对象，该对象返回发出Request的爬虫。Request和Response类都有子类，这些子类添加了基类中不需要的功能。这些将在下面的Request子类和Response子类中进行描述。
scrapy各种请求参数
wang785994599的博客
10-10 3320
get请求 查询字符串 1.使用FormRequest start_urls = ['https://careers.tencent.com/tencentcareer/api/post/Query?'] def start_requests(self): for url in self.start_urls: params = { "timestamp": str(int(time.time() * 1000)),
网络爬虫---scrapy项目中Request()的meta参数详解
jing_zzy的博客
11-19 3812
1.scrapy项目中Request()的参数meta的解释 Request的参数meta作用是将meta中的信息传递个下一个回调函数，使用过程可以理解为： ''' 如果回调函数中，需要用到上一个函数中的数据， 那我们可以把该数据赋值给变量meta，让它传给回调函数， 但是meta只接受字典类型的数据， 所以，我们要把待传递的信息改成字典形式，即： meta = {'key1': 'value1', 'key2': 'value2'} 如果想在下一个函数中取出value1: value = r
Python爬虫：scrapy中间件及一些参数
彭世瑜的博客
07-22 1898
scrapy中间件 from scrapy.settings import default_settings 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, 'scrapy.do...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值