python 爬虫学习入门6 requests库添加代理proxies等其他信息

最新推荐文章于 2024-08-26 14:01:09 发布

陈逸飞_p

最新推荐文章于 2024-08-26 14:01:09 发布

阅读量2.4k

点赞数

分类专栏：爬虫文章标签： python

本文链接：https://blog.csdn.net/qq_43475705/article/details/113182305

版权

爬虫专栏收录该内容

8 篇文章 0 订阅

订阅专栏

Requests库

通过Requests 库提供的方法我们可以在请求页面的时候同时添加更多的信息，在urllib库中同时添加代理和User-agent 我在网上找了一些办法但是没有理解就不在这里说了，有兴趣的可以自己在网上搜。

Request 库中添加代理的方法是传递proxies 参数, 同时可以设置超时等待条件
如下：

import requests

url = "http://www.baidu.com"


# 添加User-agent 等 头部信息
headers = {
    'User-agent': '123'
}

# 添加proxies

proxies = {
	'http': '195.123.123.123:8080'
}

response = request.get(url, headers=headers, proxies=proxies, timeout=10)

data = response.content.decode('utf-8')

下面是request中定义的参数, 可以根据下面的注释内容来根据自己的需求来添加信息

def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.

    :param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary, list of tuples or bytes to send
        in the query string for the :class:`Request`.
    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
        object to send in the body of the :class:`Request`.
    :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
        ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
        or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
        defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
        to add for the file.
    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
    :param timeout: (optional) How many seconds to wait for the server to send data
        before giving up, as a float, or a :ref:`(connect timeout, read
        timeout) <timeouts>` tuple.
    :type timeout: float or tuple
    :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
    :type allow_redirects: bool
    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
    :param verify: (optional) Either a boolean, in which case it controls whether we verify
            the server's TLS certificate, or a string, in which case it must be a path
            to a CA bundle to use. Defaults to ``True``.
    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response

    Usage::

      >>> import requests
      >>> req = requests.request('GET', 'https://httpbin.org/get')
      >>> req
      <Response [200]>
    """