基本库的使用

最新推荐文章于 2024-05-31 15:18:38 发布

a447380142

最新推荐文章于 2024-05-31 15:18:38 发布

阅读量176

点赞数

urllib

python3中没有urllib2

是内置的http请求库

包含4个模块：一、request，二、error，三、parse，四、roborparser（最后一个基本不用，可以判断哪些网站可以爬或者不可以爬）

一、request:

1.urlopen():可以模拟浏览器的请求过程

result：

可以发现，它是一个 HTTPResposne 类型的对象，主要包含 read （）、 readinto （）、 getheader(name ）、
getheaders （） fileno （）等方法，以及 msg、version、status、reason、debuglevel、 closed 等属性

得到这个对象之后，我们把它赋值为 response 变量，然后就可以调用这些方法和属性，得到返回结果的一系列信息了。

例如，调用 read （）方法可以得到返回的网页内容，调用 status 属性可以得到返回结果的状态码
如200 代表请求成功， 404 代表网页未找到等

>>> help(urllib.request.urlopen)
Help on function urlopen in module urllib.request:

urlopen(url, data=None, timeout=<object object at 0x00000000005AE2F0>, *, cafile=None, capath=None, cadefault=False, context=None)
    Open the URL url, which can be either a string or a Request object.

    *data* must be an object specifying additional data to be sent to
    the server, or None if no such data is needed. See Request for
    details.

urllib.request module uses HTTP/1.1 and includes a "Connection:close"
header in its HTTP requests.
-----------------------------------------------------------data-----------------------------------------------------------------------------------

data参数是可选的。如果要添加该参数，并且如果他是字节流编码格式的内容（bytes类型），则需要通过bytes()方法转化。另外，如果传递了这个参数，则它的请求方式就不再是GET方式，而是POST方式。

这里请求的站点是 httpbin.org ，它可以提供 HTTP 请求测试本次我们请求的 URL为http://httpbin.org/post ，这个链接可以用来测试 POST 请求，它可以输出请求的一些信息，其中包含我们传递的 data
参数

result：

------------------------------------------------------data-----------------------------------------------------------------------------------------

-----------------------------------------------------timeout-----------------------------------------------------------------------------------------

    The optional *timeout* parameter specifies a timeout in seconds for
    blocking operations like the connection attempt (if not specified, the
    global default timeout setting will be used). This only works for HTTP,
    HTTPS and FTP connections.

timeout:如上，如果请求超出了这个时间，还没得到响应，就抛出异常。未fq状态下访问Google肯定超时的啦。

..........

so，可以通过设置这个超时时间来控制一个网页如果长时间未响应，就跳过它的抓取。这可
利用 try except 语句来实现，相关代码如下：

result：

-----------------------------------------------------------------timeout--------------------------------------------------------------------------------------

---------------------------------------------------------------其他参数-------------------------------------------------------------------------------------

除了 data 参数和 timeout 参数外，还有 context 参数，它必须是 ssl.SSLContext 类型，用来指定SSL设置
此外， cafile和capath 这两个参数分别指定 CA 证书和它的路径，这个在请求 HTTPS 链接时会有用
cadefault 参数现在已经弃用了，其默认值为 False

    If *context* is specified, it must be a ssl.SSLContext instance describing
    the various SSL options. See HTTPSConnection for more details.

    The optional *cafile* and *capath* parameters specify a set of trusted CA
    certificates for HTTPS requests. cafile should point to a single file
    containing a bundle of CA certificates, whereas capath should point to a
    directory of hashed certificate files. More information can be found in
    ssl.SSLContext.load_verify_locations().

    The *cadefault* parameter is ignored.

    This function always returns an object which can work as a context
    manager and has methods such as

    * geturl() - return the URL of the resource retrieved, commonly used to
      determine if a redirect was followed

    * info() - return the meta-information of the page, such as headers, in the
      form of an email.message_from_string() instance (see Quick Reference to
      HTTP Headers)

    * getcode() - return the HTTP status code of the response. Raises URLError
      on errors.

    For HTTP and HTTPS URLs, this function returns a http.client.HTTPResponse
    object slightly modified. In addition to the three new methods above, the
    msg attribute contains the same information as the reason attribute ---
    the reason phrase returned by the server --- instead of the response
    headers as it is specified in the documentation for HTTPResponse.

    For FTP, file, and data URLs and requests explicitly handled by legacy
    URLopener and FancyURLopener classes, this function returns a
    urllib.response.addinfourl object.

    Note that None may be returned if no handler handles the request (though
    the default installed global OpenerDirector uses UnknownHandler to ensure
    this never happens).

    In addition, if proxy settings are detected (for example, when a *_proxy
    environment variable like http_proxy is set), ProxyHandler is default
    installed and makes sure the requests are handled through the proxy.