二. Urllib库详解

Urllib库的详解

什么是Urllib？

• urllib.request 请求模块

• urllib.error 异常处理模块

• urllib.parse url解析模块

• urllib.robotparser robots.txt解析模块

相比Python2变化

Python2:

import urllib2
response=urllib2.urlopen("http://www.baidu.com")

Python3:

import urllib.request
response=urllib.request.urlopen("http://www.baidu.com")

urlopen（）

urlopen()的函数原型：urllib.request.urlopen(url,data=None, [timeout,]*,…)

举例如下:

1.
 import urllib.request
 response=urllib.request.urlopen("http://www.baidu.com")
 print(response.read().decode("utf8"))

2.
import urllib.parse
import urllib.request
data=bytes(urllib.parse.urlencode({"word":"hello"}),encoding="utf8")
response=urllib.request.urlopen("http://httbin.org/get",data=data)                                   //http测试网站
print(response.read())

响应

import urllib.request
response=urllib.request.urlopen("http://www.google.com")
print (type(response))

状态码，响应头

import urllib.request
response=urllib.request.urlopen("http://www.google.com")
print(response.status)
print(response.getheaders())
print(response.getheader("Server"))

Request

from urllib import request,parse
url="http://httpbin.org/post"
headers={"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)",
"host":"httpbin.org"
}
dict={"name":"Germey"}
data=bytes(parse.urlencode(dict),encoding="utf8")
req=request.Request(url=url,data=data,headers=headers,method="post")
response=urllib.request.urlopen(req)
print(response.read()).decode("utf8")

Handler

代理

import urllib.request
proxy_handler=urllib.request.ProxyHandler({"http":"http://127.0.0.1:9743","https":"https://127.0.0.1:9743"})
opener=urllib.request.build_opener(proxy_handler)
response=opener.open("http://www.baidu.com")
print(response.read())

Cookie

cookie是用来保持用户对话的。

import urllib.request
import http.cookiejar

cookie=http.cookiejar.CookieJar()
handler=urllib.request.HTTPCookieProcessor(cookie)
opener=urllib.request.build_opener(handler)
response=opener.open("http://www.baidu.com")
for item in cookie:
    print(item.name+"="+item.value)

当然也可以把cookie保存成txt文件：

import urllib.request
import http.cookiejar

cookie=http.cookiejar.MozillaCookieJar(filename="cookie.txt")
handler=urllib.request.HTTPCookieProcessor(cookie)
opener=urllib.request.build_opener(handler)
response=opener.open("http://www.baidu.com")
cookie.save(ignore_discard=True,ignore_expires=True)

cookie的读取：

import urllib.request
import http.cookiejar

cookie=http.cookiejar.MozillaCookieJar(filename="cookie.txt")
cookie.load("cookie.txt",ignore_discard=True,ignore_expires=True)
handler=urllib.request.HTTPCookieProcessor(cookie)
opener=urllib.request.build_opener(handler)
response=opener.open("http://www.baidu.com")
print(response.read().decode("utf8"))

异常处理

from urllib import request,error
try:
response=request.urlopen("http://www.bangiuegiuidududu.com")
except error.URLError as e:
    print(e.reason)

URL解析

urllib.parse.urlparse(urlstring, scheme=”, allow_fragments=True)

from urllib.parse import urlparse
result=urlparse("http://www.baidu.com/index.html;user?id=5#comment")
print(type(result),result)

输出结果为+ ParseResult(scheme=’http’, netloc=’www.baidu.com’, path=’/index.html’, params=’user’, query=’id=5’, fragment=’comment’)

urlencode()

把字典对象转化成url请求参数

from urllib.parse import urlencode

parameters={"name":"germeny","id":"887"}
base_url="http://www.baidu.com?"
url=base_url+urlencode(parameters)
print(url)