个人学习爬虫笔记之urllib模块的使用二​

Handler之代理：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 from urllib import request, parse
  6 
  7 proxy_handler = urllib.request.ProxyHandler({
  8     '代理http的ip',
  9     '代理https的ip'})
 10 opener = urllib.request.built_opener(proxy_handler)
 11 response = opener.open('http://httpbin.org/get')
 12 print(response.read())
 13 

 cookie(用于维持状态，例如登陆；)：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 
  8 cookie = http.cookiejar.CookieJar()
  9 handler = urllib.request.HTTPCookieProcessor(cookie)
 10 opener = urllib.request.build_opener(handler)
 11 response = opener.open('http://www.baidu.com')
 12 for item in cookie:
 13     print(item.name + '=' + item.value)
 14 

把cookie保存成文件：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 
  8 filename = 'cookie.txt'
  9 cookie = http.cookiejar.MozillaCookieJar(filename)
 10 handler = urllib.request.HTTPCookieProcessor(cookie)
 11 opener = urllib.request.build_opener(handler)
 12 response = opener.open('http://www.baidu.com')
 13 cookie.save(ignore_discard=True, ignore_expires=True)
 14     

cookie另一种保存方法：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 
  8 filename = 'cookie.txt'
  9 cookie = http.cookiejar.LWPCookieJar(filename)
 10 handler = urllib.request.HTTPCookieProcessor(cookie)
 11 opener = urllib.request.build_opener(handler)
 12 response = opener.open('http://www.baidu.com')
 13 cookie.save(ignore_discard=True, ignore_expires=True)
 14 

 把cookie读取出来，并且附进网站：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 
  8 cookie = http.cookiejar.LWPCookieJar()
  9 cookie.load('cookie.txt', ignore_discard=True, ignore_expires=True)
 10 handler = urllib.request.HTTPCookieProcessor(cookie)
 11 opener = urllib.request.build_opener(handler)
 12 response = opener.open('http://www.baidu.com')
 13 print(response.read().decode('utf-8'))
 14 

异常处理：

 

from urllib import request, error
try:
    response = request.urlopen('http://cuiqingcai.com.index.html')
except error.URLError as e:
    print(e.reason)

父类异常与子类异常：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 from urllib import request, error
  8 
  9 url1 = 'http://cuiqingcai.com.index.html'
 10 url2 = 'http://www.baidu.com'
 11 try:
 12     response = request.urlopen(url2)
 13 except error.HTTPError as e:
 14     print(e.reason, e.headers, sep='\n')
 15 except error.URLError as e:
 16     print(e.reason)
 17 else:
 18     print('Request Sussessfully')
 19 

 异常判断：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 import http.cookiejar
  6 from urllib import request, parse
  7 from urllib import request, error
  8 
  9 url1 = 'http://cuiqingcai.com.index.html'
 10 url2 = 'http://www.baidu.com'
 11 try:
 12     response = request.urlopen(url2, timeout=0.01)
 13 except error.URLError as e:
 14     print(type(e.reason))
 15     if isinstance(e.reason, socket.timeout):
 16         print('Time Out')
 17