个人学习爬虫笔记之urllib模块的使用一

打开某一网站（两种请求类型）：

  1 import urllib.request
  2 
  3 response = urllib.request.urlopen('http://www.baidu.com') #request baidu,(type:get)
  4 print(response.read().decode('utf-8'))  #print the source webpage code of baidu

 

  1 import urllib.request
  2 import urllib.parse 
  6 
  7 data = bytes(urllib.parse.urlencode({'world': 'hello'}), encoding='utf-8')
  8 response = urllib.request.urlopen('http://httpbin.org/post', data=data)
  9 print(response.read())

超时并且抓取错误：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 
 15 
 16 try:
 17    response = urllib.request.urlopen('http://httpbin.org/post', timeout=0.1) #timeout and give
     error
 18 except urllib.error.URLError as e:
 19    if isinstance(e.reason, socket.timeout):
 20       print('TIME OUT')

响应： 

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 
  6 response = urllib.request.urlopen('http://www.python.org')
  7 print(type(response))

 状态码和响应头：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 
  6 response = urllib.request.urlopen('http://www.python.org')
  7 print(response.status)
  8 print(response.getheaders())
  9 print(response.getheader('Server'))
 10 
 11 

响应体的内容：

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 
  6 response = urllib.request.urlopen('http://www.python.org')
  7 print(response.read().decode('utf-8'))
  8 

request:

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 
  6 request = urllib.request.Request('http://www.python.org')
  7 response = urllib.request.urlopen(request)
  8 print(response.read().decode('utf-8'))
  9 

request的整体结构： 

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 from urllib import request, parse
  6 
  7 url = 'http://httpbin.org/post'
  8 headers = {
  9     'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64)',
 10     'Host': 'httpbin.org'}
 11 dict = {'name': 'Germey'}
 12 data = bytes(parse.urlencode(dict), encoding='utf-8')
 13 req = request.Request(url=url, data=data, headers=headers, method='POST')
 14 response = request.urlopen(req)
 15 print(response.read().decode('utf-8'))

利用方法添加header:

  1 import socket
  2 import urllib.request
  3 import urllib.parse
  4 import urllib.error
  5 from urllib import request, parse
  6 
  7 url = 'http://httpbin.org/post'
  8 dict = {'name': 'Germey'}
  9 data = bytes(parse.urlencode(dict), encoding='utf-8')
 10 req = request.Request(url=url, data=data, method='POST')
 11 req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64)')
 12 response = request.urlopen(req)
 13 print(response.read().decode('utf-8'))
 14