Python爬虫—urllib的基本使用

京茶吉鹿

已于 2022-07-03 20:12:30 修改

阅读量1.2k

点赞数 1

分类专栏： Python爬虫文章标签： python 爬虫开发语言

于 2022-07-03 15:50:29 首次发布

本文链接：https://blog.csdn.net/weixin_52372879/article/details/125585000

版权

Python爬虫专栏收录该内容

5 篇文章 0 订阅

订阅专栏

urllib的基本使用

文章包含了urllib的基本使用，本文不包含具体的使用说明，多通过实例来展示其在实际应用中的使用。

1、基本使用

使用步骤如下

定义一个要访问的url

url = 'http://www.baidu.com'

模拟浏览器向服务器发送请求

response = urllib.request.urlopen(url)

获取响应中的页面的源码

# 注：通过read()返回的是二进制数据,需要转换为字符串
content = response.read().decode('utf-8')

打印数据

print(content)

2、一个类型和六个方法

一个类型: HTTPResponse
六个方法:
- response.read() 一个字节一个字节读取,返回二进制数据
  - response.read(x) 返回 x 个字符串
- response.readline() 读取一行,返回二进制字符串
- response.readlines() 读取所有行,返回字符串列表
- response.getcode() 返回状态码,200表示成功
- response.geturl() 返回 url 地址
- response.getheaders() 获取一些状态信息

3、下载(urllib.request.urlretrieve())

下载格式：urllib.request.urlretrieve(url,filename)

3.1、下载网页

import urllib.request

url = 'http://www.baidu.com'
urllib.request.urlretrieve(url,'baidu.html')

3.2、下载图片

import urllib.request

url = 'https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fimg.pconline.com.cn%2Fimages%2Fupload%2Fupc%2Ftx%2Fitbbs%2F1402%2F19%2Fc2%2F31366623_1392766468790_mthumb.jpg&refer=http%3A%2F%2Fimg.pconline.com.cn&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=auto?sec=1659408581&t=b73ab24660920aee6e9ea4e32304f985'
urllib.request.urlretrieve(url,'sunset.jpg')

3.3、下载视频

import urllib.request

url = 'https://vd2.bdstatic.com/mda-jkp3m3c677244gen/sc/mda-jkp3m3c677244gen.mp4'
urllib.request.urlretrieve(url,'sunset.mp4')

4、请求对象的定制

4.1、查找UA(User-Agent)

以 chrome 为例:

打开百度首页 —> f12 —> 刷新页面 —> Network —> 找到www.baidu.com —> Headers --> 滑到底部

注：后续 headers 中包含的内容，都可以通过此种方式查找

4.2、应用实例

import urllib.request

url = 'https://www.baidu.com'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
# 因为参数顺序问题,所以需要使用关键字传参
request = urllib.request.Request(url=url,headers=headers)
response = urllib.request.urlopen(request)
content = response.read().decode('utf-8')
print(content)

5、get请求

5.1、quote方法

说明：将文字转换为 Unicode 编码格式
使用场景：url中含有单个字符串
应用实例：

import urllib.parse

base_url = 'https://www.baidu.com/s?&wd='
name = urllib.parse.quote('邓紫棋')
url = base_url + name
# 下面的内容同 请求对象的定制 应用实例

5.2、urlencode方法

说明：将字典参数转换为 url 查询字符串
使用场景：url中含有多个字符串
实例：

import urllib.parse

base_url = 'https://www.baidu.com/s?'
data = {
    'wd': '邓紫棋',
    'sex': '女',
    'location': '中国台湾'
}
new_data = urllib.parse.urlencode(data)
url = base_url + new_data
# 下面的内容同 请求对象的定制 应用实例

5.3、应用实例：获取豆瓣电影的前十页

import urllib.parse
import urllib.request

def create_request(page):
    base_url = 'https://movie.douban.com/j/chart/top_list?type=13&interval_id=100%3A90&action=&'
    data = {
        'start':(page-1)*20,
        'limit':20
    }
    url = base_url + urllib.parse.urlencode(data)
    headers = {
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
    request = urllib.request.Request(url=url,headers=headers)
    return request

def get_content(request):
    response = urllib.request.urlopen(request)
    content = response.read().decode("utf-8")
    return content

def down_load(page,content):
    with open('douban'+ str(page) +'.json','w',encoding='utf-8') as fp:
        fp.write(content)

if __name__ == '__main__':
    start_page = int(input("请输入起始页码: "))
    end_page = int(input("请输入结束页码: "))
    for page in range(start_page,end_page+1):
        request = create_request(page)
        # 获取响应的数据
        content = get_content(request)
        # 保存数据
        down_load(page, content)

6、post请求

6.1、应用实例:百度翻译

import json
import urllib.request

url = 'https://fanyi.baidu.com/sug'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
data = {
    'kw':'spider'
}
# post请求的参数，必须进行编码
data = urllib.parse.urlencode(data).encode("utf-8")

request = urllib.request.Request(url=url,data=data,headers=headers)
response = urllib.request.urlopen(request)
content = response.read().decode("utf-8")

obj = json.loads(content)
print(obj)

6.2、应用实例:百度翻译之详细翻译

import json
import urllib.request

url = 'https://fanyi.baidu.com/v2transapi?from=en&to=zh'
headers = {
    'Cookie': 'BIDUPSID=D722A196F9933FA81783CCC2D7F9C5F6; PSTM=1656228555; BAIDUID=D722A196F9933FA8D6BE0025261BFB25:FG=1; H_PS_PSSID=36549_36465_36726_36455_31254_36691_36166_36694_36698_36570_36736_26350_36467_36651; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; BAIDUID_BFESS=D722A196F9933FA8D6BE0025261BFB25:FG=1; delPer=0; PSINO=6; BA_HECTOR=21a52ha4aka120ah0l1hbshos15; ZFY=CE:BKsZiBwgpUTSY8QcuPYnou1hGa5Pzb:Bf1tQSZVCkU:C; Hm_lvt_64ecd82404c51e03dc91cb9e8c025574=1656637214; APPGUIDE_10_0_2=1; REALTIME_TRANS_SWITCH=1; FANYI_WORD_SWITCH=1; HISTORY_SWITCH=1; SOUND_SPD_SWITCH=1; SOUND_PREFER_SWITCH=1; ab_sr=1.0.1_MDBmNDRiYzM4YTVmMGJmODhiODY1ODBmMDAzYzc3MmYwZWY5NjZmNDYxMDFiZWZiNTUzZDQ2YTIxZmYxOWYzNzcwNjk2YmE5YjAxMjJmYTFkM2IzNTY5MTgyODlmZWMzNjY3NWZlYzZjOWVhMmZlMjRjNGU4YTk2YzAzYWVlNGJlY2JmOWI3ODJjYTZhNzg4YzBkMWM1MWQ5NGJmMmIzNA==; Hm_lpvt_64ecd82404c51e03dc91cb9e8c025574=1656638816',
}
data = {
    'from': 'en',
    'to': 'zh',
    'query': 'love',
    'simple_means_flag': '3',
    'sign': '198772.518981',
    'token': '9ae6c958cg675b93cb940e82a71f2ecc',
    'domain': 'common'
}
# post 请求的参数,必须进行编码
data = urllib.parse.urlencode(data).encode('utf-8')
request = urllib.request.Request(url=url,data=data,headers=headers)
response = urllib.request.urlopen(request)
content = response.read().decode('utf-8')
obj = json.loads(content)
print(obj)

6.3、应用实例:爬取肯德基官网

import urllib.request
import urllib.parse

def create_request(page):
    base_url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=cname'
    data = {
        'cname': '成都',
        'pid':'',
        'pageIndex': page,
        'pageSize': "10"
    }
    data = urllib.parse.urlencode(data).encode("utf-8")
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
    request = urllib.request.Request(url=base_url,data=data,headers=headers)
    return request

def get_content(request):
    response = urllib.request.urlopen(request)
    content = response.read().decode("utf-8")
    return content

def down_load(page, content):
    with open('kfc_'+str(page)+'.json','w',encoding='utf-8') as fp:
        fp.write(content)

if __name__ == '__main__':
    start_page = int(input("请输入起始页码: "))
    end_page = int(input("请输入结束页码: "))
    for page in range(start_page, end_page + 1):
        # 请求对象的定制
        request = create_request(page)
        # 获取响应的数据
        content = get_content(request)
        # 保存(下载)数据
        down_load(page, content)

7、handler处理器

使用handler访问百度,获取网页源码

import urllib.request

url = 'http://www.baidu.com'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
request = urllib.request.Request(url=url,headers=headers)

handler = urllib.request.HTTPHandler()
opener = urllib.request.build_opener(handler)
response = opener.open(request)

content = response.read().decode('utf-8')
print(content)

8、代理及代理池

代理 IP 可以在快代理网站上获取免费的IP使用

8.1、代理

import urllib.request

url = 'http://www.baidu.com/s?wd=ip'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
proxies = {
	'http':'47.106.105.236:80'
}
request = urllib.request.Request(url=url,headers=headers)

handler = urllib.request.ProxyHandler(proxies=proxies)
opener = urllib.request.build_opener(handler)
response = opener.open(request)

content = response.read().decode('utf-8')
print(content)

8.2、代理池

import random
import urllib.request

url = 'http://www.baidu.com/s?wd=ip'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
proxies_poll = {
    {'http': '47.106.105.236:80'},
    {'http': '118.24.219.151:16817'}
}
request = urllib.request.Request(url=url, headers=headers)

proxies = random.choice(proxies_poll)
handler = urllib.request.ProxyHandler(proxies=proxies)
opener = urllib.request.build_opener(handler)
response = opener.open(request)

content = response.read().decode('utf-8')
print(content)