模拟get和post请求，获取分页数据

get请求数据

一、导入模块

import urllib.request
import urllib.parse

二、根据get参数，生成完整的url，构造headers

page = int(input('请输入您想要的第几页电影：'))
base_url = url = 'https://movie.douban.com/j/chart/top_list?type=5&interval_id=100%3A90&action=&'
start = (page-1)*20
limit = 20

data = {
    'start':start,
    'limit':limit
}

• 将参数data拼接为query_string

data = urllib.parse.urlencode(data)

• 构造完整的url

url = base_url+data

设置user-agent

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36'
}

三、模拟请求

request = urllib.request.Request(url=url,headers=headers)

四、获取响应数据

response = urllib.request.urlopen(request)

五、输出响应结果

print(response.read().decode('utf-8'))

post获取数据

import urllib.request
import urllib.parse

data = {
    'cname': '北京',
    'pid': '',
    'pageIndex': '3',
    'pageSize': '10',
}
data = urllib.parse.urlencode(data).encode('utf-8')

url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=cname'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36'
}

request = urllib.request.Request(url=url, headers=headers, data=data)
reponse = urllib.request.urlopen(request)

print(reponse.read().decode('utf-8'))