python爬虫——Ajax-get

1.Ajax-get方法抓取豆瓣

    ~~~     1.1 准备工作

搜索‘豆瓣电影’-》排行榜-》选择一个类别（此处笔者选的动作）-》向下滑动可以看见最右边的滚动条滑到底部会弹回上面，此页面中就含有Ajax接口

用fidder抓包看json格式的信息

url解读：

https://movie.douban.com/j/chart/top_list?type=5&interval_id=100%3A90&action=&start=20&limit=20 

    ~~~     1.2编写代码

import urllib.request
import urllib.parse

url = 'https://movie.douban.com/j/chart/top_list?type=5&interval_id=100%3A90&action=&'
page = int(input("请输入你想要第几页："))
number = 5  # 设定每一页5条数据
# 构建get参数
data = {
    'start': (page - 1) * number,
    'limit': number,
}
# 将字典转化为query_string
query_string = urllib.parse.urlencode(data)
# 修改url
url += query_string
# 伪装头部
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36',
}
# 构建请求对象
request = urllib.request.Request(url=url, headers=headers)
# 发出请求
response = urllib.request.urlopen(request)
print(response.read().decode())

抓取的5部电影和前面一样

[{"rating":["9.4","50"],"rank":1,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p511118051.jpg","is_playable":true,"id":"1295644","types":["剧情","动作","犯罪"],"regions":["法国"],"title":"这个杀手不太冷","url":"https:\/\/movie.douban.com\/subject\/1295644\/","release_date":"1994-09-14","actor_count":20,"vote_count":1515720,"score":"9.4","actors":["让·雷诺","娜塔莉·波特曼","加里·奥德曼","丹尼·爱罗","彼得·阿佩尔","迈克尔·巴达鲁科","艾伦·格里尼","伊丽莎白·瑞根","卡尔·马图斯维奇","弗兰克·赛格","麦温","乔治·马丁","罗伯特·拉萨多","亚当·布斯奇","马里奥·托迪斯科","萨米·纳塞利","让·雨果·安格拉德","埃莱娜·卡多纳","大塚明夫","沈晓谦"],"is_watched":false},{"rating":["9.2","45"],"rank":2,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p462657443.jpg","is_playable":true,"id":"1851857","types":["剧情","动作","科幻","犯罪","惊悚"],"regions":["美国","英国"],"title":"蝙蝠侠：黑暗骑士","url":"https:\/\/movie.douban.com\/subject\/1851857\/","release_date":"2008-07-14","actor_count":39,"vote_count":629899,"score":"9.2","actors":["克里斯蒂安·贝尔","希斯·莱杰","艾伦·艾克哈特","迈克尔·凯恩","玛吉·吉伦哈尔","加里·奥德曼","摩根·弗里曼","莫尼克·加布里埃拉·库尔内","罗恩·迪恩","基里安·墨菲","黄经汉","内斯特·卡博内尔","埃里克·罗伯茨","里特奇·科斯特","安东尼·迈克尔·豪尔","基斯·斯扎拉巴基克","柯林·麦克法兰","约书亚·哈尔托","美琳达·麦格劳","内森·甘宝","迈克尔·维约","迈克尔·斯托扬诺夫","威廉·斯米利","丹尼·高德林","迈克尔·加·怀特","马修·奥尼尔","威廉·菲克纳","欧鲁米基·欧拉乌米","格雷格·比姆","爱利克·赫尔曼","毕崔斯·罗森","文森佐·尼克力","陈冠希","罗南·萨莫尔斯","大卫·达斯马齐连","玛丽扎·卡布蕾拉","理查德·迪兰","利萨·麦克阿里斯特","Adam Kalesperis"],"is_watched":false},{"rating":["9.2","45"],"rank":3,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p1910825503.jpg","is_playable":true,"id":"1291552","types":["剧情","动作","奇幻","冒险"],"regions":["美国","新西兰"],"title":"指环王3：王者无敌","url":"https:\/\/movie.douban.com\/subject\/1291552\/","release_date":"2004-03-12","actor_count":38,"vote_count":463426,"score":"9.2","actors":["维果·莫腾森","伊利亚·伍德","西恩·奥斯汀","丽芙·泰勒","伊恩·麦克莱恩","奥兰多·布鲁姆","凯特·布兰切特","米兰达·奥图","安迪·瑟金斯","雨果·维文","多米尼克·莫纳汉","比利·博伊德","马尔顿·索克斯","卡尔·厄本","克里斯托弗·李","约翰·瑞斯-戴维斯","阿兰·霍华德","伊安·霍姆","萨德文·布罗菲","布雷特·麦肯齐","劳伦斯·马克奥雷","Maisy McLeod-Riera","乔尔·托贝克","萨拉·贝克","杰德·布罗菲","姜广涛","彼得·杰克逊","张磊","伯纳德·希尔","约翰·贝西","Sarah McLeod","Noel Appleby","阿利斯泰尔·布朗宁","肖恩·宾","佩特·史密斯","大卫·文翰","约翰·诺贝尔","Harry Sinclair"],"is_watched":false},{"rating":["9.2","50"],"rank":4,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2215886505.jpg","is_playable":true,"id":"1295399","types":["动作","冒险","剧情"],"regions":["日本"],"title":"七武士","url":"https:\/\/movie.douban.com\/subject\/1295399\/","release_date":"1954-04-26","actor_count":20,"vote_count":118242,"score":"9.2","actors":["三船敏郎","志村乔","稻叶义男","宫口精二","千秋实","加东大介","木村功","本间文子","小杉义男","高木新平","中岛春雄","小川虎之助","东野英治郎","土屋嘉男","高堂国典","津岛惠子","藤原釜足","仲代达矢","左卜全","马野都留子"],"is_watched":false},{"rating":["9.1","45"],"rank":5,"cover_url":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p909265336.jpg","is_playable":true,"id":"1291572","types":["剧情","动作","奇幻","冒险"],"regions":["美国","新西兰"],"title":"指环王2：双塔奇兵","url":"https:\/\/movie.douban.com\/subject\/1291572\/","release_date":"2003-04-25","actor_count":29,"vote_count":429017,"score":"9.1","actors":["伊利亚·伍德","西恩·奥斯汀","伊恩·麦克莱恩","维果·莫腾森","奥兰多·布鲁姆","克里斯托弗·李","丽芙·泰勒","安迪·瑟金斯","雨果·维文","卡尔·厄本","凯特·布兰切特","多米尼克·莫纳汉","大卫·文翰","比利·博伊德","布拉德·道里夫","伯纳德·希尔","约翰·瑞斯-戴维斯","米兰达·奥图","肖恩·宾","陆建艺","杰德·布罗菲","丹·汉纳","菲利普·格里夫","张磊","姜广涛","克雷格·帕克","约翰·诺贝尔","劳伦斯·马克奥雷","彼得·杰克逊"],"is_watched":false}]