python爬虫翻页代码 豆瓣_Python爬虫 豆瓣动态页面的爬取

动态页面和静态页面可通过检查元素查看爬取的信息和查看源代码中的信息是否一致，一致则为静态页面，反则为动态页面。因为检查查看的代码是经过处理后生成的，而查看源代码形式是你实际抓取的页面。

1.爬虫思路

构造url，关键词的部分需要通过urllib.parse.urlencode()解码

通过urllib.request.Request()包装url

通过urllib.request.urlopen()返回一个response对象

最后通过simplejson装载response.read()

2.上代码

from urllib.request import Request,urlopen

from urllib import parse

import simplejson

base_url = "https://movie.douban.com/j/search_subjects?"

d = {

"type":"movie",

"tag" : "热门",

"page_limit": 50,

"page_start": 0

}

ua = {

'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'

}

#url关键字的部分需要编码，看到的url是经内部转换成中文

newurl = "{}{}".format(base_url,parse.urlencode(d))

req = Request(newurl,headers = ua)

#urllb.request.Requests是进一步包装url

with urlopen(req) as rsp:

#urlopen返回一个httpresponse对象用rsp接受

subjects = simplejson.loads(rsp.read())

print(subjects['subjects'])

3.结果

:\python\venv\Scripts\python.exe C:/python/dynamicspider.py

[{'rate': '8.8', 'cover_x': 1500, 'title': '寄生虫', 'url': 'https://movie.douban.com/subject/27010768/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2561439800.jpg', 'id': '27010768', 'cover_y': 2138, 'is_new': False}, {'rate': '7.7', 'cover_x': 1500, 'title': '恶人传', 'url': 'https://movie.douban.com/subject/30211551/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2555084871.jpg', 'id': '30211551', 'cover_y': 2145, 'is_new': False}, {'rate': '6.6', 'cover_x': 1200, 'title': '三更半夜居然要香蕉 爱的真实故事', 'url': 'https://movie.douban.com/subject/30203549/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2537936671.jpg', 'id': '30203549', 'cover_y': 1698, 'is_new': True}, {'rate': '6.6', 'cover_x': 1000, 'title': '作家的谎言：笔忠诱罪', 'url': 'https://movie.douban.com/subject/30170120/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2564834942.jpg', 'id': '30170120', 'cover_y': 1407, 'is_new': True}, {'rate': '8.9', 'cover_x': 2000, 'title': '绿皮书', 'url': 'https://movie.douban.com/subject/27060077/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2549177902.jpg', 'id': '27060077', 'cover_y': 3167, 'is_new': False}, {'rate': '7.9', 'cover_x': 1786, 'title': '流浪地球', 'url': 'https://movie.douban.com/subject/26266893/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2545472803.jpg', 'id': '26266893', 'cover_y': 2500, 'is_new': False}, {'rate': '5.5', 'cover_x': 2499, 'title': '追龙Ⅱ', 'url': 'https://movie.douban.com/subject/30175306/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2559100280.jpg', 'id': '30175306', 'cover_y': 3500, 'is_new': False}, {'rate': '8.0', 'cover_x': 3600, 'title': '疾速备战', 'url': 'https://movie.douban.com/subject/26909790/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2551393832.jpg', 'id': '26909790', 'cover_y': 5550, 'is_new': False}, {'rate': '6.4', 'cover_x': 960, 'title': '疯狂的外星人', 'url': 'https://movie.douban.com/subject/25986662/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2541901817.jpg', 'id': '25986662', 'cover_y': 1359, 'is_new': False}, {'rate': '7.5', 'cover_x': 5000, 'title': '阿丽塔：战斗天使', 'url': 'https://movie.douban.com/subject/1652592/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2544987866.jpg', 'id': '1652592', 'cover_y': 7413, 'is_new': False}, {'rate': '8.6', 'cover_x': 3000, 'title': '复仇者联盟4：终局之战', 'url': 'https://movie.douban.com/subject/26100958/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2552058346.jpg', 'id': '26100958', 'cover_y': 4440, 'is_new': False}, {'rate': '5.8', 'cover_x': 5906, 'title': '妈阁是座城', 'url': 'https://movie.douban.com/subject/26304201/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2557267786.jpg', 'id': '26304201', 'cover_y': 8268, 'is_new': False}, {'rate': '6.9', 'cover_x': 7142, 'title': '飞驰人生', 'url': 'https://movie.douban.com/subject/30163509/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2542973862.jpg', 'id': '30163509', 'cover_y': 10000, 'is_new': False}, {'rate': '8.2', 'cover_x': 683, 'title': '沦落人', 'url': 'https://movie.douban.com/subject/30140231/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2536165101.jpg', 'id': '30140231', 'cover_y': 960, 'is_new': False}, {'rate': '7.9', 'cover_x': 5315, 'title': '白蛇：缘起', 'url': 'https://movie.douban.com/subject/30331149/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2544313786.jpg', 'id': '30331149', 'cover_y': 7441, 'is_new': False}, {'rate': '7.2', 'cover_x': 2050, 'title': '毒液：致命守护者', 'url': 'https://movie.douban.com/subject/3168101/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2537158013.jpg', 'id': '3168101', 'cover_y': 2910, 'is_new': False}, {'rate': '8.6', 'cover_x': 2740, 'title': '痛苦与荣耀', 'url': 'https://movie.douban.com/subject/30198539/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2554080075.jpg', 'id': '30198539', 'cover_y': 3732, 'is_new': False}, {'rate': '8.4', 'cover_x': 1944, 'title': '孟买酒店', 'url': 'https://movie.douban.com/subject/26794701/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2545020183.jpg', 'id': '26794701', 'cover_y': 2880, 'is_new': False}, {'rate': '7.6', 'cover_x': 6750, 'title': '海王', 'url': 'https://movie.douban.com/subject/3878007/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2541280047.jpg', 'id': '3878007', 'cover_y': 10000, 'is_new': False}, {'rate': '8.6', 'cover_x': 2200, 'title': '蜘蛛侠：平行宇宙', 'url': 'https://movie.douban.com/subject/26374197/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2542867516.jpg', 'id': '26374197', 'cover_y': 3123, 'is_new': False}, {'rate': '8.1', 'cover_x': 1429, 'title': '无名之辈', 'url': 'https://movie.douban.com/subject/27110296/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2539661066.jpg', 'id': '27110296', 'cover_y': 2000, 'is_new': False}, {'rate': '8.1', 'cover_x': 1000, 'title': '无双', 'url': 'https://movie.douban.com/subject/26425063/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2535260806.jpg', 'id': '26425063', 'cover_y': 1400, 'is_new': False}, {'rate': '5.7', 'cover_x': 1800, 'title': '黑衣人：全球追缉', 'url': 'https://movie.douban.com/subject/19971676/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2558701068.jpg', 'id': '19971676', 'cover_y': 2571, 'is_new': False}, {'rate': '7.8', 'cover_x': 1900, 'title': '阿拉丁', 'url': 'https://movie.douban.com/subject/26891256/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2553992741.jpg', 'id': '26891256', 'cover_y': 2715, 'is_new': False}, {'rate': '8.3', 'cover_x': 5906, 'title': '调音师', 'url': 'https://movie.douban.com/subject/30334073/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2551995207.jpg', 'id': '30334073', 'cover_y': 8268, 'is_new': False}, {'rate': '7.9', 'cover_x': 5852, 'title': '地久天长', 'url': 'https://movie.douban.com/subject/26715636/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2550208359.jpg', 'id': '26715636', 'cover_y': 8192, 'is_new': False}, {'rate': '5.2', 'cover_x': 1400, 'title': '跳舞吧！大象', 'url': 'https://movie.douban.com/subject/30259043/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2562934437.jpg', 'id': '30259043', 'cover_y': 2000, 'is_new': False}, {'rate': '8.7', 'cover_x': 3000, 'title': '波西米亚狂想曲', 'url': 'https://movie.douban.com/subject/5300054/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2549558913.jpg', 'id': '5300054', 'cover_y': 4418, 'is_new': False}, {'rate': '6.4', 'cover_x': 1500, 'title': '哥斯拉2：怪兽之王', 'url': 'https://movie.douban.com/subject/25890017/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2554370800.jpg', 'id': '25890017', 'cover_y': 2150, 'is_new': False}, {'rate': '5.4', 'cover_x': 1286, 'title': '最好的我们', 'url': 'https://movie.douban.com/subject/26266919/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2557157554.jpg', 'id': '26266919', 'cover_y': 1800, 'is_new': False}, {'rate': '6.6', 'cover_x': 1350, 'title': '大侦探皮卡丘', 'url': 'https://movie.douban.com/subject/26835471/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2555538168.jpg', 'id': '26835471', 'cover_y': 2000, 'is_new': False}, {'rate': '8.0', 'cover_x': 5906, 'title': '夏目友人帐', 'url': 'https://movie.douban.com/subject/27166442/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2546745948.jpg', 'id': '27166442', 'cover_y': 8268, 'is_new': False}, {'rate': '8.1', 'cover_x': 1968, 'title': '无敌破坏王2：大闹互联网', 'url': 'https://movie.douban.com/subject/20438964/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2537667301.jpg', 'id': '20438964', 'cover_y': 2756, 'is_new': False}, {'rate': '7.1', 'cover_x': 2048, 'title': '神奇动物：格林德沃之罪', 'url': 'https://movie.douban.com/subject/26147417/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2538826177.jpg', 'id': '26147417', 'cover_y': 3035, 'is_new': False}, {'rate': '8.0', 'cover_x': 2000, 'title': '小委托人', 'url': 'https://movie.douban.com/subject/33404726/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2557335459.jpg', 'id': '33404726', 'cover_y': 2865, 'is_new': False}, {'rate': '7.2', 'cover_x': 1372, 'title': '风中有朵雨做的云', 'url': 'https://movie.douban.com/subject/26728669/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2552522615.jpg', 'id': '26728669', 'cover_y': 1920, 'is_new': False}, {'rate': '7.0', 'cover_x': 1080, 'title': '大黄蜂', 'url': 'https://movie.douban.com/subject/26394152/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2541662397.jpg', 'id': '26394152', 'cover_y': 1590, 'is_new': False}, {'rate': '6.9', 'cover_x': 5907, 'title': '惊奇队长', 'url': 'https://movie.douban.com/subject/26213252/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2546360443.jpg', 'id': '26213252', 'cover_y': 8269, 'is_new': False}, {'rate': '5.7', 'cover_x': 835, 'title': '偷心女盗', 'url': 'https://movie.douban.com/subject/3230155/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2546207183.jpg', 'id': '3230155', 'cover_y': 1108, 'is_new': False}, {'rate': '7.2', 'cover_x': 3927, 'title': '影', 'url': 'https://movie.douban.com/subject/4864908/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2530513100.jpg', 'id': '4864908', 'cover_y': 5788, 'is_new': False}, {'rate': '7.2', 'cover_x': 1200, 'title': '城市猎人', 'url': 'https://movie.douban.com/subject/30335799/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2542455343.jpg', 'id': '30335799', 'cover_y': 1630, 'is_new': False}, {'rate': '5.7', 'cover_x': 1800, 'title': '新喜剧之王', 'url': 'https://movie.douban.com/subject/4840388/', 'playable': True, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2555440969.jpg', 'id': '4840388', 'cover_y': 2524, 'is_new': False}, {'rate': '7.8', 'cover_x': 1575, 'title': '海市蜃楼', 'url': 'https://movie.douban.com/subject/30164448/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2551172384.jpg', 'id': '30164448', 'cover_y': 2329, 'is_new': False}, {'rate': '7.2', 'cover_x': 1153, 'title': '密室逃生', 'url': 'https://movie.douban.com/subject/27109679/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2543631842.jpg', 'id': '27109679', 'cover_y': 1700, 'is_new': False}, {'rate': '8.3', 'cover_x': 1432, 'title': '我的一级兄弟', 'url': 'https://movie.douban.com/subject/30143333/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2548467439.jpg', 'id': '30143333', 'cover_y': 2048, 'is_new': False}, {'rate': '7.4', 'cover_x': 2000, 'title': '驯龙高手3', 'url': 'https://movie.douban.com/subject/19899707/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2546335362.jpg', 'id': '19899707', 'cover_y': 2975, 'is_new': False}, {'rate': '6.8', 'cover_x': 1500, 'title': '红海潜水俱乐部', 'url': 'https://movie.douban.com/subject/27006407/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2564653267.jpg', 'id': '27006407', 'cover_y': 2222, 'is_new': False}, {'rate': '8.2', 'cover_x': 3000, 'title': '罗马', 'url': 'https://movie.douban.com/subject/1950330/', 'playable': True, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2555541430.jpg', 'id': '1950330', 'cover_y': 4393, 'is_new': False}, {'rate': '5.6', 'cover_x': 1458, 'title': '十二个想死的孩子', 'url': 'https://movie.douban.com/subject/30328584/', 'playable': False, 'cover': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2543516380.jpg', 'id': '30328584', 'cover_y': 2048, 'is_new': False}, {'rate': '6.7', 'cover_x': 1200, 'title': '假面饭店', 'url': 'https://movie.douban.com/subject/27126336/', 'playable': False, 'cover': 'https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2531219578.jpg', 'id': '27126336', 'cover_y': 1694, 'is_new': False}]