python爬取知乎话题："你见过最搞笑的视频有哪些？"下的所有视频

①打开知乎话题 https://www.zhihu.com/question/21086211

②打开开发者工具，构造Request url，和上次的爬取照片思路一样（https://blog.csdn.net/YiXiao1997/article/details/86655584）

'''
requests.get()返回的是json
对它进行字典化，好提取键值
'''
from urllib import request
from bs4 import BeautifulSoup
import requests
import re
import json
import math
def getVideo():
    m = 0#计数字串个数
    num = 0#回答者个数
    kv = {'user-agent':'Mozillar/5.0'}
    for i in range(math.ceil(783/20)):
        url = "https://www.zhihu.com/api/v4/questions/21086211/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=20&offset="+str(i*20)+"&platform=desktop&sort_by=default"
        r = requests.get(url,headers = kv)
        dicurl = json.loads(r.text)#转化成python的对象

③使用谷歌浏览器的一个开发者工具JSONview，可以看到打开的url中有一个content，这里面就是我们要找的回答内容，视频url也在里面

 ④将返回的json转化成python对象后，获取其中content里面的内容

content = dicurl["data"][k]["content"]

⑤打开获取的content，找到href后面的url，打开看一下

 （1）打开后视频正是我们要的内容，但是发现url不是我们获取的这个https://link.zhihu.com/target=https%3A//www.zhihu.com/video/978544217094533120，仔细观察后发现，这个url发生了跳转

 （2）想要知道如何跳转来的，我们再次F12，打开开发者工具，发现请求了一个新的URL

 (3)https://v.vzuu.com/video/978544217094533120，这个就是视频播放地址了，这个地址也很好构造。末尾的一串数字是不是很熟悉，data-lens-id里面就是这串数字

 （4）对这个地址进行构造

​
videoUrl = "https://lens.zhihu.com/api/v4/videos/"+str(data_lens[j])

​

⑥对构造url进行循环，然后就是访问这些url，通过urllib库和os库进行存储了，然后贴出全部代码

from urllib import request
from bs4 import BeautifulSoup
import requests
import re
import json
import math
def getVideo():
    m = 0#计数字串个数
    num = 0#回答者个数
    kv = {'user-agent':'Mozillar/5.0'}
    for i in range(math.ceil(783/20)):
        try:
            url = "https://www.zhihu.com/api/v4/questions/21086211/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=20&offset="+str(i*20)+"&platform=desktop&sort_by=default"
            r = requests.get(url,headers = kv)
            dicurl = json.loads(r.text)
            for k in range(20):#每条dicurl里可以解析出20条content数据
                name = dicurl["data"][k]["author"]["name"]
                ID = dicurl["data"][k]["id"]
                question = dicurl["data"][k]["question"]["title"]
                content = dicurl["data"][k]["content"]
                data_lens = re.findall(r'data-lens-id="(.*?)"',content)
                print("正在处理第" + str(num+1) + "个回答--回答者昵称:" + name + "--回答者ID:" + str(ID) + "--" + "问题:" + question)
                num = num + 1  # 每次碰到一个content就增加1，代表回答者人数
                for j in range(len(data_lens)):
                    try:
                        videoUrl = "https://lens.zhihu.com/api/v4/videos/"+str(data_lens[j])
                        R = requests.get(videoUrl,headers = kv)
                        Dicurl = json.loads(R.text)
                        playurl = Dicurl["playlist"]["LD"]["play_url"]
                        #print(playurl)#跳转后的视频url
                        videoread = request.urlopen(playurl).read()
                        print(">>>>>>>>>>>>>>>>>第---" + str(m+1) + "---个视频下载完成<<<<<<<<<<<<<<<<<")
                        with open("C:/Users/23504/Desktop/Python知乎数据/知乎话题：你见过最搞笑的视频有哪些？/"+str(m+1)+".mp4", 'wb') as video:
                            video.write(videoread)
                        m = m+1
                    except:
                        print("此URL为外站视频,不符合爬取规则")
        except:
            print("构造第"+str(i+1)+"条json数据失败")

if __name__ == "__main__":
    getVideo()

Tip1：有些用户的视频是转发外站的，这些都没有太多规则，不能用这个批量爬取，因为这个没有规则的视频很少，不影响爬取量，直接使用try规避掉就行了

Tip2:可以搞一搞多线程，这个太慢了

ShowTime: