哔哩哔哩视频弹幕爬取

首先进入一个视频的界面，F12浏览network,限定XHR，播放视频，搜索list,这时会有一个如下截图一样的请求

Request URL:https://api.bilibili.com/x/player/pagelist?bvid=BV11D4y127tj&jsonp=jsonp，对该url发起请求后，会得到一个json，json里包含了cid

通过cid的拼接，‘https://comment.bilibili.com/’ + cid + ‘.xml’,得到一个url，该url里面包含了所有的弹幕内容
对上述url发起请求后得到的内容

我们发现所有的弹幕都存在d标签下面，通过find_all(’'d)即可获得所有的d标签,之后再依次提取标签里的text文本就获得了弹幕

总代码如下

import requests
from bs4 import BeautifulSoup
import re

url = 'https://api.bilibili.com/x/player/pagelist?bvid=BV11D4y127tj&jsonp=jsonp'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
}

response = requests.get(url=url,headers=headers)
list_data = response.json()
for list in list_data['data']:
    cid = str(list['cid'])
    url1 = 'https://comment.bilibili.com/' + cid + '.xml'
    dm_response = requests.get(url1)
    dm_response.encoding='utf-8'
    soup = BeautifulSoup(dm_response.text,'lxml')
    result=soup.find_all('d')
    fp = open('./danmu.txt','w',encoding='utf-8')
    for l in result:
        fp.write(l.text)
        fp.write('\n')
    print("over")