python爬取B站《元龙》评论

本次爬取B站评论的目标，我们选取最近正在热播的《元龙》。
1、前期准备
首先我们先进入到《元龙》的页面

接下来按F12进入开发者模式
接下来点击headers,我们发现了一个url，这个url里面就存在我们做需要的评论的数据。https://api.bilibili.com/x/v2/reply/main?callback=jQuery17209057814175272192_1631448075244&jsonp=jsonp&next=3&type=1&oid=673934753&mode=3&plat=1&_=1631449433781

把这个url复制到浏览器打开可以看到里面的json，但是如果直接复制到浏览器页面会出错，需要对这个url进行处理，处理成https://api.bilibili.com/x/v2/reply?&type=1&oid=673934753&np=2这种形式就可以复制到浏览器打开了。

2、编程部分
首先对网页内容进行爬取

import requests
import json
import time

def fenchUrl(url):
    headers =  {
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
    }
    r = requests.get(url, headers=headers)
    r.raise_for_status()
    print(url)
    return r.text

接下来对json中所需内容进行提取，本次所需内容为：用户名以及评论内容

def parserHtml(html):
    try:
        s = json.loads(html)
    except:
        print("error")
    commentList = []
    hlist = []
    hlist.append("姓名")
    hlist.append("评论")

    for i in range(20):
        comment = s['data']['replies'][i]
        blist = []
        userName = comment['member']['uname']
        content = comment['content']['message']
        blist.append(userName)
        blist.append(content)
        commentList.append(blist)
    writePage(commentList)
    print("---" * 20)

最后对所爬取的内容进行保存

def writePage(urating):
    import pandas as pd
    dataFrame = pd.DataFrame(urating)
    print(dataFrame)
    dataFrame.to_csv(r"E:\test.csv",mode='a', index=False, sep=',', header=False)

爬取结果如下

完整代码：

import requests
import json
import time

def fenchUrl(url):
    headers =  {
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
    }
    r = requests.get(url, headers=headers)
    r.raise_for_status()
    print(url)
    return r.text
def parserHtml(html):
    try:
        s = json.loads(html)
    except:
        print("error")
    commentList = []
    hlist = []
    hlist.append("姓名")
    hlist.append("评论")

    for i in range(20):
        comment = s['data']['replies'][i]
        blist = []
        userName = comment['member']['uname']
        content = comment['content']['message']
        blist.append(userName)
        blist.append(content)
        commentList.append(blist)
    writePage(commentList)
    print("---" * 20)
def writePage(urating):
    import pandas as pd
    dataFrame = pd.DataFrame(urating)
    print(dataFrame)
    dataFrame.to_csv(r"E:\test.csv",mode='a', index=False, sep=',', header=False)
if __name__ == '__main__':
    for page in range(0, 5):
        url = 'https://api.bilibili.com/x/v2/reply?type=1&oid=461496816&pn=' + str(page)
        html = fenchUrl(url)
        parserHtml(html)
        if page % 20 ==0:
            time.sleep(5)