新手入门python爬虫实战（三）爬取B站综合排名前两页的js教程视频

最新推荐文章于 2024-05-24 16:30:33 发布

Drone_xjw

最新推荐文章于 2024-05-24 16:30:33 发布

阅读量221

点赞数

分类专栏：数据挖掘文章标签： python 数据挖掘

本文链接：https://blog.csdn.net/xjw9602/article/details/106106302

版权

数据挖掘专栏收录该内容

13 篇文章 1 订阅

订阅专栏

如果觉得有帮助，麻烦点赞鼓励一下谢谢

重点

这里我遇到一个问题是当我获得了li标签里面的整个内容后，我应该怎么样才能得到li标签里面的a标签的title，原来可以是a[‘title’]

import requests
import bs4

def open_url(url):
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"}
    res = requests.get(url, headers=headers)
    return res

def get_title(res):
    soup = bs4.BeautifulSoup(res.text,"html.parser")
    content = soup.find_all("li",class_="video-item matrix")
    return content
    
def main():
    result = []
    with open("bilibili-js.txt", "w", encoding="utf-8") as file:
        ##这里只取了前两页
        for i in range(1, 3):
            page = "&page=" + str(i)
            url = "https://search.bilibili.com/all?keyword=js&order=totalrank&duration=4&tids_1=0" + page
            print(url)
            res = open_url(url)
            result = get_title(res)
            for each in result:
                print(each.a['title'])#li标签下的a标签，
                file.write(each.a['title'])

if __name__ == '__main__':
    main()