bilibili排行榜爬虫

最新推荐文章于 2024-06-12 23:26:22 发布
最新推荐文章于 2024-06-12 23:26:22 发布
阅读量1.8k 收藏 10
点赞数 1
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_40962064/article/details/90183027
版权

建立数据库，将有关数据库的操作放在一个python文件中

def connect_mysql(self):
        self.connection = pymysql.connect(host='localhost',
                                          user='root',#这里输入mysql用户名
                                          password='0711',#这里输入mysql密码
                                          db='test',#这里输入mysql库名
                                          charset='utf8mb4',
                                          cursorclass=pymysql.cursors.DictCursor)

    def select_mysql(self):
        self.connect_mysql()
        try:
            with self.connection.cursor() as cursor:

                sql = "SELECT `country`, `ip`,`port`,`protocol` FROM `xicidaili`"
                cursor.execute(sql)
                result = cursor.fetchall()
        finally:
            self.connection.close()

        return result

建立python键值对，根据键值对查找对应的网页

self.targetUrlMap= {
           "BiliTrendAll":{
               "index":"/all",
               "content":{
                   "all":"/0/0/3",
                   "cartoon":"/1/0/3",
                   "orgin":"/168/0/3",
                   "music":"/3/0/3",
                   "dance":"/129/0/3",
                   "game":"/4/0/3",
                   "technique":"/36/0/3",
                   "machine":"/188/0/3",
                   "life":"/160/0/3",
                   "guichu":"/119/0/3",
                   "fashion":"/155/0/3",
                   "yuLe":"/5/0/3",
                   "movie":"/181/0/3"
               }

           }

对爬下来的网页用XPath进行路径分析，提取我们需要的元素

       	def __getRankItemInformation(self):
        try:
            xPathUrl = '//div[@class = "rank-list-wrap"]/ul[@class = "rank-list"]/li[@class = "rank-item"]'
            # 视频排名
            self.rank = self.html.xpath(xPathUrl+'/div[@class = "num"]/text()')

            # 视频名称
            self.Title = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "info"]/a[@class = "title"]/text()')

            # 视频封面
            self.Rankimg = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "img"]/a/@href')

            # 视频链接
            self.href = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "info"]/a[@class = "title"]/@href')

            # 播放次数
            self.playCount = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/span[1]/text()')

            # 观看次数
            self.view = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/span[2]/text()')

            # up主的链接
            self.upHref = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/a/@href')

            # up主的名字
            self.upName = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/a/span/text()')

            # 综合得分
            self.pts = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "pts"]/div/text()')

        except Exception as e:
            print(e)
完整代码
spiderT
# * coding:utf-8 *

from urllib import request
from lxml import etree

class Spider():
    def __init__(self):
        self.headers = {
            'Host': 'www.bilibili.com',
            'Referer': 'https://www.bilibili.com/',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                          'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 '
                          'Safari/537.36 Edge/16.16299'
        }

        self.targetUrlMap= {
            "BiliTrendAll":{
                "index":"/all",
                "content":{
                    "all":"/0/0/3",
                    "cartoon":"/1/0/3",
                    "orgin":"/168/0/3",
                    "music":"/3/0/3",
                    "dance":"/129/0/3",
                    "game":"/4/0/3",
                    "technique":"/36/0/3",
                    "machine":"/188/0/3",
                    "life":"/160/0/3",
                    "guichu":"/119/0/3",
                    "fashion":"/155/0/3",
                    "yuLe":"/5/0/3",
                    "movie":"/181/0/3"
                }

            },
            "BiliTrendOrigin":{
                "index":"/origin",
                "content":{
                        "all": "/0/0/3",
                        "cartoon": "/1/0/3",
                        "orgin": "/168/0/3",
                        "music": "/3/0/3",
                        "dance": "/129/0/3",
                        "game": "/4/0/3",
                        "technique": "/36/0/3",
                        "machine": "/188/0/3",
                        "life": "/160/0/3",
                        "guichu": "/119/0/3",
                        "fashion": "/155/0/3",
                        "yuLe": "/5/0/3",
                        "movie": "/181/0/3"
                }
            },
            "BiliTrendBangumi":{
                "index":"/bangumi",
                "content":{
                    "Japan":"/13/0/3",
                    "China":"/167/0/3"
                }

            },
            "BiliTrendCinema":{
                "index":"/cinema",
                "content":{
                    "newsreel":"/177/0/3",
                    "movie":"/23/0/3",
                    "teleplay":"/11/0/3"

                }
            },
            "BiliTrendRookie":{
                "index":"/rookie",
                "content":{
                        "all": "/0/0/3",
                        "cartoon": "/1/0/3",
                        "music": "/3/0/3",
                        "dance": "/129/0/3",
                        "game": "/4/0/3",
                        "technique": "/36/0/3",
                        "machine": "/188/0/3",
                        "life": "/160/0/3",
                        "guichu": "/119/0/3",
                        "fashion": "/155/0/3",
                        "yuLe": "/5/0/3",
                        "movie": "/181/0/3"
                }

            },
        }

        self.targetUrlIndex = "https://www.bilibili.com/ranking"
        self.content = None
        # 设置代理默认关闭
        self.setProxyIP()
        self.setHttpProxy(False)
        # save the result in list
        self.resultList = []
        self.__setReusltDic()


    def run(self):
        self.__start()

    # 开始爬虫
    def __start(self):

        for key in self.targetUrlMap:
            dictMap = self.targetUrlMap[key]
            # print(dictMap['content'])
            self.targetUrl = self.targetUrlIndex+dictMap['index']
            self.targetUrlS = self.targetUrlIndex+dictMap['index']

            for t in dictMap['content']:
                self.targetUrl = self.targetUrl + dictMap['content'][t]
                print("start to spider:",self.targetUrl)

                self.setRequest()
                self.response = self.opener.open(self.req)
                status = self.response.getcode()

                # 禁止
                if(status != 200):
                    self.setHttpProxy(True)
                    self.response = self.opener.open(self.req)

                res = self.response.read()
                self.content = res.decode("utf-8")
                # print(self.content)
                self.__getHtml()
                self.__getRankItemInformation()
                self.__createDict()
                self.saveDataToMysql(key+t)
                print("table:"+key+t)
                self.targetUrl = self.targetUrlS


    # 开始请求
    def setRequest(self):
        self.req = request.Request(self.targetUrl, headers=self.headers)

    # 设置头部
    def setHeaders(self,headers):
        self.headers = headers

    def setHttpProxy(self,proxySwitch = False,mysqlSource=True):
        # mysqlsource是源代码的按钮：如果打开，则源代码将来自mysql，或者必须在此函数之前设置代理IP。
        # 无代理
        nullproxy_handler = request.ProxyHandler({})
        print("proxy",proxySwitch)
        if proxySwitch:
            if(mysqlSource):
                import ProxyIP
                item = ProxyIP.ProxyIP()
                item.readProxyIP()
                proxyIPinformation = item.getProxyIP(self.targetUrl, self.headers)
                print(proxyIPinformation)

                if(proxyIPinformation):
                    self.proxyIP = self.setProxyIP(proxyIPinformation['protocol'],proxyIPinformation['ip'],proxyIPinformation['port'])
                else:
                    print("None legally proxy ip or consider to close this function")
                    import sys
                    sys.exit(-1)

            httpproxy_handler = request.ProxyHandler(self.proxyIP)
            self.opener = request.build_opener(httpproxy_handler)

        else:
            self.opener = request.build_opener(nullproxy_handler)
        # req = request.Request("http://www.baidu.com")

        # reponse = self.opener.open(req)
        # print(reponse.read())

    def setProxyIP(self,protocal = "http",ip = "110.52.235.114",port = "9999"):
        self.proxyIP = {
            protocal:ip+":"+port
        }

    def __getHtml(self):
        self.html = etree.HTML(self.content)

    def __setReusltDic(self,title=None,rank=None,rankimg=None,href =None,
                       playCount= None,view = None,upHref = None,upName = None,pts = None):
        self.resultItem  = {
            "rank":rank,
            "title":title,
            "rankimg":rankimg,
            "herf":href,
            "playCount":playCount,
            "view":view,
            "upHref":upHref,
            "upName":upName,
            "pts":pts
        }


    def __getRankItemInformation(self):
        try:
            xPathUrl = '//div[@class = "rank-list-wrap"]/ul[@class = "rank-list"]/li[@class = "rank-item"]'
            # 视频排名
            self.rank = self.html.xpath(xPathUrl+'/div[@class = "num"]/text()')

            # 视频名称
            self.Title = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "info"]/a[@class = "title"]/text()')

            # 视频封面
            self.Rankimg = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "img"]/a/@href')

            # 视频链接
            self.href = self.html.xpath(xPathUrl+'/div[@class = "content"]/div[@class = "info"]/a[@class = "title"]/@href')

            # 播放次数
            self.playCount = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/span[1]/text()')

            # 观看次数
            self.view = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/span[2]/text()')

            # up主的链接
            self.upHref = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/a/@href')

            # up主的名字
            self.upName = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "detail"]/a/span/text()')

            # 综合得分
            self.pts = self.html.xpath(xPathUrl + '/div[@class = "content"]/div[@class = "info"]/div[@class = "pts"]/div/text()')

        except Exception as e:
            print(e)
    def __createDict(self):
        length = len(self.rank)

        for i in range(0,length):
            thisRank = self.rank[i]
            thisTitle = self.Title[i]
            thisImg = self.Rankimg[i]
            thisHref = self.href [i]
            thisPlayCount = self.playCount[i]
            thisView = self.view[i]
            thisUpHref = self.upHref[i]
            thisUpName = self.upName[i]
            thisPts = self.pts[i]
            self.__setReusltDic(thisTitle,thisRank,thisImg,thisHref,thisPlayCount,thisView,thisUpHref,thisUpName,thisPts)
            self.resultList.append(self.resultItem)

        print(len(self.resultList))
        print(self.resultList)
    def saveDataToMysql(self,tableName):
        import mySQLConnect
        this = mySQLConnect.MySqlConnection()
        this.saveInformation(tableName,self.resultList)
        self.resultList = []

if __name__ == '__main__':
    a = Spider()
    # a.setHttpProxy(True)
    a.run()
mysqlConnect
# * coding:utf-8 *

import pymysql
import time, datetime

class MySqlConnection():
    def __init__(self):
        pass
    def connect_mysql(self):
        self.connection = pymysql.connect(host='localhost',
                                          user='root',#这里输入mysql用户名
                                          password='0711',#这里输入mysql密码
                                          db='test',#这里输入mysql库名
                                          charset='utf8mb4',
                                          cursorclass=pymysql.cursors.DictCursor)



    def select_mysql(self):
        self.connect_mysql()
        try:
            with self.connection.cursor() as cursor:

                sql = "SELECT `country`, `ip`,`port`,`protocol` FROM `xicidaili`"
                cursor.execute(sql)
                result = cursor.fetchall()
        finally:
            self.connection.close()

        return result


    def save_myself(self,dataList):
        self.connect_mysql()
        try:
            with self.connection.cursor() as cursor:
                sql = "INSERT INTO `xicidaili` (`country`, `ip`,`port`,`type`,`protocol`,`speed`,`connectTime`,`aliveTime`,`myaliveTime`,`DatabasecreatedTime`,`DatabaseupdateTime`) " \
                      "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s )"
                for data in dataList:
                    # Create a new record
                    thisCountry = data['country']
                    thisip = data['ip']
                    thisport = data['port']
                    thistype = data['type']
                    thisprotocol = data['protocol']
                    thisspeed = data['speed']
                    thisconnectTime = data['connectTime']
                    thisaliveTime = data['aliveTime']
                    myaliveTime = 0
                    DatabasecreatedTime = int(round(time.time() * 1000))
                    cursor.execute(sql, (thisCountry,thisip,thisport,thistype,thisprotocol,thisspeed,thisconnectTime,thisaliveTime,myaliveTime,DatabasecreatedTime,DatabasecreatedTime))
                    self.connection.commit()
        finally:
            self.connection.close()

    def saveInformation(self,tableName,dataList):
        connection = pymysql.connect(host='localhost',
                                  user='root',  # 这里输入mysql用户名
                                  password='0711',  # 这里输入mysql密码
                                  db='test',  # 这里输入mysql库名
                                  charset='utf8mb4',
                                  cursorclass=pymysql.cursors.DictCursor)
        try:
            with connection.cursor() as cursor:
                sql = "INSERT INTO `"+tableName+"` (`rank`, `title`,`herf`,`playCount`,`view`,`upHerf`,`upName`,`pts`,`DatabasecreatedTime`,`DatabaseupdateTime`) " \
                      "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s )"
                for data in dataList:
                    # Create a new record
                    thisRank = data['rank']
                    thisTitle = data['title']
                    thisHerf = data['herf']
                    thisPlayCount = data['playCount']
                    thisview = data['view']
                    thisUpHerf = data['upHref']
                    thisUpName = data['upName']
                    thisPts = data['pts']
                    DatabasecreatedTime = int(round(time.time() * 1000))
                    cursor.execute(sql, (thisRank,thisTitle,thisHerf,thisPlayCount,thisview,thisUpHerf,thisUpName,thisPts,DatabasecreatedTime,DatabasecreatedTime))
                    connection.commit()
        finally:
            connection.close()


if __name__ == '__main__':
    a = MySqlConnection()
    s = a.select_mysql()
    print(s)
进击的荷兰豆
关注
  • 1
    点赞
  • 踩
  • 10
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
WebMagic简单爬取Bilibili网页数据
OrangeHap
01-15 748
http://webmagic.io/ http://webmagic.io/docs/zh/ 依赖： <dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-core</artifactId> <version>0.7...
bilibili用户python爬虫程序
11-20
这是一个用于在Bilibili平台上进行用户信息抓取的Python爬虫程序。通过该程序，用户可以自动获取Bilibili上的用户信息，包括用户ID、昵称、关注数、粉丝数、投稿视频等数据，实现批量采集和分析Bilibili用户信息的...
Bilibili排行榜爬虫
qq_52895917的博客
08-05 192
# -*- coding: utf-8 -*- # @Time : 2021/8/4 15:42 # @Author : sanchez # @File : bilibili排行榜 # @Software : Pycharm Community Edition import requests from bs4 import BeautifulSoup import xlwt start_url = 'https://www.bilibili.com/v/popular/rank/' data_dict
【2024年最新】Bilibili/B站视频/动态评论爬虫
最新发布
weixin_51869009的博客
06-12 2260
bilibili爬虫
排名前50个开源的Web爬虫
热门推荐
iw1210的专栏
10-03 1万+
排名前50个开源的Web爬虫， 转自：http://www.open-open.com/lib/view/open1422112155796.html 项目名     开发语言 平台 Heritrix Java Linux Nutch Java Cross-platform Scrapy Python Cross-platfo
福布斯排行榜爬虫
weixin_45198279的博客
07-05 910
#encoding = utf-8 import requests import re #传入要爬的网络地址url def page(url): respons = requests.get(url) return respons.text #根据网页编写正则表达式 def parse(html): patter = re.compile('<td.*?<td...
【Python简单爬虫练习--Bilibili榜单】爬取+梳理B站排行榜页面
weixin_45575017的博客
03-15 2056
如题，B站老铁爬来玩玩～～～ 思路： Reuqests.get取得网页元素 BeautifulSoup清洗，梳理出框架 输出存为csv文件 各模块的代码： 1： from requests import * from bs4 import * from re import * def geturl(url): try: pg = get(url) ...
Bilibili排行榜-爬虫和数据分析实战.zip
03-01
爬虫（Web Crawler）是一种自动化程序，用于从互联网上收集信息。其主要功能是访问网页、提取数据并存储，以便后续分析或展示。爬虫通常由搜索引擎、数据挖掘工具、监测系统等应用于网络数据抓取的场景。 爬虫的...
Bilibili 用户爬虫.zip
11-09
《Bilibili用户爬虫详解》 Bilibili，国内知名的弹幕视频分享网站，以其丰富的二次元内容和独特的互动体验吸引了大量用户。在数据挖掘和分析领域，Bilibili用户数据具有很高的研究价值，例如了解用户行为模式、热门...
Bilibili 视频爬虫.zip
11-09
标题 "Bilibili 视频爬虫.zip" 暗示了这个压缩包包含了一个用于爬取Bilibili视频信息的程序或者代码库。Bilibili是中国一个非常流行的弹幕视频分享网站，拥有大量的用户生成内容，包括动画、游戏、音乐、科技等多个...
bilibili排行榜数据可视化项目.zip
04-26
1. **Python爬虫技术**：文件名`bilibili-rank-spider-master`暗示了项目中可能包含一个Bilibili排行榜数据的爬虫。Python的爬虫框架如Scrapy或BeautifulSoup可能被用于抓取B站的排行榜信息，如视频标题、作者、播放...
python 爬取编码格式为gbk的网页时，保存为html出现乱码问题
qq_30116343的博客
01-17 971
1 1、问题：今天在爬取百度文科时，百度文库的编码模式为gbk,因此在保存爬取数据时，函数代码如下： def get_response(self): response = requests.get(url=self.url, headers=self.headers) data = response.content.decode('GBK') #print(data)...
bilibili爬虫+数据分析
qq_44937183的博客
07-05 9006
Python爬虫+数据分析+数据可视化实战 Python爬虫+数据分析+数据可视化实战 1. 背景介绍 2. 需求目标 3. 基于urllib的bangumi和bilibili一键爬虫脚本的编写 3.1 bangumi网站分析及爬虫脚本的编写 3.1.1 网站分析 3.1.2 代码实现 3.2 bilibili网站分析及爬虫脚本的编写 3.2.1 网站分析 3.2.2 代码实现 4. 基于pandas的综合数据分析和基于matplotlib的数据可视化 4.1 导入依赖库 4.2 数据清洗
B站热门top100排行榜爬虫与可视化、词云分析
weixin_49756675的博客
06-16 847
爬虫获取bilibili排行榜top100数据分析与词云可视化
Python爬虫学习3----xpath爬取哔哩哔哩排行榜
qq_42467713的博客
08-11 2850
爬取哔哩哔哩月排行榜，并输出csv格式文件。 import requests import lxml.html import csv source = requests.get('https://www.bilibili.com/ranking/all/0/0/30').content.decode('utf-8') selector = lxml.html.fromstring(source) items = selector.xpath('//ul[@class="rank-list"]/li')
简单爬虫入门爬取哔哩哔哩排行榜
weixin_45393946的博客
04-29 1043
需要用到： requests库 re库 主要步骤： 获取排行榜页面 → 分析html文件写出所需内容的正则表达式 → 匹配出所需内容 → 保存 代码如下（导出的是text文档所以有些空格是为了稍微好看一些） import json import requests from requests import RequestException import re #获取网页 def get_...
哔哩哔哩 番列表爬取
weixin_30262255的博客
08-06 197
import re import requests import json #from urllib import parse #from lxml import etree #UA信息： headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTM...
爬虫之对B站排行榜前100作品的名称及地址获取
Jason Zeng
11-22 1379
怎么将小破站的每日实时排行榜作品及地址爬取下来？ 打开B站，点击B站作品排行榜进入排行榜 鼠标右击检查源码，找到作品对应的源码位置，此时我们知道作品在代码的相应大致位置。 接着在Pycharm里导入requests、BeautifulSoup import requests from bs4 import BeautifulSoup 因为作品对应的大致位置在div块（class= ‘info’），因此用find_all查找所有作品的此位置。 url = 'https://www.bilibili.c
ajax headers 参数有什么用_动态爬虫(ajax)-爬取bilibili热门视频信息
weixin_34378888的博客
01-16 239
前言使用python爬虫爬取bilibli每日热门视频的数据使用的第三方软件包括requests、my_fake_useragent一、页面分析在进行爬虫之前，我们先要对要爬取的页面进行分析，找到想要使用的接口bilibili热门排行的地址：https://www.bilibili.com/v/popular/all?spm_id_from=333.851.b_7072696d6172795061...
爬取bilibili排行榜
07-28
爬取bilibili排行榜的过程可以分为三个步骤。首先，需要获取网页的HTML代码。在这个例子中，使用的是Pycharm编译器对哔哩哔哩网站的综合热榜的排行进行爬取。\[1\]其次，需要对获取到的HTML代码进行分析，使用正则...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值