斗鱼是大家常用的直播网站,相信大家也有喜欢的主播,那么我们今天使用之前提到的xpath方式来将斗鱼的各大主播的房间信息拿到
第一步 ,我们先来分析一下斗鱼,直播分类中的房间信息列表页接口
觉得今天的MSI不能错过!!!!!!!!!!
接口
url = 'https://www.douyu.com/directory/all'
我们就准备爬取此界面直播下的房间内容
这里就不做分页的处理了只爬取首页的内容
接下来直接上代码了
import requests
from lxml import etree
url = 'https://www.douyu.com/directory/all'
headers = {
'user-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
}
response = requests.get(url= url ,headers = headers).content.decode('utf-8')
# with open('douyu.html','w',encoding='utf-8') as fp:
# fp.write(response)
# print(response)
# 提取房间名称
tree = etree.HTML(response)
li_list = tree.xpath('//ul[@class="layout-Cover-list"]/li')
print(len(li_list))
for li in li_list:
# 提取房间名称
name_list = li.xpath('.//h3[@class="DyListCover-intro"]/text()')
name = name_list[0]
# print(name)
# 标签
tag_list = li.xpath('.//span[@class="DyListCover-zone"]/text()')
tag = tag_list[0]
# print(tag)
# 主播
zhubo_list = li.xpath('.//h2[@class="DyListCover-user is-template"]/text()')
zhubo = zhubo_list[0]
# print(zhubo)
# 关注度
guanzhu_list = li.xpath('.//span[@class="DyListCover-hot is-template"]/text()')
guanzhu = guanzhu_list[0]
# print(guanzhu)
infor = name+'--'+tag+'--'+zhubo+'--'+guanzhu
print(infor)
希望大家感兴趣