python爬虫过程出现‘POST body missing, invalid Content-Type, or JSON object has no keys.‘怎么办?(求助)

时间:

2022/11/20

背景:

我准备爬取下方页面,是英雄联盟比赛的数据(课程作业)。想爬取网页中的相应数据,里面有选手的名字和一些比赛的数据。本人还是网络爬虫新手,好多别人的代码看不懂QWQ。2022 Worlds Schedule, Standings and Match Results | QWER.GGicon-default.png?t=M85Bhttps://qwer.gg/leagues/Worlds/2022?tournament=%22969%22

问题:

在爬取的过程中,我尝试获取网页的源码,但是出现了一点小问题,所以我转而去爬取data数据,如下图所示,但是爬取之后出现问题,无法获得数据,想一下怎么解决。

代码:

import requests
from lxml import etree

if __name__ == '__main__':
    # UA伪装&参数
    headers = {
        'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.42 ",
    }
    param = {
        "operationName": "ListPlayerStatisticsByTournament",
        "variables": {"tournamentId": "969"},
        "query": "fragment CorePlayer on Player {\n  id\n  nickName\n  firstName\n  lastName\n  imageUrl\n  birthday\n  nationality\n  position\n  __typename\n}\n\nfragment CoreTeam on Team {\n  id\n  name\n  acronym\n  imageUrl\n  nationality\n  __typename\n}\n\nfragment CorePlayerStatistic on PlayerStatistic {\n  player {\n    ...CorePlayer\n    __typename\n  }\n  team {\n    ...CoreTeam\n    __typename\n  }\n  playerId\n  teamId\n  tournamentId\n  position\n  games\n  wins\n  loses\n  winRate\n  kda\n  kills\n  deaths\n  assists\n  wardsPlaced\n  wardsKilled\n  dpm\n  dtpm\n  gpm\n  cspm\n  dpgr\n  firstBlood\n  firstTower\n  __typename\n}\n\nquery ListPlayerStatisticsByTournament($tournamentId: ID!, $playerId: ID, $teamId: ID) {\n  playerStatisticsByTournament(\n    tournamentId: $tournamentId\n    playerId: $playerId\n    teamId: $teamId\n  ) {\n    ...CorePlayerStatistic\n    __typename\n  }\n}"
    }
    url_html = "https://qwer.gg/leagues/Worlds/2022?tournament=%22969%22"
    url_date = "https://qwer.gg/general/graphql"

    # 发起请求 & 获取页面数据
    rp = requests.get(url=url_date, headers=headers, params=param)
    text = rp.text
    date = rp.json()

 对爬虫的一些疑问:

1、param怎么确定;

2、headers除了'User-Agent'还需要哪些头,需要Cookie吗;

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值