爬虫Ajax手动请求失败,Python爬虫--拦截ajax请求获取参数

前言

在学了py的静态爬虫之后，总是感觉有点缺什么，仔细一想，静态的网页会了，但是有的网站人家用的是ajax搞的，这可怎么办啊！于是乎，就想搞点动态爬虫玩玩。

开搞

1.安装requests和json

2.伪装浏览器

url = 'https://book.qidian.com/ajax/book/category?_csrfToken=BXnzDKmnJamNAgLu4O3GknYVL2YuNX5EE86tTBAm&bookId=3144877'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',

'Referer': 'https://book.qidian.com/info/3144877',

'Cookie': '_csrfToken=BXnzDKmnJamNAgLu4O3GknYVL2YuNX5EE86tTBAm; newstatisticUUID=1564467217_1193332262; qdrs=0%7C3%7C0%7C0%7C1; showSectionCommentGuide=1; qdgd=1; lrbc=1013637116%7C436231358%7C0%2C1003541158%7C309402995%7C0; rcr=1013637116%2C1003541158; bc=1003541158%2C1013637116; e1=%7B%22pid%22%3A%22qd_P_limitfree%22%2C%22eid%22%3A%22qd_E01%22%2C%22l1%22%3A4%7D; e2=%7B%22pid%22%3A%22qd_P_free%22%2C%22eid%22%3A%22qd_A18%22%2C%22l1%22%3A3%7D'

}

3.使用requests发送请求 ，并转码

res = requests.get(url=url, params=headers)

s = str(res.content, "utf-8")

4.使用json进行str到json的转换

load = json.loads(s)

打印结果

5.逐层解析

for (k, v) in load.items(): # 类型为dict

if k == "data":

for (i, j) in v.items(): # 类型为dict

if i == "vs":

for p in j: # 类型为list

for (a, b) in p.items(): # 类型为dict

if a == "vId" or a == "cCnt" or a == "vN": # py自带的判断

print(a, b)

if a == "cs":

for q in b:

print(q)

打印结果

6.这些只是用于解析非vip的书，VIP的咱还不会啊