爬取某乎某个问题的数据，并进行分析（一）

本文选取某乎的一个问题爬取所有回复存入mysql数据库
项目地址：github

首先经过抓包发现，回复都储存在一个get请求当中，如图：
接下来对请求进行分析发现，其参数主要包含两个：
limit是多少条，offset是偏移多少位，这下就好办了。先选取一个问题：[问题]
(https://www.zhihu.com/question/275359100)
然后上代码

import json
import requests 

header = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0",
"Accept-Encoding": "gzip", #指定gzip编码，br编码requests无法识别
"x-requested-with": "fetch"
}
url = "https://www.zhihu.com/api/v4/questions/275359100/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=100&offset=0&platform=desktop&sort_by=default"

session = requests.session()
session.headers = header #添加请求头
repl = session.get(url)
content = json.loads(repl.content) #以json格式接收传回文件
print(content['paging']['next'])
session.close

核心在Accept-Encoding上，直接指定gzip，否则无法解析。之后的无非就是多线程批量爬取了。
关于存入数据库的更多代码在项目地址当中。欢迎点星星，仅供学习使用