首先呢,得会用Fiddler这个工具,他下载比较容易,网上一找一大堆,下载完之后呢要对他进行一些设置,我抓取的是pc端的微信文章,这里有详细的配置说明(https://blog.csdn.net/Tester_xjp/article/details/80087014),配置完成之后,可以打开浏览器随便查询一下,看有没有流量包,如果有,则说明配置成功。下面就让我们进行微信公众号的爬虫吧,在流量包很多的情况下如图所示:
点击图中所示图标,下拉菜单中点击remove all 即可清除所有的包,然后进入你要爬取的公众号,在历史信息中下拉让他刷新,就会有流量包生成,如图:
点击链接,然后按图中顺序依次点击,就会看到你想要的json数据,上面部分是url和请求头,复制粘贴即可,粘贴完之后就开始写代码啦,
import requests
import time
import json
import pymysql
import random
from lxml import etree
url1 = "https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MjM5MjAxNDM4MA==&f=json&offset=1364&count=10&is_ok=1&scene=&uin=MjIzMzAyMTc3Mw%3D%3D&key=89d12b870c1b66b55dda3f5d96949191facfdbe5b85fb04febea6507359e2933e7047e8a492e96459539339c329c204b4ebafb430f7f9abd1140e0f41683cad25e1c63b841858a7210dd801df3e696a3&pass_ticket=i8vG65b0f5w3YbINsxgKoJKE%2BADk1WM8sxZ1LYi22FC3WC5aSatNLYe6YZzz5RdB&wxtoken=&appmsg_token=997_%252FomavAR9WcqYeWKQ_IZYJxtOMPFKYXGaIRpjnQ~~&x5=0&f=json HTTP/1.1"
def weixin_spider(url1,author):
headers = {
# 'Host':'mp.weixin.qq.com',
# 'Connection':'keep-alive',
# 'Accept': '*/*',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 MicroMessenger/6.5.2.501 NetType/WIFI WindowsWechat QBCore/3.43.901.400 QQBrowser/9.0.2524.400',
# 'X-Requested-With':'XMLHttpRequest',
# 'Referer: https':'//mp.weixin.qq.com/mp/profile_ext?action=home&__biz=MjM5MjAxNDM4MA==&uin=MjIzMzAyMTc3Mw%3D%3D&key=904312b286f32b60a8dbd9f5fe33159b791adcb96ba37270028681196ab81e4b243785c64ccfd243be4e72664b413c323ade80dcffa498ed2758ad33bc9a85d30932503