本文用于学习交流使用,如有侵权,联系删除
1 爬取需求
1.1 七麦网简介
七麦网(https://www.qimai.cn/),该平台支持提供iOS、Android应用市场、微信、小程序等数据查询,是同时打通App数据、微信公众号数据、小程序数据的数据分析平台。
1.2 爬取需求
根据关键词文件,在检索栏内输入检索关键词。以滴滴出行为例,获得如下界面,在这里点击第一个获得相应的appID(554499054),在此基础上获得更详细的信息。
2 url构造
2.1 analysis加密
相信来看的各位都是因为七麦网的analysis的加密,参考了一系列的文章解决了这个问题,由于侵权问题,就不放破解的js代码了。下面将参考用的博客放给大家
- https://mp.weixin.qq.com/s/1E_ONUnMwTFozd9-dB70Yw
- https://blog.csdn.net/wg5foc08/article/details/102831489
- https://blog.csdn.net/Herishwater/article/details/90733267
由于此文做的是研究某一个APP的ID,之前人做的JS是研究IOS的排行信息,所以在JS传参方向有细节不同。可以通过申请网址来看(https://www.qimai.cn/search/index/country/cn/search/%E6%BB%B4%E6%BB%B4%E5%87%BA%E8%A1%8C),这里传参的param不一样,除此以外,加密方式一样。
2.2 构造链接代码
我的代码实在jupyter notebook里面写的,下面是图片,把代码集中贴在后面。
import requests
import execjs
import urllib
import os
import json
import random
import re
import csv
# 读入整理的app名称
search_list = []
with open("/Users/didi/Downloads/七麦网爬虫/key_word.txt", "r") as f:
for line in f.readlines():
line = line.strip('\n')
search_list.append(line)
# 读入解密js
with open('/Users/didi/Downloads/七麦网爬虫/qimai_app.js', 'r', encoding='utf-8') as f:
get_analy_js = f.read()
jsData = execjs.compile(get_analy_js)
# 准备UA
user_agent = [
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)",
"Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999",
"Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25",
]
# 通过访问七麦得到时间戳
def get_synct():
headers = {
"User-Agent": random.choice(user_agent),
'Origin': 'https://www.qimai.cn',
'Referer': 'https://www.qimai.cn/rank',
"Accept": "application/json,text/plain,*/*",
}
resp = requests.get('https://www.qimai.cn/rank', headers=headers, verify=False)
t = resp.cookies.get_dict()
synct = t.get('synct')
return synct
# 获得加密后的url
def get_analysis_url(synct,app_name):
params = {
'country':'cn',
'search':app_name
}
analysis = jsData.call('getAnalysis',synct,params)
base_url = "https://api.qimai.cn/search/index?analysis="
params_str = "&country=cn&search=" + str(app_name)
url = str(base_url) + str(analysis) + str(params_str)
return url
def get_appID(url):
headers = {
"User-Agent": random.choice(user_agent),
'Origin': 'https://www.qimai.cn'
}
html = requests.get(url, headers=headers)
response = html.text
try:
pattern = r'"appId":"(.*?)","appName"'
appID = re.search(pattern, response, re.I)[1]
except:
appID = None
return appID
def write_csv(file,appInfo):
with open(file, 'a', encoding='utf-8',newline='') as csv_file:
writer = csv.weiter(csv_file)
writer.writerow(appInfo)
def main(app_name):
synct = get_synct()
url = get_analysis_url(synct,app_name)
appID = get_appID(url)
print(appID)
if __name__ == '__main__':
for i in range(len(search_list))
app_name = "滴滴出行"
main(app_name)
3 其他反爬
破解analysis后还有其他的反爬,下面是一张测试结果,这是对于访问频率的封禁,可以看出七麦网的反爬是非常厉害的。