如果需要帮忙爬取数据,请加QQ:2422035338
一.前言:
1、本博客主要介绍百度指数爬取,360指数获取类似;
2、想要获取数据必须先登录百度指数,频繁登陆会导致要求输入验证码和手机验证码;
3、百度指数的数值是采用html格式+加密二进制传输, 不能够通过直接获取节点进而获取数值。
二.爬取思路:
1、首先使用selenium+chrome模拟登陆百度账号,获取cookie;
2、由于有时候需要验证码登陆,所以需要保存cookie模拟登陆;
3、模拟登陆,输入关键词进入有指数页面,截整个图保存本地;
4、读取图片,找到搜索指数所在区域,截取图片;
5、使用Tesseract-OCR进行图像识别,若数字识别不准确,需使用jTessBoxEditor训练数据提高准确度。
三.主要代码介绍:
1.登录
url = 'http://index.baidu.com/'
driver = webdriver.Chrome(executable_path='C:/Program Files
(x86)/Google/Chrome/Application/chromedriver.exe')
driver.get(url)
cookieList = []
for cookie in cookieList:
driver.add_cookie(cookie)
driver.get(url)
time.sleep(3)
driver.refresh()
此处cookieList已被我删除,获取方法:第一次模拟登陆时手动输入账号和密码,通过driver.get_cookies()获取,程序如下(该段程序只是获得cookies,获得的cookies添加到cookieList中,以后这段程序就无需放到爬虫程序中了):
url = 'http://index.baidu.com/'
driver = webdriver.Chrome(executable_path='C:/Program Files
(x86)/Google/Chrome/Application/chromedriver.exe')
driver.get(url)
time.sleep(30)
cookies=driver.get_cookies()
print(cookies)
设置中间停顿30秒,输入账号,把打印下来的cookies(字典形式)粘贴到原来代码的cookieList中,这样就可以跳过验证码和输入密码登录
2.输入关键词并最大化界面
WebDriverWait(driver, 10, 0.5).until(
EC.element_to_be_clickable((By.XPATH, "//input[@class='search-input']")))
driver.find_element_by_xpath("//input[@class='search-input']").send_keys(keyword)
WebDriverWait(driver, 10, 0.5).until(
EC.element_to_be_clickable((By.XPATH, "//span[@class='search-input-cancle']")))
driver.find_element_by_xpath("//span[@class='search-input-cancle']").click()
driver.maximize_window()
3.鼠标移动到指数所在矩形框并进行移动使出现viewbox
time.sleep(2)
WebDriverWait(driver, 10, 0.5).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, '#trend > svg > rect')))
element = driver.find_elements_by_css_selector('#trend > svg > rect')[1]
time.sleep(2)
ActionChains(driver).move_to_element_with_offset(element, x, y).perform()
time.sleep(3)
driver.get_screenshot_as_file(str(index)+'.png')
WebDriverWait(driver, 10, 0.5).until(
EC.element_to_be_clickable((By.XPATH, "//div[@id='viewbox']")))
4.获取viewbox位置截图并进行图像识别
element = driver.find_element_by_xpath("//div[@id='viewbox']")
getElementImage(driver,element, str(index)+'.png', 'day'+str(index)+'.png',keyword)
time.sleep(2)
number = Image.open('day'+str(index)+'.png')
number = pytesseract.image_to_string(number,lang='fontyp')
number = re.sub(r',?\.?\s?', '', number)
number=number.replace('z','2').replace('i','7').replace('e','9')
print(number)
def getElementImage(driver,element,fromPath,toPath,keyword):
"""
该元素所对应的截图
:param element: 元素
:param fromPath: 图片源
:param toPath: 截图
"""
# 找到图片坐标
locations = element.location
# 跨浏览器兼容
scroll = driver.execute_script("return window.scrollY;")
top = locations['y'] - scroll
# 找到图片大小
sizes = element.size
# 构造关键词长度
add_length = (len(keyword) - 2) * sizes['width'] / 15
# 构造指数的位置
rangle = (
int(locations['x'] + sizes['width'] / 4 + add_length)-2, int(top +
sizes['height'] / 2),
int(locations['x'] + sizes['width'] * 2 / 3)+2, int(top + sizes['height']))
time.sleep(2)
image = Image.open(fromPath)
cropImg = image.crop(rangle)
cropImg.save(toPath)
四、优化
1.若想获取30天的数据,则鼠标往右移动的宽度为41.68像素较为合适,但这个宽度不是数据所在矩形框的平均值(41.86),前者使用没有问题,后者使用会不出现viewbox,小编也不知道为什么,有知道的朋友麻烦留言告知一下,非常感谢。
2.使用jTessBoxEditor训练数据集提高识别准确率,具体见https://www.cnblogs.com/zhang-ke/p/7606572.html
五、结尾
本博客主要介绍的是爬取30天每天的百度指数,读者可以拓展爬取其他时间段或者地区的指数。360指数爬取类似,不过到小编写这篇博客为止,360指数上有地区选项但仍然无法点开!
百度指数爬取代码在github:https://github.com/kingdomrushing/SpiderbaiduIndex-python
CSDN下载:https://download.csdn.net/download/qq_37913997/10711752
百度指数爬虫最新版(2020年12月1号):
import requests
import sys
import time
word_url = 'http://index.baidu.com/api/SearchApi/thumbnail?area=0&word={}'
COOKIES = ''
headers = {
'Accept': 'application/json, text/plain, */*',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Cache-Control': 'no-cache',
'Cookie': COOKIES,
'DNT': '1',
'Host': 'index.baidu.com',
'Pragma': 'no-cache',
'Proxy-Connection': 'keep-alive',
'Referer': 'http://index.baidu.com/v2/main/index.html',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
}
def decrypt(t,e):
n = list(t)
i = list(e)
a = {}
result = []
ln = int(len(n)/2)
start = n[ln:]
end = n[:ln]
for j,k in zip(start, end):
a.update({k: j})
for j in e:
result.append(a.get(j))
return ''.join(result)
def get_ptbk(uniqid):
url = 'http://index.baidu.com/Interface/ptbk?uniqid={}'
resp = requests.get(url.format(uniqid), headers=headers)
if resp.status_code != 200:
print('获取uniqid失败')
sys.exit(1)
return resp.json().get('data')
def get_index_data(keyword, start='2011-01-03', end='2019-08-05'):
keyword = str(keyword).replace("'", '"')
url = f'http://index.baidu.com/api/SearchApi/index?area=0&word={keyword}&area=0&startDate={start}&endDate={end}'
resp = requests.get(url, headers=headers)
if resp.status_code != 200:
print('获取指数失败')
sys.exit(1)
content = resp.json()
data = content.get('data')
user_indexes = data.get('userIndexes')[0]
uniqid = data.get('uniqid')
ptbk = get_ptbk(uniqid)
while ptbk is None or ptbk == '':
ptbk = get_ptbk(uniqid)
all_data = user_indexes.get('all').get('data')
result = decrypt(ptbk, all_data)
result = result.split(',')
print(result)
if __name__ == '__main__':
words = [[{"name": "酷安", "wordType": 1}]]
get_index_data(words)
该代码由网友提供,详情见https://zhuanlan.zhihu.com/p/78634149
百度指数后台代码经常更新,如需最新代码请加QQ:2422035338