按照崔大佬的书目录,最终是做成一个分布式的爬虫,用框架爬取所有的微博.So,我就按着步骤来,从代理池,cookies池,到最后的crapy框架.
首先,分析一下微博四宫格验证码,它长下面这个样子哈.那么一共有4*6=24中验证码.一种方法是从图像处理的方式来做,但是有个问题.
上面这个图里面,我姑且称为4->3->2->1形验证码.那么1->2->3->4形验证码是不是跟这个验证码非常的想象呢?是的.他们只有中间的三个箭头方向相反,其他一模一样.
所以如果采用图像处理算法来做,必须非常的精确才能做到啊.那么我们就采用第二种方法,对比法.只有24种验证码,那么把所有的验证码都保存在文件夹里面.然后登陆的时候把验证码按固定的位置截屏和本地保存的验证码逐一对比,设置一个阈值(0.98),这样,就可以精确的找出当前的验证码.再把本地24种验证码的图片名字都存储1432.png这种类型,对比成果后提取名字前的数字,然后做成list就可以用selenium的ActionChains模块拖动.模拟人拖动验证码登录了.
需要的库
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from selenium.webdriver import ActionChainsfrom PIL import Image
import time
from io import BytesIO
from os.path import abspath, dirname
from os import listdir
这里我用了无头的Chrome(),平时调试用的是有头的.哈哈哈.最后试用了一下无头的,也能成功.
账号密码
USERNAME = 'username'
PASSWORD = 'password'
TEMPLATES_FOLDER = dirname(abspath('__file__')) + '/templates/'username你得换成你自己的账号,password同理,你的密码.TEMPLATES_FOLEDER是你保存24种验证码的图片的文件.
创造类,并初始化
class WeiboCookies(object):
def __init__(self, username=USERNAME, password=PASSWORD):
self.url = 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=https%3A%2F%2Fm.weibo.cn%2F&sudaref=m.weibo.cn'
self.username = USERNAME
self.password = PASSWORD
self.browser = self.init_browser()
#self.browser = webdriver.Chrome()
self.wait = WebDriverWait(self.browser, 20)
有头Chrome就是注释掉那一行,如果要用有头的把注释取消,然后把上面那一行注释.(下面的代码是无头的初始化,如果要有头那前面操作然后下面的代码可以不用写):
def init_browser(self):
options = webdriver.ChromeOptions()
options.add_argument('--headless')
self.browser = webdriver.Chrome(options = options)
return self.browser
打开登录界面,并输入账号密码点击登录
def open(self):
self.browser.get(self.url)
username = self.wait.until(EC.presence_of_element_located((By.ID, 'loginName')))
password = self.wait.until(EC.presence_of_element_located((By.ID, 'loginPassword')))
submit = self.wait.until(EC.element_to_be_clickable((By.ID, 'loginAction')))
username.send_keys(self.username)
password.send_keys(self.password)
time.sleep(1)
submit.click()
密码错误或者直接登录
def password_error(self):
try:
return WebDriverWait(self.browser, 5).until(
EC.text_to_be_present_in_element((By.ID, 'errorMsy'),'用户名或密码错误'))
except TimeoutException:
return False
def login_successfully(self):
try:
return bool(WebDriverWait(self.browser, 5).until(
EC.presence_of_element_located((By.CLASS_NAME, 'lite-iconf'))))
except TimeoutException:
return False
回到之前,登录之后有三种状态,一是直接登录成功,二是账号密码错误,三是验证码.微博点击登录之后,一般是出现验证码,滑动之后判断有没有账号密码出错.
处理验证码
def get_position(self):
try:
img = self.wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'patt-shadow')))
except TimeoutException as e:
print('验证码未出现')
self.open()
time.sleep(2)
location = img.location
size = img.size
top, bottom, left, right = location['y'], location['y'] + size['height'], location['x'], location['x'] + size['width']
return (top,bottom,left,right)
def get_screenshot(self):
screenshot = self.browser.get_screenshot_as_png()
screenshot = Image.open(BytesIO(screenshot))
return screenshot
def get_image(self, name='captcha.png'):
top, bottom, left, right = self.get_position()
screenshot = self.get_screenshot()
captcha = screenshot.crop((left,top,right,bottom))
captcha.save(name)
return captcha
def is_pixel_equal(self, image1, image2, x, y):
pixel1 = image1.load()[x,y]
pixel2 = image2.load()[x,y]
threshold = 20
if abs(pixel1[0] - pixel2[0]) < threshold and abs(pixel1[1] - pixel2[1]) < threshold and abs(pixel1[2] - pixel2[2]) < threshold:
return True
else:
return False
def same_image(self, image, template):
threshold = 0.98
count = 0
for x in range(image.width):
for y in range(image.height):
if self.is_pixel_equal(image, template, x, y):
count += 1
result = float(count) / (image.width * image.height)
if result > threshold:
print('成功匹配')
return True
return False
def detect_image(self, image):
for template_name in listdir(TEMPLATES_FOLDER):
print('正在匹配', template_name)
if template_name == '.DS_Store':
continue
template = Image.open(TEMPLATES_FOLDER + template_name)
if self.same_image(image, template):
numbers = [int(number) for number in list(template_name.split('.')[0])]
print('拖动顺序', numbers)
return numbers
def move(self, numbers):
try:
circles = self.browser.find_elements_by_css_selector('.patt-wrap .patt-circ')
dx = dy = 0
for index in range(4):
circle = circles[numbers[index] - 1]
if index == 0:
ActionChains(self.browser).move_to_element_with_offset(circle, circle.size['width'] / 2, circle.size['height'] / 2) \
.click_and_hold().perform()
else:
times = 30
for i in range(times):
print(dx,dy)
ActionChains(self.browser).move_by_offset(dx / times, dy / times).perform()
time.sleep(1 / times)
if index == 3:
ActionChains(self.browser).release().perform()
else:
dx = circles[numbers[index + 1] - 1].location['x'] - circle.location['x']
dy = circles[numbers[index + 1] - 1].location['y'] - circle.location['y']
except:
print('滑动失败,重来')
self.main()
上面get_position()是获取验证码位置的,用于截屏之后提取目标图片.get_screenshot()是用来截屏的.get_image()是用来得到四宫格图像的.is_pixel_queal()是用来判定两幅图像里面的像素点一样不,循环两幅图的所以像素点就可以得到有多少像素点相同,用于判定最后图像是不是相同.detect_image()是登录时候得到验证码后将验证码和本地的所以24张验证码做对比,最后得到相同的验证码.move()是得到验证码顺序之后,用selenium滑动验证码.
得到cookies
def get_cookies(self):
return self.browser.get_cookies()
登录成功之后获取cookies
程序主要流程
self.open()
if self.password_error():
return {
'status': 2,
'content': '用户名或密码错误'
}
if self.login_successfully():
cookies = self.get_cookies()
return {
'status': 1,
'content': cookies
}
image = self.get_image('captcha.png')
numbers = self.detect_image(image)
self.move(numbers)
if self.login_successfully():
cookies = self.get_cookies()
return{
'status': 1,
'content': cookies
}
else:
return{
'status': 3,
'content': '登录失败'
}
有三种状态,2是密码错误,1是登录成功,3是登录失败,就是验证码滑动不对.
用例子示范
if __name__ == '__main__':
weibocookies = WeiboCookies()
t = weibocookies.main()
print(t)
总结
mac上如果阈值设定为0.99,将匹配不上,0.98刚刚能匹配出来,太低也不行.
mac上本地存储24个验证码的时候文件夹里面有个.DS_Store的必须要跳过它,不然会失败.Windows上没必要.
7724
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
