简单数字验证码的机器识别探索

问题

在某网站登录后，查询过程中需要输入数字加减操作的图形验证码

解决方法

利用opencv提供的图片处理能力，进行切割、降噪、门限等操作，抓取特性，再送入opencv自带的knn和svm线性分类器，最终实现数字验证码结果的机器识别

过程

最开始觉得图片数字验证码挺简单的，先采用opencv进行预处理，包括图片分割、平滑、门限等聚焦和降噪处理，寄希望通过cv.matchTemplate模板匹配，就可以解决问题。

后来发现，问题没有那么简单，即使使用多个数字或操作符的模板图片，做集成裁决，以及扩大输入图片的裁剪范围，避免模板图片卷积范围过小，识别率也不能达到可以接收的程度。

在一篇博文的指引下，为何不尝试下，opencv自带的机器学习模块呢？

通过，初步试验操作符的识别，发现通过训练后，识别率竟然出奇的高！似乎可以进行深入探索。

最终在训练完所有左侧操作数、中间操作符、右侧操作数后，对图片进行识别，虽然在模板匹配的阶段，对于数字图形验证码中3和5，对于人工都很难区别，但最终的准确率，也是令人印象深刻！

对现代人工智能的初次感悟

发现现在以大数据为基础的人工智能，与通过模板图片的严格比对，存在思路和出发点上的相当大差异，模板图片再多，也没有达到大数据的程度，而且适应变化能力弱…

现代人工智能通过学习大数据中，总结出来的规律，或超平面种类划分，可将随机样本的随机性限定在某个范围和集合中。

如果新出现的样本，也在规律中的话，那么就自会落入其中，可以实现自动识别。

进一步探索

即使初步战胜了数字验证码，继续挑战登录页面的风景图片加印汉字，用户通过点击有顺序的汉字完成登录验证，又进入了一个新的识别阶段。

初步探索，难度比前者又大了不少，大伙有好的方法可以交流：）

注意，应避免调用其它平台的远程API

---

附录

OpenCV预处理

class ImageOpers():
    def __init__(self, img):
        self.img = img

    def zoom(self, factors=(7,7)):
        self.img =  cv.resize(self.img, None, fx=factors[0], fy=factors[1], interpolation = cv.INTER_CUBIC )
        return self

    def threshold(self,):
        ret, self.img =  cv.threshold(self.img, 0, 255, cv.THRESH_BINARY+cv.THRESH_OTSU)
        return self
    
    # 对于噪点比较不错
    def blur(self, filter=(3, 3)):
        self.img  =  cv.medianBlur(self.img, filter[0])
        return self
    
    # 是否可以通过描边来凸显主要特征
    def edges(self, para=(100, 200)):
         self.img = cv.Canny(self.img, *para)
         return self

    def gray(self):
        self.img = cv.cvtColor(self.img, cv.COLOR_BGR2GRAY)
        return self

    # 拉普拉斯锐化
    def laplacian(self):
        self.img = cv.Laplacian(self.img, cv.CV_64F)
        return self

    def get(self):
        return self.img
        
def splitOrigImg(img):

    origi_lhs_split_end     = 12

    origi_rhs_split_begin   = 25
    origi_rhs_split_end     = 40

    origi_oper_split_begin  = 12
    origi_oper_split_end    = 26

    lhs  = img[:, :origi_lhs_split_end]
    rhs  = img[:,  origi_rhs_split_begin:origi_rhs_split_end]
    oper = img[:,  origi_oper_split_begin:origi_oper_split_end]

    return (lhs, rhs , oper)


def processOrigiImg(img):
       
    opers       = ImageOpers(img)
    img         = opers.gray().get()
    img         = opers.blur().get()
    img         = opers.threshold().get()
  
    return img

OpenCV人工智能

# svm
def svmAI(trainData, train_labels, kernel=cv.ml.SVM_LINEAR, savedFile='svm_data.dat'):
    svm = cv.ml.SVM_create()
    svm.setKernel(kernel)
    svm.setType(cv.ml.SVM_C_SVC)
    svm.setC(2.67)

    if kernel != cv.ml.SVM_LINEAR:
        svm.setGamma(5.383)

    svm.train(trainData, cv.ml.ROW_SAMPLE, train_labels)
    svm.save(savedFile)
    result = svm.predict(trainData)[1]

    # stat
    mask = result==train_labels 
    correct = np.count_nonzero(mask)
    print("stat:", correct*100.0/result.size)

    svm2   = cv.ml.SVM_load(savedFile)

    result = svm2.predict(trainData)[1]

    # stat
    mask = result==train_labels 
    correct = np.count_nonzero(mask)
    print("stats when reload:", correct*100.0/result.size)

# knn
def knnAI(trainData, train_labels, savedFile='knn_data.npz', k = 5):
    knn = cv.ml.KNearest_create()
    knn.train(trainData, cv.ml.ROW_SAMPLE, train_labels)

    ret,result,neighbours,dist = knn.findNearest(trainData, k)

    # stat
    mask = result==train_labels 
    correct = np.count_nonzero(mask)
    print("stat:", correct*100.0/result.size)

    np.savez(savedFile, train=trainData, train_labels=train_labels)

    with np.load(savedFile) as data:
        trainData_load        = data['train']
        train_labels_load     = data['train_labels']

    knn2 = cv.ml.KNearest_create()
    knn2.train(trainData_load, cv.ml.ROW_SAMPLE, train_labels_load)

    ret,result,neighbours,dist = knn2.findNearest(trainData, k)
    print(ret, result.shape, neighbours.shape, dist.shape)

    # stat
    mask = result==train_labels 
    correct = np.count_nonzero(mask)
    print("stat when reload:", correct*100.0/result.size)

# 训练
def digitTrain(section):
    
    home         = os.getcwd()
    digitHome    = os.path.join(home, 'data', section)
    trainData    = None
    train_labels = None


    for digit in range(0, 10):
        images = []

        walkFiles(os.path.join(digitHome, '%d'%digit), lambda h,f: collectImg(h, f, images))  

        labels    = np.repeat(digit, len(images))[:, np.newaxis]
        nd_imges  = np.asarray(images)

        if trainData is not None: 
            trainData     = np.concatenate((trainData, nd_imges))
            train_labels  = np.concatenate((train_labels, labels))
        else:
            trainData     = nd_imges
            train_labels  = labels

    if trainData is not None and trainData.size > 0 : 
        trainData       = np.float32(trainData).reshape(trainData.shape[0], trainData.shape[1]*trainData.shape[2])

        print('trainData:', trainData.shape, train_labels.shape)
        svmAI(trainData, train_labels, os.path.join(home, 'ai', '%s.dat'%section))


def lhsTrain():
   digitTrain('lhs')

def rhsTrain():
   digitTrain('rhs')

# 预测左侧操作数、右侧操作数、操作符共同代码
# img事先要经过分割和处理，与训练样本相同
def svmPredictResult(svm, img):
    nd_oper      = np.asarray(img)
    nd_oper      = np.float32(nd_oper).reshape(-1, nd_oper.shape[0]*nd_oper.shape[1])
    result       = svm.predict(nd_oper)[1]
    result       = np.int16(result)
    return result[0][0]


def svmExternalTest(svm, img, relustOper):
    relustOper(svmPredictResult(svm, img))
        
   
def operatorResultProc(result, home, file, toDir):
    if result == 1:
        shutil.copyfile(os.path.join(home,file), os.path.join(toDir, "+", file))
    else:
        shutil.copyfile(os.path.join(home,file), os.path.join(toDir, "-", file))

def digitResultProc(result, home, file, toDir):
  
    shutil.copyfile(os.path.join(home,file), os.path.join(toDir, "%d"% result, file))
  
        

def pngPredict():
    home         = os.getcwd()
    pngHome      = os.path.join(home, 'test', 'png')
    resultHome   = os.path.join(home, 'result', 'png')
    resultLhsHome   = os.path.join(home, 'result', 'png','lhs')
    resultRhsHome   = os.path.join(home, 'result', 'png','rhs')

    svmOper      = cv.ml.SVM_load(os.path.join(home, 'ai', 'operator.dat'))
    svmLhs       = cv.ml.SVM_load(os.path.join(home, 'ai', 'lhs.dat'))
    svmRhs       = cv.ml.SVM_load(os.path.join(home, 'ai', 'rhs.dat'))

    plusDir      = os.path.join(resultHome, '+')
    minusDir     = os.path.join(resultHome, '-')

    shutil.rmtree(resultHome)   

    os.makedirs(plusDir, exist_ok=True)
    os.makedirs(minusDir, exist_ok=True)
    os.makedirs(resultLhsHome, exist_ok=True)
    os.makedirs(resultRhsHome, exist_ok=True)

    for digit in range(0,10):
       os.makedirs(os.path.join(resultLhsHome, '%d'%digit), exist_ok=True)
       os.makedirs(os.path.join(resultRhsHome, '%d'%digit), exist_ok=True)          
      
   
    for h,childDirs,files in os.walk(pngHome):
        for file in files:
            img          = cv.imread(os.path.join(h, file))
            img          = processOrigiImg(img)
            lhs,rhs,oper = splitOrigImg(img)

            svmExternalTest(svmOper, oper,  lambda r: operatorResultProc(r, h, file, resultHome))
            svmExternalTest(svmLhs,  lhs,   lambda r: digitResultProc(r, h, file, resultLhsHome))
            svmExternalTest(svmRhs,  rhs,   lambda r: digitResultProc(r, h, file, resultRhsHome))

OpenCV参考

OpenCV-Python Tutorials