官方例程学习分享-apply_bead_inspection_model

一、项目需求

点胶是否合格：[ErrorType]
①少胶，胶路宽度小于标准值：too thin
②多胶，胶路宽度大于标准值: too thick
③点胶位置超出可容忍的区域: incorrect position
④断胶，胶路没有形成闭环: no bead

【图1】黑色的点胶的路径

二、思路步骤

a、方法：采用模板匹配的方式：
①来料定位采用模板匹配，用矫正后的参考图的ROI去作为模板，对来料进行定位
②以标准的胶路作为模板，去匹配来料的胶路，并进行对比
b、步骤：
第一步：手动创建四个标准坐标点，对参考图进行矫正，然后提取ROI创建模板

•     核心算子：vector_to_proj_hom_mat2d(): 两个矩阵之间建立映射关系
               projective_trans_image(): 将映射矩阵投影到图像中，相当于纠正图像
               create_planar_uncalib_deformable_model(): 拿纠正后的图形去创建平面可变形模板A
  

第二步：绘制曲线，即：标准的点胶胶路，并将这条标准的曲线制作成模板

•     核心算子：gen_contour_nurbs_xld(): 生成曲线xld
               create_bead_inspection_model() 针对点胶胶路去创建模板B
  

第三步：读取一张标准有点胶胶路的图像，匹配第一步制作的模板，去纠正来料图像，并生成点胶胶路

•     核心算子：find_planar_uncalib_deformable_model(): 拿模板A去匹配(可设置金字塔)来料图像,得到映射矩阵
                hom_mat2d_invert(): 将映射矩阵翻转
                hom_mat2d_translate(): 将映射矩阵平移
                projective_trans_image(): 将映射矩阵投影到图像中,纠正来料图像
  

第四步：初始化显示窗口设置
第五步：遍历来料图像，并对应显示出来

•      核心算子：apply_bead_inspection_model(): 这个太关键了，输入纠正后的来料图像和模板B，可以直接输出ErrorType!
                intersection_closed_contours_xld(): 求两个闭环xld的交集，将ErrorType框选出来
                area_center_points_xld(): 获取xld中心点坐标和面积
  

创建的外部函数：

• prepare_alignment()
  align_bead()
  gen_display_segments()
  close_parallels()
  

三、官方程序及注释和理解

Main函数（外部函数这里就不展开了）

dev_update_off ()
* 
* Create the planar deformable model to align images with the
* reference contour
* 创建平面可变形模型在参考轮廓上去对齐图像

prepare_alignment (RegionPart, RowT, ColumnT, ModelID)
smallest_rectangle1 (RegionPart, PartRow1, PartColumn1, PartRow2, PartColumn2)
* 
* Define the reference path of the adhesive beads. This path could
* also be generated by drawing it on a reference image with, e.g.,
* draw_xld.
* 规划点胶的参考路径，这个路径是在参考图上绘制xld生成的

* 绘制一条曲线
rows := [701.767,626.953,538.867,443.54,390.447,360.28,354.247,363.9,400.1,458.02,509.907,588.34,659.533,696.94]
cols := [319.24,336.133,367.507,431.46,489.38,546.093,646.247,722.267,776.567,826.04,869.48,912.92,934.64,929.813]
weights := [15,15,15,15,15,15,15,15,15,15,15,15,15,15]
gen_contour_nurbs_xld (ContourRef, [701.767,626.953,538.867,443.54,390.447,360.28,354.247,363.9,400.1,458.02,509.907,588.34,659.533,696.94], [319.24,336.133,367.507,431.46,489.38,546.093,646.247,722.267,776.567,826.04,869.48,912.92,934.64,929.813], 'auto', [15,15,15,15,15,15,15,15,15,15,15,15,15,15], 3, 1, 5)
* 
* Create a new bead inspection model with the following parameters
* 用以下参数创建一个新的胶路检测模板（使用绘制出来的标准的曲线去做模板）
TargetWidth := 14
WidthTolerance := 7
PositionTolerance := 30
Polarity := 'dark'
create_bead_inspection_model (ContourRef, TargetWidth, WidthTolerance, PositionTolerance, Polarity, [], [], BeadInspectionModel)
* 
* Show a correct adhesive bead together with the reference contour
read_image (Image, 'bead/adhesive_bead_01')
align_bead (Image, ImageAligned, ModelID, RowT, ColumnT)
* 
* Create two parallel contours to give an impression of how
* thick a correct adhesive bead should be
* 创建两道平行轮廓去呈现标准正确的点胶胶路的厚度
gen_parallel_contour_xld (ContourRef, ModelSide1, 'regression_normal', TargetWidth * 0.5)
gen_parallel_contour_xld (ContourRef, ModelSide2, 'regression_normal', -TargetWidth * 0.5)
concat_obj (ModelSide1, ModelSide2, ModelSides)
* 
* Create two parallel contours to give an impression of where
* a correct adhesive bead should be located
* 创建两道平行轮廓去呈现定位哪里才是正确的点胶胶路
gen_parallel_contour_xld (ContourRef, PositionToleranceSide1, 'regression_normal', PositionTolerance)
gen_parallel_contour_xld (ContourRef, PositionToleranceSide2, 'regression_normal', -PositionTolerance)
concat_obj (PositionToleranceSide1, PositionToleranceSide2, PositionToleranceSides)
dev_close_window ()
dev_open_window_fit_size (0, 0, PartColumn2 - PartColumn1 + 1, PartRow2 - PartRow1 + 41, -1, -1, WindowHandle)
set_display_font (WindowHandle, 16, 'mono', 'true', 'false')
dev_set_part (PartRow1 - 20, PartColumn1, PartRow2 + 20, PartColumn2)
dev_display (ImageAligned)
dev_set_line_width (2)
dev_set_color ('green')
dev_display (ContourRef)
dev_set_line_width (1)
dev_display (ModelSides)
dev_set_color ('yellow')
dev_display (PositionToleranceSides)
Message := 'Correct adhesive bead and the reference contour. The'
Message[1] := 'yellow contours indicate the range of position tolerance.'
dev_disp_text (Message, 'window', 12, 12, 'black', [], [])
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
stop ()
* 
* 
* Now, perform the inspection task
TextOffset := 20
NumImages := 7
for Index := 1 to NumImages by 1
    read_image (Image, 'bead/adhesive_bead_' + Index$'02')
    * 
    * Align the input image with the reference image
    align_bead (Image, ImageAligned, ModelID, RowT, ColumnT)
    * 
    * Apply the bead inspection model to the aligned image
    * 在对齐后的图像中应用bead检测模型
    apply_bead_inspection_model (ImageAligned, LeftContour, RightContour, ErrorSegment, BeadInspectionModel, ErrorType)
    * 
    * Display the segmented adhesive bead with its error segments
    set_window_param (WindowHandle, 'flush', 'false')
    dev_clear_window ()
    dev_display (ImageAligned)
    dev_set_line_width (1)
    dev_set_color ('white')
    dev_display (ContourRef)
    dev_display (ModelSides)
    dev_display (PositionToleranceSides)
    dev_set_line_width (2)
    dev_set_color ('green')
    dev_display (LeftContour)
    dev_display (RightContour)
    dev_set_color ('red')
    dev_display (ErrorSegment)
    if (|ErrorType| == 0)
        * No errors detected
        Message := 'Adhesive bead is OK'
        dev_disp_text (Message, 'window', 12, 12, 'white', 'box_color', 'forest green')
        dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
        * 刷新界面内容
        flush_buffer (WindowHandle)
        set_window_param (WindowHandle, 'flush', 'true')
        stop ()
    else
        * Display errors by error class
        Message[0] := 'Adhesive bead is not OK:'
        * 
        ErrorClasses := ['no bead','too thin','too thick','incorrect position']
        for ClassIndex := 0 to |ErrorClasses| - 1 by 1
            Class := ErrorClasses[ClassIndex]
            ErrorIndices := find(ErrorType,Class)
            if (ErrorIndices != -1)
                select_obj (ErrorSegment, SelectedSegments, ErrorIndices + 1)
                dev_set_color ('red')
                dev_set_line_width (3)
                if (Class != 'no bead')
                    gen_display_segments (SelectedSegments, LeftContour, RightContour, ErrorParts)
                    dev_display (ErrorParts)
                else
                    dev_display (SelectedSegments)
                endif
                * 计算每一个xld的面积和中心坐标，目的: 为了得到每一xld坐标在对应的位置显示信息
                area_center_points_xld (SelectedSegments, Area, Row, Column)
                for E := 0 to |ErrorIndices| - 1 by 1
                    dev_disp_text (ErrorIndices[E] + 1, 'image', Row[E], Column[E] - TextOffset, 'white', 'box_color', 'red')
                    TextOffset := 20 - TextOffset
                endfor
            endif
        endfor
        dev_disp_text (Message, 'window', 12, 12, 'white', 'box_color', 'red')
        dev_disp_text ([1:|ErrorType|] + ': ' + ErrorType, 'image', 500, 500, 'red', 'box', 'false')
        if (Index < NumImages)
            dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
            flush_buffer (WindowHandle)
            set_window_param (WindowHandle, 'flush', 'true')
            stop ()
        endif
    endif
endfor
flush_buffer (WindowHandle)
set_window_param (WindowHandle, 'flush', 'true')

四、理解后编写的

①Main函数

dev_update_off ()
* 第一步：手动创建四个标准坐标点，对参考图进行矫正，然后提取ROI创建模板
* 创建外部函数，得到ROI以及ROI的中心坐标、模板ID
prepare_alignment (RegionPart, RowT, ColumnT, ModelID)
* 获取标准ROI的矩形区域的对角线两个坐标
smallest_rectangle1 (RegionPart, PartRow1, PartColumn1, PartRow2, PartColumn2)


* 第二步：绘制曲线，即：标准的点胶胶路，并将这条标准的曲线制作成模板
* 这里可用ps软件打开一张带有点胶胶路的标准图像，在胶路上面均匀的找点的坐标
rows := [701.767,626.953,538.867,443.54,390.447,360.28,354.247,363.9,400.1,458.02,509.907,588.34,659.533,696.94]
cols := [319.24,336.133,367.507,431.46,489.38,546.093,646.247,722.267,776.567,826.04,869.48,912.92,934.64,929.813]
weights := [15,15,15,15,15,15,15,15,15,15,15,15,15,15]
dev_set_color ('green')
gen_contour_nurbs_xld (ContourRef, rows, cols, 'auto', weights, 3, 1, 5)
* 制作模板
* 官方函数里边就有针对点胶胶路的模板制作：create_bead_inspection_model()
TargetWidth := 14
WidthTolerance := 7
PositionTolerance := 30
create_bead_inspection_model (ContourRef, TargetWidth, WidthTolerance, PositionTolerance, 'dark', [], [], BeadInspectionModel)


* 第三步：读取一张标准有点胶胶路的图像，匹配第一步制作的模板，去纠正来料图像，并生成点胶胶路
read_image (Image, 'bead/adhesive_bead_01.png')
align_bead (Image, ImageAligned, RowT, ColumnT, ModelID)
* 绘制标准路径
dev_set_color ('red')
gen_parallel_contour_xld (ContourRef, ModelSide1, 'regression_normal', TargetWidth*0.5)
gen_parallel_contour_xld (ContourRef, ModelSide2, 'regression_normal', -TargetWidth*0.5)
concat_obj (ModelSide1, ModelSide2, ModelSides)
* 绘制可容忍的路径
dev_set_color ('yellow')
gen_parallel_contour_xld (ContourRef, ToleranceSide1, 'regression_normal', PositionTolerance)
gen_parallel_contour_xld (ContourRef, ToleranceSide2, 'regression_normal', -PositionTolerance)
concat_obj (ToleranceSide1, ToleranceSide2, ToleranceSides)

* 第四步：初始化显示窗口设置
dev_close_window ()
dev_open_window_fit_size (0, 0, PartColumn2-PartColumn1+1, PartRow2-PartRow1+40, -1, -1, WindowHandle)
dev_set_part (PartRow1-20, PartColumn1, PartRow2+20, PartColumn2)
dev_display (ImageAligned)
set_display_font (WindowHandle, 16, 'mono', 'true', 'false')


* 第五步：遍历来料图像，并对应显示出来
Textoffset := 20
NumImages := 7
for Index := 1 to NumImages by 1
    dev_clear_window ()
    read_image (Image1, 'bead/adhesive_bead_' + Index$'02')
    align_bead (Image1, ImageAligned1, RowT, ColumnT, ModelID) 
    * 这个算子太神奇了，只需传入纠正后的图像，以及创建好的点胶胶路模板；输出胶路的两边和错误分割的部分
    apply_bead_inspection_model (ImageAligned1, LeftContour, RightContour, ErrorSegment, BeadInspectionModel, ErrorType)
    * 进行计算和显示
    dev_display (ImageAligned1)
    dev_set_color ('white')
    dev_set_line_width (1)
    dev_display (ToleranceSides)
    dev_set_color ('yellow')
    dev_set_line_width (2)
    dev_display (LeftContour)
    dev_display (RightContour)
    dev_set_color ('red')
    dev_display (ErrorSegment)
    
    if(|ErrorType| == 0)
        * ok料
        message := 'Adhesive bead is ok'
        dev_disp_text (message, 'window', 12, 12, 'white', 'box_color', 'forest green')
        
    else
        message[0] := 'Adhesive bead is not ok'
        ErrorsClasses := ['no bead', 'too thin', 'too thick', 'incorrect position']
        for ClassIndex := 0 to |ErrorsClasses|-1 by 1
            Class := ErrorsClasses[ClassIndex]
            ErrorIndices := find(ErrorType, Class)
            if(ErrorIndices != -1)
                * ng显示处理
                select_obj (ErrorSegment, SelectedSegments, ErrorIndices+1)
                dev_set_color ('red')
                dev_set_line_width (3)
                if(Class != 'no bead')
                    * 处理显示'too thin','too thick','incorrect position'这三种情况
                    gen_display_segments (SelectedSegments, LeftContour, RightContour, ErrorParts)
                    dev_display (ErrorParts)
                else
                    dev_display (SelectedSegments)
                endif
                * 获取每个分割处理的ng的xld坐标信息
                area_center_points_xld (SelectedSegments, Area, Row, Column)
                for e := 0 to |ErrorIndices|-1 by 1
                    dev_disp_text (ErrorIndices[e]+1, 'image', Row[e], Column[e]-Textoffset, 'black', 'box_color', 'red')
                    Textoffset := 20 - Textoffset
                endfor
                
            endif
            
        endfor
        dev_disp_text (message, 'window', 12, 12, 'white', 'box_color', 'red')
        * 采用元组一次性显示出来，就不用单独去显示
        dev_disp_text ([1:|ErrorType|] + ': ' + ErrorType, 'image', 500, 500, 'red', 'box', 'false')
             
    endif
stop ()

endfor

②外部函数——prepare_alignment()

* 使用PS软件查看参考图的ROI四个点的坐标记录如下：
* 或者把参考图读取进来，然后使用draw_point()去绘制选择那四个点，也能得到四个点的坐标
Row := [658.232,291.923,314.817,691.533]
Col := [330.071,316.617,971.032,947.008]

* 通过计算，把四个点坐标转换到标准的正正方方的矩形四个点上面
w := 5.4
h := 3.4
Row1 := 300 + [h, 0, 0, h] * 120.0
Col1 := 300 + [0, 0, w, w] * 120.0

* 制作映射变换矩阵
vector_to_proj_hom_mat2d (Row, Col, Row1, Col1, 'normalized_dlt', [], [], [], [], [], [], HomMat2D, Covariance)

read_image (Image, 'bead/adhesive_bead_ref.png')
* 生成四个坐标点，验证坐标点正不正确
gen_cross_contour_xld (Cross, Row, Col, 36, 0.785398)

* 把矩阵投影到图像当中
projective_trans_image (Image, TransImage, HomMat2D, 'bilinear', 'false', 'false')
gen_cross_contour_xld (Cross1, Row1, Col1, 36, 0.785398)

* 提取ROI
binary_threshold (TransImage, Region, 'smooth_histo', 'light', UsedThreshold)
opening_circle (Region, RegionOpening, 5.5)
connection (RegionOpening, ConnectedRegions)
area_center (ConnectedRegions, Area, Row2, Column2)
select_obj (ConnectedRegions, ObjectSelected, sort_index(-Area)[0]+1)
* 使用select_shape()也是可以提取的
*select_shape (ConnectedRegions, SelectedRegions, 'area', 'and', 77115.4, 200000)
* 还是需要谨慎，对边缘进行处理，不能直接立马就裁剪出roi，不然得到的面积和中心坐标不准
*reduce_domain (TransImage, ObjectSelected, ImageReduced)
*area_center (ObjectSelected, Area1, Row3, Column3)
*gen_cross_contour_xld (Cross2, Row3, Column3, 26, 0.785398)
fill_up (ObjectSelected, RegionFillUp)
dev_set_draw ('margin')
dilation_circle (RegionFillUp, RegionPart, 3.5)
reduce_domain (TransImage, RegionPart, ImageReduced1)
area_center (RegionPart, Area2, RowT, ColumnT)
gen_cross_contour_xld (Cross3, RowT, ColumnT, 26, 0.785398)

* 把roi创建无标定可变行匹配模板
create_planar_uncalib_deformable_model (ImageReduced1, 'auto', [], [], 'auto', 1, [], 'auto', 1, [], 'auto', 'none', 'use_polarity', 'auto', 'auto', [], [], ModelID)

return ()

③外部函数——align_bead()

* 之前是用create_plannr_uncalib_deformable_model()创建的模板
* 这次就用find_plannr_uncalib_deformable_model()去匹配模板
* 参数NumLevels：金字塔层数：0-5调整过程中能够矫正图像的姿态，>5之后姿态变回去了，修改这个数，得到最佳姿态
find_planar_uncalib_deformable_model (Image, ModelID , -0.39, 0.78, 1, 1, 1, 1, 0.4, 1, 1, 5, 0.9, [], [], HomMat2D, Score)
hom_mat2d_invert (HomMat2D, HomMat2DInvert)
hom_mat2d_translate (HomMat2DInvert, RowT, ColumnT, HomMat2DTranslate)
* 这里的投影变换不能直接使用HomMat2D，需要使用invert翻转一下，然后使用translate平移
projective_trans_image (Image, ImageAligned, HomMat2DTranslate, 'bilinear', 'false', 'false')

return ()

④外部函数——gen_display_segments()

* 构造轮廓，用于可视化错误分割出来的部分
gen_parallel_contour_xld (ToothinSegments, ParallelContours1, 'regression_normal', 50)
gen_parallel_contour_xld (ToothinSegments, ParallelContours2, 'regression_normal', -50)
*dev_set_color ('green')
*dev_display (ParallelContours1)
*dev_display (ParallelContours2)
* 这个算法主要是把ng的胶路部分框出来,做法：把ng部分尽量框大，然后减去标准胶路的框，
close_parallels (ParallelContours1, ParallelContours2, Contours)
close_parallels (LeftContour, RightContour, Contours1)
* 求两个闭合xld区域的交集部分，并用闭合形式显示
intersection_closed_contours_xld (Contours, Contours1, ContourIntersection)

return ()

⑤外部函数——close_parallels()

count_obj (ParallelContours1, Number)
gen_empty_obj (Contours)
for Index := 1 to Number by 1
    select_obj (ParallelContours1, Parallel1, Index)
    select_obj (ParallelContours2, Parallel2, Index)
    get_contour_xld (Parallel1, Row1, Col1)
    get_contour_xld (Parallel2, Row2, Col2)
    * 生成多变形
    gen_contour_polygon_xld (CloseContour, [Row1, inverse(Row2), Row1[0]], [Col1, inverse(Col2), Col1[0]])
    concat_obj (Contours, CloseContour, Contours)
 
endfor

return ()

【图2】结果图展示
（点胶在白色线以内都是合格的）分析有不到位的，请各位点评指正~