【python】OpenCV—Augmented Reality Using Aruco Markers

1、任务描述

借助 Aruco Markers，替换墙面上画面中的内容

2、Aruco Markers

OpenCV 中的 aruco 模块共有 25 个预定义的标记字典。字典中的所有标记包含相同数量的块或位（4×4、5×5、6×6 或 7×7），每个字典包含固定数量的标记（50、100、250 或 1000）

上面的 drawMarker 函数让我们从 250 个标记的集合中选择具有给定 id（第二个参数 - 66）的标记，这些标记的 id 从 0 到 249。

drawMarker 函数的第三个参数决定生成的标记的大小。在上面的例子中，它会生成一个 200×200 像素的图像。

第四个参数表示将存储生成的标记的对象（上面的markerImage）。

最后，第五个参数是厚度参数，它决定了应该添加多少块作为生成的二进制模式的边界。

在上面的示例中，将在 6×6 生成的模式周围添加 1 位边界，以在 200×200 像素图像中生成具有 7×7 位的图像。

下面我们来生成一些 markers

import cv2 as cv
import numpy as np

size = 200
padding = 1

# Load the predefined dictionary
dictionary = cv.aruco.Dictionary_get(cv.aruco.DICT_6X6_250)

# Generate the marker
markerImage = np.zeros((size, size), dtype=np.uint8)

for id in range(0, 250):
    markerImage = cv.aruco.drawMarker(dictionary, id, size, markerImage, padding)
    cv.imwrite(f"./images/marker{id}.png", markerImage)

3、代码实现

# This code is written by Sunita Nayak at BigVision LLC. It is based on the OpenCV project. It is subject to the license terms in the LICENSE file found in this distribution and at http://opencv.org/license.html

# Usage example:   python3 augmented_reality_with_aruco.py --image=test.jpg
#                  python3 augmented_reality_with_aruco.py --video=test.mp4

import cv2 as cv
#from cv2 import aruco
import argparse
import sys
import os.path
import numpy as np

parser = argparse.ArgumentParser(description='Augmented Reality using Aruco markers in OpenCV')
parser.add_argument('--image', help='Path to image file.')
parser.add_argument('--video', help='Path to video file.')
args = parser.parse_args()

im_src = cv.imread("1.jpg") # 选择插入的图片

outputFile = "ar_out_py.avi"
if (args.image):
    # Open the image file
    if not os.path.isfile(args.image):
        print("Input image file ", args.image, " doesn't exist")
        sys.exit(1)
    cap = cv.VideoCapture(args.image)
    outputFile = args.image[:-4]+'_ar_out_py.jpg'
elif (args.video):
    # Open the video file
    if not os.path.isfile(args.video):
        print("Input video file ", args.video, " doesn't exist")
        sys.exit(1)
    cap = cv.VideoCapture(args.video)
    outputFile = args.video[:-4]+'_ar_out_py.avi'
    print("Storing it as :", outputFile)
else:
    # Webcam input
    cap = cv.VideoCapture(0)

# Get the video writer initialized to save the output video
if (not args.image):
    vid_writer = cv.VideoWriter(outputFile, cv.VideoWriter_fourcc('M','J','P','G'), 28,
                                (round(2*cap.get(cv.CAP_PROP_FRAME_WIDTH)),
                                 round(cap.get(cv.CAP_PROP_FRAME_HEIGHT))))

winName = "Augmented Reality using Aruco markers in OpenCV"

while cv.waitKey(1) < 0:
    try:
        # get frame from the video
        hasFrame, frame = cap.read()
        
        # Stop the program if reached end of video
        if not hasFrame:
            print("Done processing !!!")
            print("Output file is stored as ", outputFile)
            cv.waitKey(3000)
            break

        #Load the dictionary that was used to generate the markers.
        dictionary = cv.aruco.Dictionary_get(cv.aruco.DICT_6X6_250)
        
        # Initialize the detector parameters using default values
        parameters = cv.aruco.DetectorParameters_create()
        
        # Detect the markers in the image
        markerCorners, markerIds, rejectedCandidates = cv.aruco.detectMarkers(frame, dictionary, parameters=parameters)

        if True:
            if len(markerCorners) > 0:
                # Flatten the ArUCo IDs list
                ids = markerIds.flatten()
                image = frame.copy()
                # Loop over the detected ArUCo corners
                for (markerCorner, markerID) in zip(markerCorners, markerIds):
                    # Extract the markers corners which are always returned in the following order:
                    # TOP-LEFT, TOP-RIGHT, BOTTOM-RIGHT, BOTTOM-LEFT
                    corners = markerCorner.reshape((4, 2))
                    (topLeft, topRight, bottomRight, bottomLeft) = corners
                    # Convert each of the (x, y)-coordinate pairs to integers
                    topRight = (int(topRight[0]), int(topRight[1]))
                    bottomRight = (int(bottomRight[0]), int(bottomRight[1]))
                    bottomLeft = (int(bottomLeft[0]), int(bottomLeft[1]))
                    topLeft = (int(topLeft[0]), int(topLeft[1]))
                    # Draw the bounding box of the ArUCo detection
                    cv.line(image, topLeft, topRight, (0, 255, 0), 2)
                    cv.line(image, topRight, bottomRight, (0, 255, 0), 2)
                    cv.line(image, bottomRight, bottomLeft, (0, 255, 0), 2)
                    cv.line(image, bottomLeft, topLeft, (0, 255, 0), 2)
                    # Compute and draw the center (x, y) coordinates of the ArUCo marker
                    cX = int((topLeft[0] + bottomRight[0]) / 2.0)
                    cY = int((topLeft[1] + bottomRight[1]) / 2.0)
                    cv.circle(image, (cX, cY), 4, (0, 0, 255), -1)
                    # Draw the ArUco marker ID on the image
                    cv.putText(image, str(markerID), (topLeft[0], topLeft[1] - 15), cv.FONT_HERSHEY_SIMPLEX,
                                0.5, (0, 255, 0), 2)
                    print("[INFO] ArUco marker ID: {}".format(markerID))
                    # write the output image
                    # cv.imwrite("{}_{}.jpg".format(args["type"], markerID), image)
                    # Show the output image
                    cv.imshow("Image", image)
                    cv.waitKey(0)

        index = np.squeeze(np.where(markerIds == 25))  # id 25 左上角
        refPt1 = np.squeeze(markerCorners[index[0]])[1]
        
        index = np.squeeze(np.where(markerIds == 33))  # id 33 右上角
        refPt2 = np.squeeze(markerCorners[index[0]])[2]

        distance = np.linalg.norm(refPt1-refPt2)  # 左上右上的欧氏距离
        
        scalingFac = 0.02
        pts_dst = [[refPt1[0] - round(scalingFac*distance),
                    refPt1[1] - round(scalingFac*distance)]]  # 边界外扩

        pts_dst = pts_dst + [[refPt2[0] + round(scalingFac*distance),
                              refPt2[1] - round(scalingFac*distance)]]  # 边界外扩
        
        index = np.squeeze(np.where(markerIds == 30))  # id 30 右下
        refPt3 = np.squeeze(markerCorners[index[0]])[0]
        pts_dst = pts_dst + [[refPt3[0] + round(scalingFac*distance),
                              refPt3[1] + round(scalingFac*distance)]]  # 边界外扩

        index = np.squeeze(np.where(markerIds == 23))  # id 30 左下
        refPt4 = np.squeeze(markerCorners[index[0]])[0]
        pts_dst = pts_dst + [[refPt4[0] - round(scalingFac*distance),
                              refPt4[1] + round(scalingFac*distance)]]  # 边界外扩
        # [[519.0, 184.0], [968.0, 142.0], [978.0, 654.0], [509.0, 640.0]]

        pts_src = [[0,0], [im_src.shape[1], 0], [im_src.shape[1], im_src.shape[0]], [0, im_src.shape[0]]]
        # [[0, 0], [1707, 0], [1707, 1280], [0, 1280]]
        
        pts_src_m = np.asarray(pts_src)
        pts_dst_m = np.asarray(pts_dst)

        # Calculate Homography
        h, status = cv.findHomography(pts_src_m, pts_dst_m)
        """
        h
            array([[ 2.01635985e-01, -2.38521041e-02,  5.19000000e+02],
                   [-3.36113857e-02,  3.36082325e-01,  1.84000000e+02],
                   [-6.34282837e-05, -3.15119923e-05,  1.00000000e+00]])
               
        status
            array([[1],
                   [1],
                   [1],
                   [1]], dtype=uint8)
        """
        
        # Warp source image to destination based on homography
        warped_image = cv.warpPerspective(im_src, h, (frame.shape[1],frame.shape[0]))
        
        # Prepare a mask representing region to copy from the warped image into the original frame.
        mask = np.zeros([frame.shape[0], frame.shape[1]], dtype=np.uint8)
        cv.fillConvexPoly(mask, np.int32([pts_dst_m]), (255, 255, 255), cv.LINE_AA)

        # Erode the mask to not copy the boundary effects from the warping
        element = cv.getStructuringElement(cv.MORPH_RECT, (3,3))
        mask = cv.erode(mask, element, iterations=3)

        # Copy the mask into 3 channels.
        warped_image = warped_image.astype(float)
        mask3 = np.zeros_like(warped_image)
        for i in range(0, 3):
            mask3[:,:,i] = mask/255

        # Copy the warped image into the original frame in the mask region.
        warped_image_masked = cv.multiply(warped_image, mask3)
        frame_masked = cv.multiply(frame.astype(float), 1-mask3)
        im_out = cv.add(warped_image_masked, frame_masked)
        
        # Showing the original image and the new output image side by side
        concatenatedOutput = cv.hconcat([frame.astype(float), im_out])
        cv.imshow("AR using Aruco markers", concatenatedOutput.astype(np.uint8))
        
        # Write the frame with the detection boxes
        if (args.image):
            cv.imwrite(outputFile, concatenatedOutput.astype(np.uint8))
        else:
            vid_writer.write(concatenatedOutput.astype(np.uint8))


    except Exception as inst:
        print(inst)

cv.destroyAllWindows()
if 'vid_writer' in locals():
    vid_writer.release()
    print('Video writer released..')

aruco markers 检测结果

根据原壁画中左上右上的距离，使壁画向边缘都快扩了一些些 scalingFace = 0.02，这样方便新壁画覆盖原始壁画的时候，可以覆盖完全，不会露出来原始的画

id 25 左上角边界点（壁画的左上角）向左向上移动

id 33 右上角边界点（壁画的右上角）向右向下移动

id 30 右下角边界点（壁画的右下角）向右向下移动

id 23 左下角边界点（壁画的左下角）向左向下移动

应用单应性矩阵变化后得到的 warped_image

mask 和 mask-erode 的区别，可以看出边缘是做了一些处理，使得融合后更加自然

warped_image 和 warped_image_masked 的区别

最终结果展示

4、更多例子展示

test_ar_out_py

5、涉及到的库

cv2.findHomography

cv2.findHomography 是 OpenCV 库中的一个函数，用于计算两个平面之间的单应性矩阵（Homography Matrix）。单应性矩阵是一个 3x3 的矩阵，它可以用来将一个平面上的点映射到另一个平面上的对应点，常用于图像校正、图像拼接、视角变换等场景。

一、函数原型

H, status = cv2.findHomography(srcPoints, dstPoints, method=0, ransacReprojThresh=5.0, mask=None[, maxIters=2000[, confidence=0.995]])

二、参数说明

• srcPoints: 源图像中的点集，通常是 numpy 数组，形状为 (N, 1, 2) 或 (N, 2)，其中 N 是点的数量，每个点由 (x, y) 坐标表示。
• dstPoints: 目标图像中对应点的点集，与 srcPoints 有相同的形状和数量。
• method: 计算单应性矩阵的方法。常用值为 0（默认）、cv2.RANSAC、cv2.LMEDS、cv2.RHO。cv2.RANSAC 是最常用的，因为它对噪声和异常值具有很好的鲁棒性。
• ransacReprojThresh: 仅当 method 为 cv2.RANSAC 或 cv2.LMEDS 时使用。它指定了最大重投影误差，以决定哪些点是内点。
• mask: 可选参数，输出掩码，表示哪些点是内点（值为 1）或外点（值为 0）。
• maxIters: cv2.RANSAC 方法中的最大迭代次数，默认为 2000。
• confidence: cv2.RANSAC 方法的置信度，即内点在所有点中所占的比例，默认为 0.995。

返回值

• retval: 计算得到的单应性矩阵 3x3 。
• status: 如果提供了该参数，函数会填充这个数组，表示每个点是否为内点。

三、使用示例

假设你有两张图片，并已经找到了一些对应的特征点，你可以使用这些点来计算单应性矩阵，然后使用这个矩阵进行图像变换。

import cv2  
import numpy as np  
  
# 假设 src_pts 和 dst_pts 是你已经找到的对应点  
src_pts = np.float32([[56, 65], [128, 159], [23, 145], [100, 100]])  
dst_pts = np.float32([[10, 50], [200, 200], [100, 100], [150, 150]])  
  
# 计算单应性矩阵  
H, status = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)  
  
# 使用单应性矩阵进行图像变换（此处略去）

注意，在实际应用中，特征点通常是通过特征检测算法（如 SIFT、SURF、ORB 等）自动找到的。

6、参考

• AttributeError: module ‘cv2.aruco’ has no attribute ‘Dictionary_get’

  
  
  opencv-python 也用的是 4.6.0.66 版本

• OpenCV进阶（10）在 OpenCV 中使用 ArUco 标记的增强现实

• 代码出处
  链接：https://pan.baidu.com/s/10OERAD4NQlg83eTuxj_LkQ
  提取码：123a

• https://learnopencv.com/augmented-reality-using-aruco-markers-in-opencv-c-python/

• 【python】OpenCV—Aruco