Camera Calibration

Goal

In this section, we will learn about

• types of distortion caused by cameras
• how to find the intrinsic and extrinsic properties of a camera
• how to undistort images based off these properties

Basics

Some pinhole cameras introduce significant distortion to images. Two major kinds of distortion are radial distortion and tangential distortion.

Radial distortion causes straight lines to appear curved. Radial distortion becomes larger the farther points are from the center of the image. For example, one image is shown below in which two edges of a chess board are marked with red lines. But, you can see that the border of the chess board is not a straight line and doesn't match with the red line. All the expected straight lines are bulged out. Visit Distortion (optics) for more details.

See also

This function may not be able to find the required pattern in all the images. So, one good option is to write the code such that, it starts the camera and check each frame for required pattern. Once the pattern is obtained, find the corners and store it in a list. Also, provide some interval before reading next frame so that we can adjust our chess board in different direction. Continue this process until the required number of good patterns are obtained. Even in the example provided here, we are not sure how many images out of the 14 given are good. Thus, we must read all the images and take only the good ones.

Instead of chess board, we can alternatively use a circular grid. In this case, we must use the function cv.findCirclesGrid() to find the pattern. Fewer images are sufficient to perform camera calibration using a circular grid.

Once we find the corners, we can increase their accuracy using cv.cornerSubPix(). We can also draw the pattern using cv.drawChessboardCorners(). All these steps are included in below code:

import numpy as np
import cv2 as cv
import glob
# termination criteria
criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((6*7,3), np.float32)
objp[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
images = glob.glob('*.jpg')
for fname in images:
    img = cv.imread(fname)
    gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
    # Find the chess board corners
    ret, corners = cv.findChessboardCorners(gray, (7,6), None)
    # If found, add object points, image points (after refining them)
    if ret == True:
        objpoints.append(objp)
        corners2 = cv.cornerSubPix(gray,corners, (11,11), (-1,-1), criteria)
        imgpoints.append(corners)
        # Draw and display the corners
        cv.drawChessboardCorners(img, (7,6), corners2, ret)
        cv.imshow('img', img)
        cv.waitKey(500)
cv.destroyAllWindows()

One image with pattern drawn on it is shown below:

Calibration

Now that we have our object points and image points, we are ready to go for calibration. We can use the function, cv.calibrateCamera() which returns the camera matrix, distortion coefficients, rotation and translation vectors etc.

ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)

Undistortion

Now, we can take an image and undistort it. OpenCV comes with two methods for doing this. However first, we can refine the camera matrix based on a free scaling parameter using cv.getOptimalNewCameraMatrix(). If the scaling parameter alpha=0, it returns undistorted image with minimum unwanted pixels. So it may even remove some pixels at image corners. If alpha=1, all pixels are retained with some extra black images. This function also returns an image ROI which can be used to crop the result.

So, we take a new image (left12.jpg in this case. That is the first image in this chapter)

img = cv.imread('left12.jpg')

h, w = img.shape[:2]

newcameramtx, roi = cv.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h))

1. Using cv.undistort()

This is the easiest way. Just call the function and use ROI obtained above to crop the result.

# undistort

dst = cv.undistort(img, mtx, dist, None, newcameramtx)

# crop the image

x, y, w, h = roi

dst = dst[y:y+h, x:x+w]

cv.imwrite('calibresult.png', dst)

2. Using remapping

This way is a little bit more difficult. First, find a mapping function from the distorted image to the undistorted image. Then use the remap function.

# undistort

mapx, mapy = cv.initUndistortRectifyMap(mtx, dist, None, newcameramtx, (w,h), 5)

dst = cv.remap(img, mapx, mapy, cv.INTER_LINEAR)

# crop the image

x, y, w, h = roi

dst = dst[y:y+h, x:x+w]

cv.imwrite('calibresult.png', dst)

Still, both the methods give the same result. See the result below:

You can see in the result that all the edges are straight.

Now you can store the camera matrix and distortion coefficients using write functions in NumPy (np.savez, np.savetxt etc) for future uses.

Re-projection Error

Re-projection error gives a good estimation of just how exact the found parameters are. The closer the re-projection error is to zero, the more accurate the parameters we found are. Given the intrinsic, distortion, rotation and translation matrices, we must first transform the object point to image point using cv.projectPoints(). Then, we can calculate the absolute norm between what we got with our transformation and the corner finding algorithm. To find the average error, we calculate the arithmetical mean of the errors calculated for all the calibration images.

mean_error = 0

for i in xrange(len(objpoints)):

imgpoints2, _ = cv.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)

error = cv.norm(imgpoints[i], imgpoints2, cv.NORM_L2)/len(imgpoints2)

mean_error += error

print( "total error: {}".format(mean_error/len(objpoints)) )

Additional Resources

Exercises

1. Try camera calibration with circular grid.