Python+OpenCV：Optical Flow（光流）

Python+OpenCV：Optical Flow（光流）

理论

Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera.

It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second.

Consider the image below:

It shows a ball moving in 5 consecutive frames. The arrow shows its displacement vector.

Optical flow has many applications in areas like :

• Structure from Motion
• Video Compression
• Video Stabilization ...

Optical flow works on several assumptions:

1. The pixel intensities of an object do not change between consecutive frames.
2. Neighbouring pixels have similar motion.

Consider a pixel I(x,y,t) in first frame (Check a new dimension, time, is added here. Earlier we were working with images only, so no need of time). It moves by distance (dx,dy) in next frame taken after dt time. So since those pixels are the same and intensity does not change, we can say,

Then take taylor series approximation of right-hand side, remove common terms and divide by dt to get the following equation:

where:

Above equation is called Optical Flow equation.

In it, we can find fx and fy, they are image gradients.

Similarly ft is the gradient along time.

But (u,v) is unknown. We cannot solve this one equation with two unknown variables.

So several methods are provided to solve this problem and one of them is Lucas-Kanade.

Lucas-Kanade method

We have seen an assumption before, that all the neighbouring pixels will have similar motion.

Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion.

We can find (fx,fy,ft) for these 9 points. So now our problem becomes solving 9 equations with two unknown variables which is over-determined.

A better solution is obtained with least square fit method. Below is the final solution which is two equation-two unknown problem and solve to get the solution.

( Check similarity of inverse matrix with Harris corner detector. It denotes that corners are better points to be tracked.)

So from the user point of view, the idea is simple, we give some points to track, we receive the optical flow vectors of those points. But again there are some problems.

Until now, we were dealing with small motions, so it fails when there is a large motion.

To deal with this we use pyramids. When we go up in the pyramid, small motions are removed and large motions become small motions.

So by applying Lucas-Kanade there, we get optical flow along with the scale.

Lucas-Kanade Optical Flow in OpenCV

####################################################################################################
# Lucas-Kanade光学流(Lucas-Kanade Optical Flow)
def lmc_cv_lucas_kanade_optical_flow():
    """
        函数功能: Lucas-Kanade光学流(Lucas-Kanade Optical Flow)。
    """

    # 读取视频
    parser = argparse.ArgumentParser(description='This sample demonstrates Lucas-Kanade Optical Flow calculation.')
    parser.add_argument('--input', type=str, help='Path to a video or a sequence of image.',
                        default='D:/99-Research/Python/Image/slow_traffic_small.mp4')
    args = parser.parse_args()
    cap = lmc_cv.VideoCapture(lmc_cv.samples.findFileOrKeep(args.input))
    # params for ShiTomasi corner detection
    feature_params = dict(maxCorners=100, qualityLevel=0.3, minDistance=7, blockSize=7)
    # Parameters for lucas kanade optical flow
    lk_params = dict(winSize=(15, 15), maxLevel=2,
                     criteria=(lmc_cv.TERM_CRITERIA_EPS | lmc_cv.TERM_CRITERIA_COUNT, 10, 0.03))
    # Create some random colors
    color = np.random.randint(0, 255, (100, 3))
    # Take first frame and find corners in it
    ret, old_frame = cap.read()
    old_gray = lmc_cv.cvtColor(old_frame, lmc_cv.COLOR_BGR2GRAY)
    p0 = lmc_cv.goodFeaturesToTrack(old_gray, mask=None, **feature_params)
    # Create a mask image for drawing purposes
    mask = np.zeros_like(old_frame)
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        frame_gray = lmc_cv.cvtColor(frame, lmc_cv.COLOR_BGR2GRAY)
        # calculate optical flow
        p1, st, err = lmc_cv.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
        # Select good points
        good_new = p1[st == 1]
        good_old = p0[st == 1]
        # draw the tracks
        for i, (new, old) in enumerate(zip(good_new, good_old)):
            a, b = new.ravel()
            c, d = old.ravel()
            mask = lmc_cv.line(mask, (int(a), int(b)), (int(c), int(d)), color[i].tolist(), 2)
            frame = lmc_cv.circle(frame, (int(a), int(b)), 5, color[i].tolist(), -1)
        result_image = lmc_cv.add(frame, mask)
        lmc_cv.imshow('Lucas-Kanade Optical Flow', result_image)
        keyboard = lmc_cv.waitKey(30) & 0xff
        if keyboard == 'q' or keyboard == 27:
            break
        # Now update the previous frame and previous points
        old_gray = frame_gray.copy()
        p0 = good_new.reshape(-1, 1, 2)
    return

Dense Optical Flow in OpenCV

####################################################################################################
# Lucas-Kanade密集光学流(Lucas-Kanade Dense Optical Flow)
def lmc_cv_lucas_kanade_dense_optical_flow():
    """
        函数功能: Lucas-Kanade密集光学流(Lucas-Kanade Dense Optical Flow)。
    """

    # 读取视频
    parser = argparse.ArgumentParser(
        description='This sample demonstrates Lucas-Kanade Dense Optical Flow calculation.')
    parser.add_argument('--input', type=str, help='Path to a video or a sequence of image.',
                        default='D:/99-Research/Python/Image/slow_traffic_small.mp4')
    args = parser.parse_args()
    cap = lmc_cv.VideoCapture(lmc_cv.samples.findFileOrKeep(args.input))
    ret, frame1 = cap.read()
    prvs = lmc_cv.cvtColor(frame1, lmc_cv.COLOR_BGR2GRAY)
    hsv = np.zeros_like(frame1)
    hsv[..., 1] = 255
    while True:
        ret, frame2 = cap.read()
        if not ret:
            break
        next_image = lmc_cv.cvtColor(frame2, lmc_cv.COLOR_BGR2GRAY)
        flow = lmc_cv.calcOpticalFlowFarneback(prvs, next_image, None, 0.5, 3, 15, 3, 5, 1.2, 0)
        mag, ang = lmc_cv.cartToPolar(flow[..., 0], flow[..., 1])
        hsv[..., 0] = ang * 180 / np.pi / 2
        hsv[..., 2] = lmc_cv.normalize(mag, None, 0, 255, lmc_cv.NORM_MINMAX)
        bgr = lmc_cv.cvtColor(hsv, lmc_cv.COLOR_HSV2BGR)
        lmc_cv.imshow('Lucas-Kanade Dense Optical Flow', bgr)
        keyboard = lmc_cv.waitKey(30) & 0xff
        if keyboard == 'q' or keyboard == 27:
            break
        elif keyboard == ord('s'):
            lmc_cv.imwrite('opticalfb.png', frame2)
            lmc_cv.imwrite('opticalhsv.png', bgr)
        prvs = next_image
    return