A Brief Summary of Techniques used in Image Stabilization

Author: Liang Zixuan(UESTC, Glasgow College 2017)

Introduction

With the development of technology and the popularity of electronic devices, we can easily use a variety of electronic devices such as the car recorder, handheld digital camera, mobile phone and aerial drone to obtain videos. However, during the process of recording the video, the image may be shaken due to mechanical vibration of the carrier or the natural environment, thus, the image may be unstable. For example, if a car encounters a pothole area during driving the shaking will make the video recorded by the on-board recorder to contain severe jitters. If a photographer holding a digital camera or mobile phone while walking and shooting, the captured video will also contain jitters. Therefore, Image Stabilization (IS) technology comes into being. This technology contains three major methods: Building mechanical steadicam system, Optical Image Stabilization (OIS) and electronic image stabilization (EIS). This blog introduces the three methods briefly and makes some comparisons between them.

Different Methods

Mechanical Steadicam System

The sensor that captures the image can be moved by counteracting the motion of the camera, this technique is commonly referred as mechanical image stabilization. One method of mechanical image stabilization is to move and rotate the lens itself to ensure the stability of the captured image (self-balancing platform). This method requires no additional capabilities of any camera body–lens combination consists of stabilizing the entire camera body externally rather than using an internal method. This is achieved by attaching a gyroscope to the camera body, usually using the camera’s built-in tripod mount. This lets the external gyro stabilize the camera, and is typically used in photography from a moving vehicle, when a lens or camera offering another type of image stabilization is not available.

The principle of the method above is simple (as shown in the figure): When the camera rotates and causes an angular error, the gyroscope encodes the information onto the actuator of the motion sensor. The motion sensor can maintain the image’s projection on the image plane as a function of the focal length of the lens used. Modern camera can automatically capture focal length information from modern lenses used inside it. Some lenses are modified with chips that can detect the focal length automatically. This technique relies on very accurate angular rate sensors to detect camera motion. Some manufacturers use DSP to analyze images in real time and then move the sensors appropriately. Fujifilm, Samsung, Casio Exilim and Ricoh Caplio also use such techniques in some cameras.

Another technique for stabilizing a video or motion picture camera body is the Steadicam system, which isolates the camera from the operator’s body using a harness and a camera boom with a counterweight (as shown in the following figure).

Moving the image sensor inside the lens to achieve stabilization is also widely used. The advantage of moving an image sensor instead of the camera itself or its lens is that the image can be stabilized even on a lens that is made without stability. This allows the stabilized lens to work with many other unstable lenses and reduces the weight and complexity of the lens. In addition, when sensor-based image stabilization techniques are improved, it only needs to be replaced with a camera to take advantage of the improvements, which is typically much cheaper than replacing all existing lenses with lens-based image stabilization. Some sensor-based image stabilization enables correction of camera film rotation, which is easily triggered by simply pressing the shutter button. The available “rolling” compensation also allows the camera to automatically correct the oblique field of view in the optical zone, provided it is equipped with an electronic level.

Optical image stabilization

An optical image stabilizer is a mechanism used in a still camera or video camera that stabilizes the recorded image by varying the optical path to the sensor. it redirects the light path or moves the imaging plate to achieve image stabilization. At present, optical image stabilization systems mostly rely on gyro sensors to measure the amount of movement or rotation of the device. Usually two gyro sensors are used to measure the angular velocity of the horizontal and vertical directions of the device. The output signals of the two are passed through an analog-to-digital (ADC) converter, then converted to digital information.

Although the definition of OIS is very similar with built-in image stabilization, which operates by moving the sensor as the final element in the optical path, there are significant differences between them in the figure above, sensor shift is the principle of built-in image stabilization, while lens shift is OIS. OIS is implemented in the lens itself, instead of the arrangement and adjustment of different lenses. In addition, the key element of all optical stabilization systems is that they stabilize the image projected on the sensor before the sensor converts the image into digital information. OIS also has advantages over in-body stabilization. In low-light or low-contrast situations, the autofocus system (which has no stabilized sensors) is able to work more accurately when the image coming from the lens is already stabilized. In cameras with optical viewfinders, the image seen by the photographer through the stabilized lens (as opposed to in-body stabilization) reveals more detail because of its stability, and it also makes correct framing easier. This is especially the case with longer telephoto lenses. This advantage does not occur on compact system cameras, because the sensor output to the screen or electronic viewfinder would be stabilized.

The OIS system works like this: two piezoelectric angular velocity sensors (commonly called gyro sensors) are used to detect vibration, one for detecting horizontal movement and the other for detecting vertical movement. Therefore, such an image stabilizer corrects only the pitch and yaw axis rotations, and cannot correct the rotation around the optical axis. Some lenses have an assist mode that counteracts vertical camera shake. This mode is useful when using panning technology. Switching to this mode depends on the lens; sometimes it is done by using a switch on the lens, or it can be automatic. For example, to compensate for the jitter in the video taken while walking, Panasonic introduced the Power Hybrid O.I.S + with 5-axis correction: axis rotation, horizontal rotation, vertical rotation, horizontal and vertical.

Digital image stabilization

At present, the electronic image stabilization method is generally divided into three parts, namely motion estimation, motion smoothing and motion compensation. According to different motion models in motion estimation, it can be divided into 2D method, 3D method and 2.5D method. Among them, the 2D method has a fast processing speed, but for some scenes with large depth changes, a good image stabilization result cannot often obtained. The 3D method usually uses SFM algorithms or sensors for 3D reconstruction. Therefore, the 3D method can process many videos with complex jitters, including videos with depth changes. However, 3D methods often need to track long motion trajectories, and solve non-linear problems by SFM algorithms, so the 3D method is slower and less efficient. The 2.5D method has recently been developed and is a combination of the 2D method and the 3D method. Therefore, the image stabilization result is generally better than the 2D method and worse than the 3D method.

Unlike the other two systems, digital image stabilization uses software technology to reduce the impact of shaky hands on video. Depending on the model, different camcorders will calculate the impact of your body movement and use that data to adjust which pixels on the camcorder’s image sensor are being used. It uses pixels from beyond the visible frame as motion buffer to smooth over the transition frame by frame. There are also software programs that can apply a stabilization filter to the video even after it’s been taken, by tracking the pixel movements and adjusting the frame. However, this results in either a smaller cropped image due to a reduced frame or extrapolation to fill in the lost edges. The effect of such approach is also not as obvious as adjusting when recording.

The general process of DIS goes like this:

motion estimation

Motion estimation is the process of determining camera motion vectors under a specific camera motion model that describes the amount of motion conversion between 2D images (usually continuous video frames). Camera motion vectors can be modeled using translation models, or other models that approximate the actual camera motion. The camera motion vector refers to the global motion vector associated with the motion of the entire image, and its estimation is often done by local motion vectors. The local motion vector refers to a certain part of the image. Motion vectors include the motion of rectangular blocks, arbitrary shape blocks, or even each pixel. Local motion estimation in traditional 2D image stabilization is often the first thing to do, the global motion vector is then determined by local vector.

motion compensation

Camera motion compensation is an algorithm used to generate new camera motion, which suppresses jitters generation and produces more stable camera motion. For 2D video stabilization, the motion compensation mechanism receives camera 2D motion data. The algorithm calculates new smooth 2D motion data. Motion compensation is generally divided into two types of schemes: motion path smoothing and motion path fitting. From the perspective of digital signal processing, motion path smoothing is to eliminate these motion noises. Video stabilization The smoothing filters commonly used in the scheme are moving average filter, Gaussian filter, Kalman filter particle filter and other polynomial filters.

Motion path fitting is different from motion path smoothing. Motion path fitting mimics professional film photographic paths, such as straight lines, parabolas, etc. This scheme can obtain more stable motion compensation results than motion path smoothing because it can not only remove high frequency jitters but invalid low frequency jitter in the path.

Of course, the motion path fitting method will increase the smooth motion compensation result, and also increase the missing area that needs to be repaired, which increases the difficulty of the image repair process.

3D

In 3D image stabilization, motion estimation is the problem of recovering camera pose from image information under the camera projection transformation model. 3D motion parameters include translation and rotation parameters, and structure from motion (SFM) is used to solve this problem. The problem is recovering the corresponding 3D information from the 2D image sequence, including the motion parameters of the imaging camera and the structural information of the scene. The SFM problem can be solved by the dual-view, three-view or multi-view solution, and in the 3D image stabilization, dual-view method is used, that is, the image of the camera of two angles of view is restored, and the position information of the two imaging time cameras is restored.

The process can be revealed by the following flow chart:

Comparison

The comparison between the three methods can be shown by the following table:

Usually, mechanical steadicam systems are bulky and have many other disadvantages. Although built-in image stabilization is very portable, it still requires the lens to have a larger output image circle because the sensor will move during exposure and therefore consume most of the image. Compared to lens movement in an optical image stabilization system (see below), the sensor moves very much, so efficiency is limited by the maximum range of sensor movement, with typical modern optically stable lenses having greater freedom. The speed and range of motion required for the sensor increase with the focal length of the lens used, making sensor shifting techniques less suitable for telephoto telephoto lenses, especially when using slower shutter speeds, because of the sensor’s available range of motion It quickly became insufficient to cope with the ever-increasing image displacement.

Another main drawback of the moving image sensor is that the image projected onto the viewfinder is unstable. However, this is not a problem for a camera using an electronic viewfinder (EVF) because the image projected on the viewfinder is taken from the image sensor itself. Similarly, images projected outside the image sensor phase detection auto-focus system (if used) can become also unstable.

As for the OIS, the main disadvantage is about the power. Most manufacturers recommend turning off the lens’s OIS function when mounting the lens on a tripod because it can lead to unstable results and is usually unnecessary. Many modern image stabilization lenses (especially Canon’s most recent OIS lenses) automatically detect if they are mounted on a tripod (due to extremely low vibration readings) and automatically disable OIS to prevent this from happening and the resulting image The quality is reduced. The OIS system also increases power consumption, so disabling it when it is not needed will extend battery life.

Besides, OIS has another main disadvantage, cost. Each lens requires its own image stabilization system. Also, not every lens is available in an image-stabilized version. This is often the case for fast primes and wide-angle lenses. Which are much more expensive.

For consumer digital camcorders, digital image stabilization is usually less effective than optical stabilization. Given that, it pays to look closely when a camcorder claims to have “image stabilization.” It might only be of the digital variety.

Electronic image stabilization mainly uses image processing technology to compensate for motion and obtain stable video. The electronic image stabilization effect is good, the cost and power consumption are low, and the volume is small. Therefore, the electronic image stabilization is the current mainstream video stabilization technology, which has been widely used in real life.

Reference

[1]干文杰.基于结构保持的视频稳像方法研究[D].中国科学技术大学,2018.
[2]魏闪闪, 谢巍, 贺志强. 数字视频稳像技术综述[J]. 计算机研究与发展, 2017, 第54卷(9):2044-2058.