Bayer Pattern to RGB

Sensor3彩色数字相机需要3个单色sensor获得彩色图像的R，G，B分量，成本较高。单CCD获得彩色图像的方法是在 CCD表面覆盖1个只含红、绿、蓝3色的马赛克滤镜，对其输出信号通过一定的处理算法实现。这个设计理念最初由拜尔提出，所以这种滤镜也被称作拜尔模板 (bayer pattern)。

The Bayer color filter array (CFA)  is shown below. Half of the total number of pixels are green (G), while a quarter of the total number is assigned to both red (R) and blue (B).

In order to obtain this color information, the  color image sensor is covered with either a red, a green, or a blue filter, in a repeating pattern. This pattern, or sequence, of filters can vary, but the widely adopted “Bayer“pattern, which was invented at Kodak, is a repeating 2x2 arrangement.

When the image sensor is read out, line by line, the pixel sequence comes out GRGRGR, etc., and then the alternate line sequence is BGBGBG, etc. This output is called sequential RGB (or sRGB).

Since each pixel has been made sensitive only to one color (one spectral band), the overall sensitivity of a color image sensor is lower than a monochrome (panchromatic) sensor, and in fact is typically 3x less sensitive. As a result, monochrome sensors are better for low-light applications, such as security cameras. (It is also why human eyes switch to black and white mode in the dark).

尽管通过带有Bayer滤镜的单sensor相机采集的原始图像带有R，G，B三基色分量，但是不能不加任何算法处理，仅简单地将3种分量分离。这样不仅图像分辨率很差，而且各像素点的三基色分量比例与被摄目标相比，失真也很严重。

To convert an image from the bayer format to an RGB per pixel format, we need to interpolate the two missing color values in each pixel. Several standard interpolation methods (nearest neighbor, linear, cubic, cubic spline, etc.) were evaluated on this problem in [1]. The authors have measured interpolation accuracy as well as the speed of the method and concluded that the best performance is achieved by a correlation-adjusted version of the linear interpolation. The suggested method is presented here.

Interpolating red and blue components

Four possible cases for interpolating R and B components

As suggested in [1], R and B values are interpolated linearly from the nearest neighbors of the same color. There are four are possible cases, as shown above. When interpolating the missing values of R and B on a green pixel, as in (a) and (b), we take the average values of the two nearest neighbors of the same color. For example, in  (a), the value for the blue component on a shaded G pixel will be the average of the blue pixels above and below the G pixel, while the value for the red component will be the average of the two red pixels to the left and right of the G pixel.

(c) shows the case when the value of the blue component is to be interpolated for an R pixel. In such case, we take the average of the four nearest blue pixels cornering the R pixel. Similarly, to determine the value of the red component on a B pixel in (d) we take the average of the four nearest red pixels cornering the B pixel.

Interpolating the green component

By [1], green component is adaptively interpolated from a pair of nearest neighbors. To illustrate the procedure, consider two possible cases below.

Two possible cases for interpolating G component

In  (a), the value of the green component is to be interpolated on an R pixel. The value used for the G component here is

In other words, we take into account the correlation in the red component to adapt the interpolation method. If the difference between R₁ and R₃ is smaller than the difference between R₂ and R₄, indicating that the correlation is stronger in the vertical direction, we use the average of the vertical neighbors G₁ and G₃ to interpolate the required value. If the horizontal correlation is larger, we use horizontal neighbors. If neither direction dominates the correlation, we use all four neighbors.

Similarly, for  (b) we will have

To conclude this section, note that if the speed of execution is the issue, one can safely use simple linear interpolation of the green component from the four nearest neighbors, without any adaptation.

According to [1], this method of interpolation executes twice as fast as the adaptive method, and achieves only slightly worse performance on real images. For even fast updates only two of the four green values are averaged. However, this method displays false color on edges or zipper artifacts.

 

References

[1] T. Sakamoto, C. Nakanishi and T. Hase, “Software pixel interpolation for digital still cameras suitable for a 32-bit MCU,” IEEE Trans. Consumer Electronics, vol. 44, no. 4, November 1998.