Abstract
Most demosaicking algorithms only focus on handling noise-free CFA raw data. In practice, the CFA raw data are corrupted by noise, which degrades demosaicking performance. Full-color image quality strongly depends on the performance of the demosaicking. Here, we propose a CFA raw data denoising algorithm. In the proposed algorithm, the CFA raw data is converted to a pseudo four-channel image by rearranging pixels. Then, the four-channel data are transformed based on the principal component analysis (PCA). Existing high-performance gray image denoising algorithm is applied to each transformed image. Finally, the denoised data is rearranged to obtain denoised CFA raw data. We evaluate both the denoised CFA raw data as well as the full-color image reconstructed with the noisy CFA raw data. Experimental comparisons demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms.
大多数去马赛克算法只关注于处理无噪声的CFA原始数据。在实际应用中,CFA原始数据被噪声破坏,降低了去马赛克性能。全彩图像的质量很大程度上取决于图像去马赛克的表现。本文提出了一种CFA原始数据去噪算法。在该算法中,CFA原始数据通过重新排列像素转换为伪四通道图像。然后,基于主成分分析(PCA)对四通道数据进行转换。现有的高性能灰度图像去噪算法适用于每一幅变换后的图像。最后对去噪后的数据进行重新排列,得到去噪后的CFA原始数据。我们评估了去噪后的CFA原始数据和用噪声CFA原始数据重建的全彩色图像。实验对比表明,该算法优于现有的最先进的算法。
1. INTRODUCTION
A single image sensor with a color filter array (CFA) is widely used for a color image acquisition and the most common pattern is the Bayer pattern CFA [1]. The data acquired by the single image sensor with the CFA is called CFA raw data.
彩色图像采集广泛采用带有彩色滤波器阵列(CFA)的单一图像传感器,最常见的模式是Bayer模式CFA[1]。使用CFA的单图像传感器获取的数据称为CFA原始数据。
In order to reconstruct a full-color image from the CFA raw data, we need to estimate the other two missing color pixel values. This estimation process is called demosaicking. Many demosaicking algorithms have been proposed in many literatures [2, 3, 4, 5, 6, 7].
为了从CFA原始数据中重建一幅全彩图像,我们需要估计另外两个缺失的彩色像素值。这种估计过程称为去马赛克。许多文献[2,3,4,5,6,7]都提出了许多去马赛克算法。
Most demosaicking algorithms only focus on handling the noise-free CFA raw data. However, in practice, we need to take account of the noise added to the CFA raw data. In the presence of noise, the performance of the demosaicking algorithm degrades drastically. Consequently, severe color artifacts appear in the full-color image demosaicked with the noisy CFA raw data.
大多数去马赛克算法只关注于处理无噪声的CFA原始数据。然而,在实践中,我们需要考虑添加到CFA原始数据中的噪声。在噪声存在的情况下,去马赛克算法的性能急剧下降。因此,在用噪声CFA原始数据去马赛克的全彩色图像中出现了严重的颜色伪影。
There are three approaches to generate the full-color image from the noisy CFA raw data. A straightforward approach is denoising-after-demosaicking, where demosaicking and denoising are sequentially applied. Many high-performance
demosaicking algorithms for the noise-free CFA raw data are existing. Many effective denoising algorithms assuming additive white Gaussian noise (AWGN) have also been proposed [15, 16]. But their simple combination provides poor results. One reason is that the demosaicking process changes the statistical properties of the noise. The removal of non-AWGN is a very challenging problem.
从噪声CFA原始数据生成全彩图像有三种方法。一种直接的方法是denoising -after-demosaicking,先去马赛克再去噪。针对无噪声CFA原始数据,已有许多高性能的去马赛克算法。许多有效的去噪算法也被提出假设加性高斯白噪声(AWGN)[15,16]。但是他们简单的结合带来了糟糕的结果。原因之一是去马赛克过程改变了噪声的统计特性。non-AWGN的去除是一个非常具有挑战性的问题。
The second approach is denoising-before-demosaicking. If we can sufficiently reduce the noise of the CFA raw data before the demosaicking process, we can simply apply the demosaicking algorithm for the noise-free CFA raw data to the denoised CFA raw data. However, most high-performance denoising algorithms are designed for a normal gray or fullcolor image. We cannot get the high-performance by directly applying those denoising algorithms to reduce the noise of the CFA raw data because of the underlying mosaic structure of the CFA. Therefore, we need to design a denoising algorithm for noisy CFA raw data as well as some CFA denoising algorithms [8, 9, 10].
第二种方法是denoising-before-demosaicking。如果能在去马赛克前充分降低CFA原始数据的噪声,那么只需将无噪声CFA原始数据的去噪算法应用于去噪CFA原始数据。然而,大多数高性能的去噪算法是为普通灰度或全彩色图像设计的。由于CFA的底层镶嵌结构,直接应用这些去噪算法来降低CFA原始数据的噪声是无法获得高性能的。因此,我们需要设计一种噪声CFA原始数据的去噪算法以及一些CFA去噪算法[8,9,10]。
The third approach is joint denoising and demosaicking [11, 12, 13, 14]. These algorithms of this approach tend to be complicated and require a huge computational cost.
第三种方法是联合去噪和去马赛克[11,12,13,14]。这种方法的算法比较复杂,计算量大。
In this paper, we propose a novel CFA raw data denoising algorithm. In the proposed CFA raw data denoising, the CFA raw data is first divided into four sub-images. Since the set of sub-images can be regarded as a four-channel color image, we call it pseudo four-channel image. In order to improve the denoising performance, the color space of the pseudo fourchannel image is transformed based on the principal component analysis (PCA). Then, existing high-performance denoising algorithm is applied to the pseudo four-channel image in the transformed domain. The denoised pseudo fourchannel image is restored back to the CFA raw data. Finally, the full-color image is generated with the denoised CFA raw data by using an existing high-performance demosaicking algorithm [5].
本文提出了一种新的CFA原始数据去噪算法。在CFA原始数据去噪中,首先将CFA原始数据分为四个子图像。由于这组子图像可以看作是一个四通道彩色图像,我们称之为伪四通道图像。为了提高伪四通道图像的去噪性能,基于主成分分析(PCA)对伪四通道图像的颜色空间进行了变换。然后,将现有的高性能去噪算法应用于变换域中的伪四通道图像。将去噪后的伪四通道图像恢复到CFA原始数据。最后,利用现有的高性能去马赛克算法[5],利用去噪后的CFA原始数据生成全彩图像。
A similar approach, i.e. dividing into four sub-images and denoising them, was proposed by Park et al. [10]. Our proposed method outperforms it due to two mechanisms: adaptive color transformation and block artifact reduction which are explained in the following sections.
Park等人提出了一种类似的方法,即将四个子图像进行分割和去噪。我们提出的方法之所以优于它,是因为有两种机制:自适应颜色转换和块伪影约简。
图1 去马赛克前去噪的图像处理流水线
We evaluate the performance of the CFA raw data denoising and the final full-color image quality reconstructed with the noisy CFA raw data. Experimental comparisons demonstrate that the proposed algorithm outperforms state-of-the-art algorithms.
我们评估了CFA原始数据去噪的性能,以及用噪声CFA原始数据重建的最终全彩色图像质量。实验结果表明,该算法具有较好的性能。
2. DENOISING AND DEMOSAICKING
The noisy CFA raw data y can be expressed as
噪声CFA原始数据y可以表示为
(1)
where x is the vector representation of the noise-free RGB image, M is the matrix which represents sub-sampling operation according to the CFA pattern, and n is the noise. In this paper, we assume the noise is signal independent Gaussian noise with zero-mean. The variance of the noise depends on the channel. The variances of R, G, and B channels are denoted as σ2
R, σG2 , and σB2 , respectively.
其中x为无噪声RGB图像的矢量表示,M为根据CFA模式进行分采样操作的矩阵,n为噪声。在本文中,我们假设噪声是具有零均值的信号独立高斯噪声。 噪声的方差取决于通道。R,G和B通道的方差分别表示为,和。
Let Dn(·) and Dm(·) denote the denoising and the demosaicking processes, respectively. Those processes may be a non-linear process. The full-color image generated by the denoising-after-demosaicking approach can be expressed by
设Dn(·)和Dm(·)分别表示去噪和去马赛克过程。 这些过程可能是非线性过程。 通过去马赛后去噪方法生成的全色图像可以表示为
(2)
The error after demosaicking is
去马赛克后的误差是
(3)
If this error is AWGN, the denoising algorithm can effectively reduce the noise. However, the error is not the AWGN because the demosaicking process is usually a non-linear operation. Non AWGN noise removal is a challenging problem.
如果这个误差是AWGN,去噪算法可以有效地降低噪声。然而,误差不是AWGN,因为解马赛克过程通常是非线性的。非AWGN降噪是一个具有挑战性的问题。
The full-color image generated by the denoising-beforedemosaicking approach can be expressed by
denoising-before-demosaicking方法生成的全彩色图像可以用
(4)
If CFA raw data denoising can efficiently reduce the noise of the noisy CFA raw data, we can simply apply the demosaicking algorithm which assumes the noise-free CFA raw data.
如果CFA原始数据去噪能有效地降低CFA原始数据的噪声,那么我们可以简单地应用demosaicking算法,该算法假设CFA原始数据无噪声。
3. PSEUDO FOUR-CHANNEL IMAGE DENOISING
3.1. Image processing piepeline
The overall image processing pipeline from the noisy CFA raw data to the full-color image is shown in Fig. 1. In the proposed pseudo four-channel image denoising, first, we decompose the noisy CFA raw data into four sub-images. The four sub-images can be considered as one pseudo fourchannel image. We can generate four different types of pseudo four-channel images, namely, a GRBG-channel image, an RGGB-channel image, a BGGR-channel image, and a GBRG-channel image. Each pseudo four-channel image is denoised in the transformed domain as shown in Fig. 2. The denoised four-channel images are rearranged and averaged to generate the denoised CFA raw data. The averaging process can effectively reduce block artifacts caused by a block structure of the sub-image division. Finally, the existing demosaicking algorithm is applied to the denoised CFA raw data to obtain the full-color image.
从噪声CFA原始数据到全彩图像的整体图像处理管道如图1所示。在伪四通道图像去噪中,首先将有噪声的CFA原始数据分解为四个子图像。这四个子图像可以看作是一个伪四通道图像。我们可以生成四种不同类型的伪四通道图像,即grbg通道图像、rggb通道图像、bggr通道图像和gbrg通道图像。每个伪四通道图像在变换后的域中去噪,如图2所示。对去噪后的四通道图像进行重新排列和平均,生成去噪后的CFA原始数据。平均处理可以有效地减少子图像分割的块结构造成的块伪影。最后,将现有的去马赛克算法应用到去噪后的CFA原始数据中,得到全彩色图像。
图2 四通道图像去噪
3.2. Four-channel image denoising with adaptive color transformation
Each pixel of the pseudo four-channel image like the GRBGchannel image has four-channel data, namely, G1, R, B, and G2. Although this data does not represent color, we call this data the color with analogy of the RGB-channel image. It is empirically known that the transformation of the color data into the principal component space helps denoising process because the signal energy is compact while the noise is distributed equally in all dimensions. Park et al heuristically proposed the transformation derived with off-the-shelf color transformation [10]. In this paper, we adaptively derive the transformation based on the PCA of the input noisy CFA raw data. For the PCA of the noisy data, we follow the same manner as in [9]. The covariance matrix of the noisy pseudo fourchannel image is calculated as
像GRBG通道图像这样的伪四通道图像的每个像素都有四通道数据,即G1、R、B、G2。虽然这些数据并不代表颜色,但是我们将这些数据称为与rgb通道图像类似的颜色。根据经验可知,将彩色数据转换到主成分空间有助于去噪,因为信号能量是紧凑的,而噪声在各个维度上分布均匀。Park等人提出了利用现成的颜色变换[10]推导出的变换。本文对输入的噪声CFA原始数据进行了基于主成分分析的自适应变换。对于噪声数据的PCA,我们采用与[9]相同的方式。将噪声伪四通道图像的协方差矩阵计算为
(5)
(6)
where xi is the four-channel data at i-th pixel, ni is the noise at i-th pixel, and T represents the transpose operator. Assuming that the signal and the noise are uncorrelated, we can estimate the covariance matrix of the signal as
其中xi为第i个像素处的四通道数据,ni为第i个像素处的噪声,T为转置算子。假设信号与噪声不相关,我们可以估计信号的协方差矩阵为
(7)
where diag(zi) represents the diagonal matrix composed of the elements of zi, and σk2 is the variance of the k-th channel
of the pseudo four-channel image. For the GRBG-channel image, . The transformation matrix P can be expressed with the eigen vectors vk of the covariance matrix Σ as
其中diag(zi)表示由zi的元素组成的对角矩阵,并且σk2是伪四通道图像的第k个通道的方差。对于GRBG通道图像,。变换矩阵P可以用协方差矩阵Σ的特征向量vk表示
(8)
We can transform the four-channel data into the principal component space by multiplying the transformation matrix.The variance of the noise in the k-th principal component can be estimated as
我们可以通过乘变换矩阵将四通道数据转换为主分量空间。第k个主分量的噪声方差可以估计为
(9)
Each transformed pseudo four-channel image is denoised by an existing high-performance gray-image denoiser with the estimated noise variance.
每个变换的伪四通道图像由具有估计的噪声方差的现有高性能灰度图像去噪器去噪。
图4 有无block reduction区别()
3.3 Block artifact reduction
A single pixel of the pseudo four-channel image corresponds to the four-pixel block in the CFA raw data. Therefore, it causes block artifacts in the demosaicked full-color image. In order to reduce the block artifacts, we average four different types of the pseudo four-channel images as shown in Fig. 3. By changing the start position of the four-pixel block in the CFA raw data, we can generate four different types of the pseudo four-channel images: the GRBG-channel image, the RGGB-channel image, the BGGR-channel image, and the GBRG-channel image. The four-pixel blocks of each pseudo four-channel image overlap each other in the CFA raw data. This is why the averaging process can reduce the block artifacts. Fig. 4 shows the effect of block artifact reduction.
伪四通道图像的单个像素对应于CFA原始数据中的四像素块。因此,在去马赛克的全彩色图像中,它会造成块伪影。为了减少块伪影,我们平均了四种不同类型的伪四通道图像,如图3所示。通过改变四像素块在CFA原始数据中的起始位置,可以生成四种不同类型的伪四通道图像:grbg通道图像、rggb通道图像、bggr通道图像和gpg通道图像。CFA原始数据中每个伪四通道图像的四像素块相互重叠。这就是为什么平均过程可以减少块工件。图4为块伪影约简效果。
4. EXPERIMENTAL RESULTS
In order to evaluate the proposed algorithm1, we conducted two types of experiments. First experiment is the comparison of the CFA raw data denoising performance without the demosaicking process. The Peak Signal-to-Noise Ratio (PSNR) between the denoised CFA raw data and the ground-truth data are evaluated.
为了对提出的算法进行评价,我们进行了两类实验。第一个实验是比较CFA原始数据的去噪性能,没有去马赛克过程。对去噪CFA原始数据与地面真值数据之间的峰值信噪比(PSNR)进行了评价。
We also compare the full-color image reconstruction performance from the noisy CFA raw data. For the quantitative comparisons, the color PSNR (CPSNR) between the fullcolor image reconstructed with the noisy CFA raw data and the ground-truth full-color image is calculated.
我们还比较了噪声CFA原始数据的全彩色图像重建性能。为进行定量比较,计算了用噪声CFA原始数据重建的彩色图像与地面真值彩色图像之间的彩色PSNR (CPSNR)。
表1 从Kodak数据集中去噪24 CFA原始数据的平均PSNR,其中粗体字表示每个噪声级别中的最佳性能。
We use high-resolution Kodak dataset which includes 24 2048x3072 sized images. In the proposed algorithm, we use the BM3D [16] for the gray image denoising and the residual interpolation [5] for the color demosaicking.
我们使用高分辨率柯达数据集,其中包括24个2048x3072大小的图像。在该算法中,我们使用BM3D[16]进行灰度图像去噪,使用残差插值[5]进行颜色去马赛克。
4.1. Comparisons of CFA raw data denoising
We compare the proposed algorithm with the direct BM3D [16], PCASAD [9] and Park’s algorithm [10]. The direct BM3D
denoises the CFA raw data, assuming that the CFA raw data is the gray image. The BM3D [16] is used for the gray image
denoising of the Park’s algorithm [10] as well as the proposed algorithm.
我们将提出的算法与直接BM3D [16],PCASAD [9]和Park算法[10]进行了比较。假设CFA原始数据是灰度图像,直接BM3D对CFA原始数据进行去噪。BM3D [16]用于Park算法的灰度图像去噪[10]以及所提出的算法。
Table 1 shows the numerical comparisons of CFA raw data denoising. This comparison demonstrates that the proposed algorithm outperforms other existing algorithms in almost all noise levels.
表1为CFA原始数据去噪的数值比较。结果表明,该算法在几乎所有噪声水平上都优于现有算法。
4.2. Comparisons of full-color image reconstruction
Here, we evaluate the performance of the full-color image reconstruction from the noisy CFA raw data. We compare the
proposed algorithm with existing algorithms; the combination of CFA denoising as mentioned above and demosaicking,
joint denoising and demosaicking including LPAICI [12], JDDTV [13], and LSLCD [14]. Comparison algorithms also include demosaicking only and simple denoising-afterdemosaicking method, namely, the straightforward combination of demosaicking and gray-scale denoiser.
在这里,我们评估了从噪声CFA原始数据重建全彩色图像的性能。我们将该算法与现有算法进行了比较;上述CFA去噪与去马赛克相结合,联合去噪与去马赛克包括LPAICI[12]、JDDTV[13]、LSLCD[14]。比较算法还包括单纯的去马赛克和简单的去马赛克后去噪方法,即去马赛克与灰度去噪的直接结合。
图5 视觉比较使用柯达高分辨率图像数据集()
表2 柯达数据集24幅图像的平均CPSNRs,其中粗体表示在每个噪声级别上的最佳性能。
Table 2 and Fig. 6 show the average CPSNRs of data set images. This comparison demonstrated that the proposed algorithm outperforms existing algorithms. A visual comparison is shown in Fig. 5. It shows that the proposed algorithm effectively suppresses color artifacts.
表2和图6为数据集图像的平均cpsnr。该算法的性能优于现有算法。如图5所示。结果表明,该算法有效地抑制了彩色伪影。
图6 不同噪声水平下CPSNRs的比较
5. CONCLUSION
We have proposed the simple and the effective CFA denoising algorithm. The adaptive color transformation and the block artifact reduction can effectively improve the performance. The experiments proved that the proposed algorithm can reconstruct full-color images, suppressing the noise in the CFA raw data.
提出了一种简单有效的CFA去噪算法。自适应颜色变换和块伪影约简可以有效地提高性能。实验证明,该算法能够对彩色图像进行重构,抑制CFA原始数据中的噪声。