论文阅读：Optical Flow in the Dark

1. 论文总述

从论文题目中可以看出，本文主要是想改善暗光下的光流估计效果，先前的方法主要是将暗光下的image进行提亮，再估计光流，但作者认为这样效果不好，因为提亮时候会破坏帧与帧之间的运动信息。

作者的方法是：合成数据， 作者估计暗光下的噪声特性（利用的先前的经典的噪声模型，泊松分布+高斯分布，N(x) = af(x)+b），然后将暗光特性加入到开源的光流数据集FlyingChairs，利用这样的合成数据集训练深度模型FlowNet 或者PWC-Net，然后在作者自己收集的raw格式的不同曝光的数据集上进行验证暗光下的效果，发现这样的合成数据集对预测暗光下的光流效果不错。。

paper源代码: 链接

由源代码可以知道，本文主要是贡献了两个数据集：（1）作者自己利用设备收集的大概600对images，raw格式的，不同曝光的光流数据集VBOF，注意：这个数据集的光流GT是作者利用FlowNet2光流算法估计得到的（这个光流GT并不保真。。）（2）FCDN，在FlyingChairs加暗光下的噪声特性和awb特性，简单的两个操作就合成了一个数据集，然后就在这个合成数据集上训练就可（后续可以验证一下这个思路是否真的有效）

（1）We collect a new optical flow dataset — Various
Brightness Optical Flow (VBOF) dataset, consisting of
598 raw images of various brightness and 100 optical
flow references to evaluate the ability of an optical flow
model to deal with various exposure inputs.

（2） We provide an analysis of the noise on raw images of
various brightness and reveal how the noise distribution changes as the exposure descends.

（3） We apply our noise model to synthesize a low-light optical flow dataset — FlyingChairs-DarkNoise (FCDN), and the optical flow models
trained on our dataset can relatively maintain optical flow accuracy
as the exposure descends and they outperform the existing methods
greatly on low-light images.

也重要：

We focus on simulating the noise on raw images, which
avoids complex characteristics due to the image processing
pipeline. Most existing noise analysis [2] is conducted on
bright noisy images [24] in which the noise is caused by
high ISO. Beside that, we mainly focus on the noise caused
by the low intensity of light, since the photons arriving at
the sensor are governed by Poisson distribution. And fi-
nally, we apply our noise model to create new optical flow
datasets for training. In order to test a model’s ability to
deal with various exposure inputs, we collect a new optical
flow dataset in low-light environments.

2. 以前的暗光下光流估计的方案

主要是：先提亮再光流估计，但这样效果不好

A direct solution to get optical flow from dark images
is to enhance them before computing optical flow. Researchers have proposed many techniques for denoising and
enhancing low-light images [9, 3, 6, 20]. However, the purpose of these techniques is to get better visual quality of the
enhanced images, and we get limited improvement on optical flow results or even worse ones. The main reason is
that many of the methods lose information while enhancing, such as brightness constancy which is fundamental for
optical flow estimation. As a result, we choose to make the
network learn optical flow directly from low-light images in
an end-to-end way that avoids information loss.

3. 暗光下image的处理

为了能看见暗光下的image，如今有这么几种方法：

Low-light Imaging. A variety of techniques have been
developed for low-light imaging. Infrared information is
commonly used in low-light detection [21, 20], but infrared
sensors are not widely equipped, so our experiments focus
on RGB image enhancing. Non-local means (NLM) [3] and
BM3D [9] are denoising methods based on finding similar patches around every pixel, then average all the patches
and replace the pixel with the result. Recently, Chen et al.
[6, 5] propose a learning-based method to enhance dark images. They take raw data as input to a modified U-Net [25].
However, even if some of the methods could achieve good
perception, it’s unavoidable to lose information while postprocessing and information like brightness consistency is
critical to optical flow estimation.

（1）降噪（2）基于CNN去学习提亮Low-light Image

4. VBDF的images及光流GT获取

VBOF数据集的优势：

Besides the capability to evaluate the brightness robustness of an optical flow model, VBOF also has other advantages. First, we choose to present our data in 14-bit raw
format so that we can use various methods to demosaic and
enhance images with less information loss. Second, since
the VBOF dataset is collected using 3 cameras from different manufacturers and raw format is unique to every camera
model, our dataset is able to evaluate the generalization ability for different cameras. Third, the VBOF dataset consists
of scenes both indoor and outdoor, with various lighting effects, so it comprehensively reflects scenes in real life.

5. 是否能合成一个raw训练集用来暗光下的光流估计

Training Set Synthesis. We want to use raw images as
input to avoid information loss when testing, so we need to
generate raw features on RGB images in the original training dataset, which is the core of our solution. Brooks et
al. [2] propose a method to “unprocess” RGB images to
RAW for denoising purpose, which includes adding noise,
inverting tone mapping, gamma decompression, inverting
white balance and digital gain, etc. We try to synthesize
a “raw” FlyingChairs dataset using their method but after
training and evaluating, we find the final optical flow result
turns out to be inaccurate compared to our solution (See
Figure 5). We think the main reason is that we analyze the
noise and synthesize it based on multiple datasets covering
a large range of exposure and ISO, while their analysis is
on the DND dataset [24] which is relatively limited (See (d)
in Figure 4) and other operations like compression tend to
destroy signal in 8-bit images.

文中说，效果并不好，有待验证

6. SID介绍及其缺点

We present the result of Learning to See in the Dark
(SID) [6] as a representative of learning-based enhancing
methods. Chen et al. [6] propose to use U-Net [25] to enhance dark raw images, and they provide their model trained
on Sony raw images, so we also use the Sony part of our
VBOF dataset to test the method, and we get visually great
images using the provided model. (See Figure 6,7) SID
does a good job on white balance and the result images are
noiseless, but if zooming in to take a closer look, one will
find the signal in the image is unstable especially for the
extremely dark images. Followed by an optical flow model
trained on the existing FlyingChairs dataset [10], SID also
doesn’t solve the problem of optical flow in the dark. Sometimes SID may lead to worse optical flow results showed in
Figure 7.

注：SID是在raw images上训练的

参考文献

1. CVPR2020 Optical flow in the dark
2. https://github.com/mf-zhang/Optical-Flow-in-the-Dark