deflicker（debanding） 笔记（一）：Blind Video Deflickering by Neural Filtering with a Flawed Atlas （翻译与理解）

写在前面

最近一段时间在看 deflicker（banding） 的相关，目前对 2023 年的文章《Blind Video Deflickering by Neural Filtering with a Flawed Atlas》比较中意，这里在翻译的同时写一些相关的理解，有错误或者不对的望指正！

翻译与理解

Abstract：
Many videos contain flickering artifacts; common causes of flicker include video processing algorithms, video gener-
ation algorithms, and capturing videos under specific situations. Prior work usually requires specific guidance such
as the flickering frequency, manual annotations, or extra consistent videos to remove the flicker. In this work, we
propose a general flicker removal framework that only receives a single flickering video as input without additional
guidance. Since it is blind to a specific flickering type or guidance, we name this “blind deflickering.” The core of
our approach is utilizing the neural atlas in cooperation with a neural filtering strategy. The neural atlas is a unified representation for all frames in a video that provides temporal consistency guidance but is flawed in many cases.
To this end, a neural network is trained to mimic a filter to learn the consistent features (e.g., color, brightness) and
avoid introducing the artifacts in the atlas. To validate our method, we construct a dataset that contains diverse real-
world flickering videos. Extensive experiments show that our method achieves satisfying deflickering performance
and even outperforms baselines that use extra guidance on a public benchmark. The source code is publicly available at
https://chenyanglei.github.io/deflicker.
翻译与理解：
许多视频包含了isp 处理中常见的 flicker（频闪）。造成flicker的原因有：
1 视频处理算法（video processing algorithms）ps： 后面作者说的processed 视频都指的是算法处理的这种
2 视频生成算法（video generation algorithms）
3 特殊拍摄的视频（capturing videos under specific situations）
先前的工作通常要求一些明确的导向信息，例如flicker的频率，人工标注 或者 额外的连续的视频进行辅助。
在本文，我们提出了一种通用的flicker去除框架，并且该框架之接受一个带有flicker的是in作为输入，不需要任何其他的辅助。 因为这个框架不需要特殊的flicker 类型的先验信息，因此我们命名“盲导 deflicker”。我们方法的核心就是利用 “网络图谱” （neural atlas）和 网络滤波 联合的测策略。 “网络图谱” （neural atlas） 是视频中所有帧的统一概括（unified representation）且这个 neural atlas 虽然可以在时域上提供连惯性指导（consistency guidance）但是对于很多情况是有缺陷的。到这里，神经网络会用来模仿（mimic）和学习视频的连惯性特征（例如颜色和亮度）避免在atlas中引入伪影。
为了验证我们的方法，我们构建了一个包含各种真实世界闪烁视频的数据集。大量实验证明，我们的方法实现了令人满意的去闪效果，甚至在公共基准测试中超过了使用额外指导的基线方法。源代码可以在https://chenyanglei.github.io/deflicker上公开获取。
————————————————————————————————————————————————————————

1. Introductio（导论）
A high-quality video is usually temporally consistent, but many videos suffer from flickering for various reasons, as shown in Figure 2. For example, the brightness of old movies can be very unstable since some old cameras cannot set the exposure time of each frame to be the same with low-quality hardware [16]. Besides, indoor lighting changes with a certain frequency (e.g., 60 Hz) and highspeed cameras with very short exposure time can capture the rapid changes of indoor lighting [25]. Effective processing algorithms such as enhancement [38, 45], colorization [29, 60], and style transfer [33] might bring flickering artifacts when applied to temporally consistent videos.Video from video generations approaches [46, 50, 61] also might contain flickering artifacts. Since temporally consistent videos are generally more visually pleasing, removing the flicker from videos is highly desirable in video processing [9, 13, 14, 54, 59] and computational photography. In this work, we are interested in a general approach for deflickering: (1) it is agnostic to the patterns or levels of flickering for various reasons (e.g., old movies, highspeed cameras, processing artifacts), (2) it only takes a single flickering video and does not require other guidances (e.g., flickering types, extra consistent videos). That is to say, this model is blind to flickering types and guidance, and we name this task as blind deflickering. Thanks to the blind property, blind deflickering has very wide applications.
翻译与理解
高质量的视频在时间上是一致的，但由于各种原因，许多视频会出现闪烁问题，如图2所示。例如，由于一些老旧相机无法将每帧的曝光时间设置为相同且硬件质量低劣，因此老电影的亮度可能非常不稳定[16]。此外，室内照明以一定频率变化（例如60 Hz），而高速相机使用非常短的曝光时间可以捕捉到室内照明的快速变化[25]。当应用于时间上一致的视频时，有效的处理算法（例如增强[38, 45]、上色[29, 60]和风格转换[33]）可能会引入闪烁伪影。视频生成方法（[46, 50, 61]）生成的视频也可能包含闪烁伪影。由于时域上连贯性好的视频通常更具视觉吸引力，在视频处理[9, 13, 14, 54, 59]和计算摄影中体现出去除视频中的频闪在业内的强烈需求。在这项工作中，我们对于去闪的一般方法感兴趣：（1）它不依赖于各种原因导致的闪烁模式或级别（例如老电影、高速相机、处理伪影），（2）它只需要一个闪烁视频，不需要其他的指导（例如闪烁类型、额外一致的视频）。也就是说，这个模型对于闪烁类型和指导是无感的，我们将这个任务称为“盲导去频闪”（blind deflicker）。由于其盲目特性，盲目去频闪具有广泛的应用。
————————————————————————————————————————————————————————

Blind deflickering is very challenging since it is hard to enforce temporal consistency across the whole video without any extra guidance. Existing techniques usually design specific strategies for each flickering type with specific knowledge. For example, for slow-motion videos captured by high-speed cameras, prior work [25] can analyze the lighting frequency. For videos processed by image processing algorithms, blind video temporal consistency [31, 32] obtains long-term consistency by training on a temporally consistent unprocessed video. However, the flickering types or unprocessed videos are not always available, and existing flickering-specific algorithms cannot be applied in this case. One intuitive solution is to use the optical flow to track the correspondences. However, the optical flow from the flickering videos is not accurate, and the accumulated errors of
optical flow are also increasing with the number of frames due to inaccurate estimation [7].
翻译与理解
“盲导去flicker”非常具有挑战性，因为在没有任何额外指导的情况下，很难实现整个视频的时域上的连贯性。现有的技术通常针对每种flicker类型设计特定的策略，并需要具备特定的先验知识。例如，对于高速相机捕捉的慢动作视频，之前的工作[25]可以分析照明频率。对于经过图像处理算法处理的视频，为保持未知视频（blind video）时域上的连贯性，在[31, 32]中的方法通过在时间上一致的未处理视频上进行训练来获得长期连贯性。然而，flicker类型或未处理视频并不总是可用的，而且现有的针对特定flicker的算法无法在这种情况下应用。一种直观的解决方案是使用光流来跟踪对应关系。然而，闪烁视频中的光流并不准确，并且由于估计不准确，光流的累积误差也随着帧数的增加而增加[7]。
————————————————————————————————————————————————————————

With two key observations and designs, we successfully propose the first approach for the new problem blind deflickering that can be used to remove various types of flickering artifacts without extra guidance or prior knowledge of flickering.
First, we utilize a unified video representation named neural atlas [26] to solve the major challenge of solving long-term inconsistency. This neural atlas tracks all pixels in the video, and correspondences in different frames share the same pixel in this atlas. Hence, a sequence of consistent frames can be obtained by sampling from the s