红外可见光融合数据集：MSRS

MSRS: Multi-Spectral Road Scenarios for Practical Infrared and Visible Image Fusion

1️⃣ 资料

1. 说明：这个数据集中加入了语义分割的标签，可以很好的提供带下游任务的融合网络的数据集。
2. 官方 github：https://github.com/Linfeng-Tang/MSRS
3. 官方讲解（论文中的一部分）：MFNet: Towards Real-Time Semantic Segmentation for Autonomous Vehicles with Multi-Spectral Scenes

2️⃣ 资料翻译

Github

https://github.com/Linfeng-Tang/MSRS

MSRS: Multi-Spectral Road Scenarios for Practical Infrared and Visible Image Fusion
MSRS：用于红外和可见光图像融合的多光谱道路场景

We construct a new multi-spectral dataset for infrared and visible image fusion based on the MFNet dataset. The MFNet dataset contains 1,569 image pairs (820 taken at the daytime and 749 taken at nighttime) with spatial resolution is 480 × 640. However, There are many misaligned image pairs in the MFNet dataset and most infrared images are low signal-to-noise and low contrast. To this end, we first collect 715 daytime image pairs and 729 nighttime image pairs via removing 125 misaligned image pairs. Moreover, an image enhancement algorithm based on dark channel prior is leveraged to optimize the contrast and signal-to-noise of infrared images. As a result, the released new Multi-Spectral Road Scenarios (MSRS) dataset contains 1,444 pairs of aligned infrared and visible images with high quality.
我们构建了一个新的多光谱数据集的基础上，红外和可见光图像融合的MFNet数据集。MFNet数据集包含1569个图像对（820个在白天拍摄，749个在夜间拍摄），空间分辨率为480 × 640。然而，在MFNet数据集中存在许多未对齐的图像对，并且大多数红外图像是低信噪比和低对比度的。为此，我们首先收集715白天图像对和729夜间图像对通过删除125错位图像对。此外，基于暗通道先验的图像增强算法被用来优化红外图像的对比度和信噪比。因此，新发布的多光谱道路场景（MSRS）数据集包含1，444对高质量的对齐红外和可见光图像。

You could run visualize.py to visualize segmentation labels
您可以运行visualize.py来可视化分割标签

Some typical examples are shown below(the left column are Vis images and the right column are IR images):
下面是一些典型的例子（左列是维斯图像，右列是红外图像）：


后边图片省略

MFNet 中对数据集的描述

We used an InfRec R500 as our RGB and IR camera. This camera can capture images in both the visible and thermal infrared spectrum (814μm) with different lenses and sensors. For convenience, we henceforth use the term color camera to indicate the visible spectral part of the R500 and thermal camera to indicate the thermal infrared spectral part of the R500. Both the color and thermal camera have 480×640 pixels of spatial resolution.
我们使用InfRec R500作为我们的RGB和IR相机。该相机可以通过不同的镜头和传感器捕获可见光和热红外光谱（814μm）的图像。为方便起见，我们此后使用术语彩色相机来表示R500的可见光谱部分，使用热像仪来表示R50的热红外光谱部分。彩色和热像仪的空间分辨率均为480×640像素。

The color camera has a 100◦ horizontal field of view, whereas the thermal camera has a 32◦ horizontal field of view. Because the thermal camera has a smaller field of view compared to the color camera, we cropped a fixed area from each of the RGB images and resized them to 480×640, before combining them with the corresponding IR images.
彩色摄像机具有100◦的水平视场，而热像仪具有32 \9702》的水平视场。由于热成像相机的视场比彩色相机小，我们从每个RGB图像中裁剪出一个固定区域，并将其大小调整为480×640，然后将其与相应的IR图像组合。

Fig. 4: Percentage of the number of pixels in each class of our dataset.
图4：我们数据集每类像素数的百分比。（注意，这个是 MDNet 的，不是 MSRS 的）

As far as we know, our dataset is the first RGB-Thermal urban scene image dataset with pixel-level annotation. We published this new RGB-Thermal semantic segmentation dataset in support of further development of autonomous vehicles in the future. This dataset contains 1569 images (820 taken at daytime and 749 taken at nighttime). Eight classes of obstacles commonly encountered during driving (car, person, bike, curve, car stop, guardrail, color cone, and bump) are labeled in this dataset.
据我们所知，我们的数据集是第一个具有像素级注释的RGB Thermal城市场景图像数据集。我们发布了这个新的RGB Thermal语义分割数据集，以支持未来自动驾驶汽车的进一步发展。该数据集包含1569张图像（820张白天拍摄，749张夜间拍摄）。该数据集中标记了驾驶过程中常见的八类障碍物（汽车、人、自行车、弯道、停车位、护栏、色锥和颠簸）。

Because only a few categories of each entire scene are labeled in our dataset, unlabeled pixels occupied the majority of all pixels (much sparser than VOC2012 [30]) and the number of pixels is extremely unbalanced between each class, similar to any other image segmentation dataset. The percentage of the number of pixels in each class is showed in Figure 4.
由于我们的数据集中只标记了整个场景的少数类别，因此未标记的像素占据了所有像素的大部分（比VOC2012[30]少得多），并且每个类别之间的像素数量极其不平衡，类似于任何其他图像分割数据集。图4显示了每个类中像素数的百分比。

We divided our dataset into three parts: the training set, validation set and test set. The training set includes 50% of the daytime images and 50% of the nighttime images, whereas the validation and test sets contain 25% of the daytime images and 25% of the nighttime images, respectively. The dataset was separated into three parts according to the time series when the images were captured, without balancing the frequency of each class in each set of the data manually. Therefore, the difference in distribution between each part of the dataset introduces additional challenges to the segmentation algorithm.
我们将数据集分为三个部分：训练集、验证集和测试集。训练集包括50%的白天图像和50%的夜间图像，而验证集和测试集分别包含25%的白天图像以及25%的夜间图像。根据捕获图像时的时间序列，数据集被分为三个部分，而没有手动平衡每组数据中每个类别的频率。因此，数据集各部分之间的分布差异给分割算法带来了额外的挑战。

融合+语义分割 -> 大概样子

3️⃣ 我的笔记

1. MSRS 是 MFNet 的改进版，删除了未对齐的内容。
2. MSRS的目录结构如下

   (base) kimshan@MacBook-Pro MSRS-main % ls
   README.md	detection	test		train		visualize.py
   

   train、test 和 detection 这些目录分别对应于训练集、测试集和验证集。