HPatches数据集(图像匹配)---1整体理解

最新推荐文章于 2024-08-08 07:58:59 发布

知识在于分享

最新推荐文章于 2024-08-08 07:58:59 发布

阅读量8.1k

点赞数 10

分类专栏： SLAM+SFM

本文链接：https://blog.csdn.net/baidu_40840693/article/details/109495969

版权

SLAM+SFM 专栏收录该内容

187 篇文章 42 订阅

订阅专栏

我们前面写的:

PhotoTour项目数据集Liberty NotreDame Yosemite(图像匹配)

https://blog.csdn.net/baidu_40840693/article/details/109491599

第一个流行的基于补丁的数据集是PhotoTourism. 自推出以来，使用补丁进行基准测试的许多好处变得明显.PhotoTourism引入了一个简单而明确的评估协议，我们将其称为补丁验证：给定一对补丁，任务是预测它们是否匹配，这将匹配任务减少为二进制分类问题。该公式特别适用于基于学习的方法，包括CNN和度量学习，特别是由于该数据集中可用的大量补丁。 PhotoTourism的主要限制是其稀缺的数据多样性（只有三个场景：Liberty，Notre-Dame和Yosemite），任务多样性（只有补丁验证任务）和特征类型多样性（仅提取DoG特征）。

而HPatches侧重于对局部描述子的评估,给出单应矩阵真值，反验证匹配的正确性，以此得出召回率和准确性

参考:

https://blog.csdn.net/honyniu/article/details/87923219

https://www.cnblogs.com/Alliswell-WP/p/TranslationOfPapers_Review002.html

https://blog.csdn.net/qq_21685903/article/details/103476716

今天我们介绍HPatches数据集:

https://github.com/hpatches/hpatches-benchmark

https://github.com/hpatches/hpatches-dataset

https://github.com/hpatches/hpatches-descriptors

通过以上你可以下载的文件有,hpatches-benchmark评估等代码文件

hpatches-sequences-release文件夹

i_代表light光照变化 i_代表viewpoint视角变化

每一个子文件,是一个图片序列,像Mikolajczyk数据集一样,含有H矩阵

光照的H矩阵不变:

如果是视角变换的H矩阵:

hpatches-release文件

116个子文件夹(和hpatches-sequences-release一样),代表116个图像序列提取的patches

这里面可以观察到以下几个文件:jitter(抖动)

假设图像序列:

第一张为reference image参考图像(hpatches-sequences-release/v_bird/1.ppm),而从它提取的patches为reference patches(ref.png),

代表该图像序列中1.ppm图像共提取了1272个65*65的patches

还有e1.png,e2.png,e3.png,e4.png,e5.png

h1.png,h2.png,h3.png,h4.png,h5.png

t1.png,t2.png,t3.png,t4.png,t5.png

三种不同程度几何噪声的patches匹配对(这里我们不用去做匹配,因为我们有ground truth homography),1-5代表他们分别从(hpatches-sequences-release/v_bird/2.ppm-6.ppm)中提取

e代表:easy(少量的几何噪声), h代表hard(较多的几何噪声),t代表tough(更多的几何噪声)

具体大致程度如下(仿射变换):

对应以下文件:jitter(抖动)

.rot----jitter .tr----jitter .sc----jitter .ani----sjitter

其实就是图像序列每个图像中匹配的patches添加了具体几何噪声的文件,虽然ref也有这些文件,但是值都为0,因为参考图像本身不需要添加几何噪声

那么现在还有一个文件没有解释:

e1.overlaps-e5.overlaps h1.overlaps-h5.overlaps t1.overlaps-t5.overlaps

这个文件是patches提取过程中产生的

使用 (Hessian, Harris and DoG detector)进行特征点提取,使用SIFT的角度计算方法,但是没有进行仿射变换,所以提取的都是正方形patches

类似SIFT特征点检测,在原始检测的scale范围上,扩大5倍进行patches提取,当然也要做边界检测,扩大后一定要在图像内才提取patches

为了防止在同一位置多次检测,椭圆重叠率大于50%只保留一个随机椭圆(Harris检测是椭圆,DoG应该是圆)

图中橙色是原始scale.黄色是扩大5倍后的scale

在提取patches,要使用前面说的jitter(抖动) 用于加入几何噪声

目的:The goal of the affine jitter is to simulate the geometric repeatability error of typical local features detector. 局部特征检测器会在特征点匹配时候发生误差(虽然我们没有进行匹配得到patches,但是要模拟这个匹配误差)

对于e(easy)程度,抖动后椭圆与原始的椭圆的重叠度(overlap)在0.85附近

对于h(hard)程度,抖动后椭圆与原始的椭圆的重叠度(overlap)在0.72附近

加入jitter(几何噪声)后,相当于进行了仿射变换

然后使用ground truth的H矩阵,将提取的特征点投影到目标图像中(2.ppm-6.ppm),这样就获得了特征点匹配对集合,此时各自提取各自的patches

我们先再看下ref提取的特征点:

然后e(easy)程度下, 在目标图像中(2.ppm-6.ppm)通过H矩阵投影的特征点:

然后h(hard)程度下, 在目标图像中(2.ppm-6.ppm)通过H矩阵投影的特征点:

此时各自再通过我们之前说的,特征点scale扩大5倍,考虑到边界问题,直接提取正方形patches(65*65)

e(easy)程度下:

h(hard)程度下:

那么这些匹配的patches,他们的重叠率就记录在以下文件中:

e1.overlaps-e5.overlaps h1.overlaps-h5.overlaps t1.overlaps-t5.overlaps

descriptors文件

每一个csv都是N行D列的文件,N是patches数目,D是该算法的描述符维度

关于怎么提取某个算法对于每个patches的特征描述符脚本,已经提供,这里的例子是提取patches的均值和方差,当成描述符保存

https://github.com/hpatches/hpatches-descriptors/blob/master/python/hpatches_extract.py

import sys
import glob
import os
import cv2
import numpy as np

assert len(sys.argv)==2, "Usage python hpatches_extract.py hpatches_db_root_folder"
    
# all types of patches 
tps = ['ref','e1','e2','e3','e4','e5','h1','h2','h3','h4','h5',\
       't1','t2','t3','t4','t5']

class hpatches_sequence:
    """Class for loading an HPatches sequence from a sequence folder"""
    itr = tps
    def __init__(self,base):
        name = base.split('/')
        self.name = name[-1]
        self.base = base
        for t in self.itr:
            im_path = os.path.join(base, t+'.png')
            im = cv2.imread(im_path,0)
            self.N = im.shape[0]/65
            setattr(self, t, np.split(im, self.N))
            
    
seqs = glob.glob(sys.argv[1]+'/*')
seqs = [os.path.abspath(p) for p in seqs]     

descr_name = 'misigma'

for seq_path in seqs:
    seq = hpatches_sequence(seq_path)
    path = os.path.join(descr_name,seq.name)
    if not os.path.exists(path):
        os.makedirs(path)
    descr = np.zeros((seq.N,2)) # trivial (mi,sigma) descriptor
    for tp in tps:
        print(seq.name+'/'+tp)
        for i,patch in enumerate(getattr(seq, tp)):
            mi = np.mean(patch)
            sigma = np.std(patch)
            descr[i] = np.array([mi,sigma])
        np.savetxt(os.path.join(path,tp+'.csv'), descr, delimiter=',')   # X is an array

介绍完数据集的文件,该数据集的评价指标和三个测试任务(补丁验证，图像匹配和补丁检索)

主要参考

https://github.com/hpatches/hpatches-benchmark/blob/master/docs/tasks.md

https://blog.csdn.net/honyniu/article/details/87923219

第一个,衡量描述符对于 patches的区分程度

第二个,衡量描述符对于图像匹配

第三个,衡量描述符对于超大集合的patches的检索程度

上述任务,涉及到数据集拆分:

参考:

https://github.com/hpatches/hpatches-benchmark/blob/master/docs/splits.md

https://github.com/hpatches/hpatches-benchmark/tree/master/tasks

关于怎么进行分割,使用脚本:(splits.json)

https://github.com/hpatches/hpatches-benchmark/tree/master/tasks/splits

from glob import glob
import random
import json

# read the available sequence names
f = glob('../hpatch-release/*')
f = [k.split("/")[-1] for k in f]
i = [k for k in f if 'i_' in k]
v = [k for k in f if 'v_' in k]

# rand seeds for reproducibility
seeds = [40,41,42]
names = ['a','b','c']
# init splits dict
spl = {}

# save 3 sample train-test splits. In each split 40 sequences are kept
# for testing, and the remaining can be used for training.
for k,s in enumerate(seeds):
    random.seed(s)
    i_sam = random.sample(i,20)
    v_sam = random.sample(v,20)
    spl[names[k]] = {}
    spl[names[k]]['name'] = names[k]
    spl[names[k]]['test'] = i_sam + v_sam
    spl[names[k]]['train'] = list(set(i+v) - set(spl[names[k]]['test']))

# save the full test split
spl['full'] = {}
spl['full']['test'] = i+v
spl['full']['name'] = 'full'

# save the illumination test split
spl['illum'] = {}
spl['illum']['test'] = i
spl['illum']['name'] = 'illum'

# save the viewpoint test split
spl['view'] = {}
spl['view']['test'] = v
spl['view']['name'] = 'view'

with open('splits.json', 'w') as f:
    json.dump(spl, f,ensure_ascii=True)