项目实验记录

1 Kapture

1.1 实验环境及路径

1.1.1 位置/版本

kapture: /home/jty/vs/kapture
location: /home/jty/vs/kapture-localization

1.1.2 环境

conda activate vslam

1.2 如何运行

1.2.1 tutorial

1. 位置

/home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial

2. 运行指令

• 局部特征提取

参数含义： kapture_root 表示的是想要query的图像的base路径， top-k表示只选取分数最高的多少个特征

CUDA_VISIBLE_DEVICES=2 python3 r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt --kapture-root /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/mapping --top-k 20000 --max-size 9999

• mapping

python3 kapture-localization/pipeline/kapture_pipeline_mapping.py -v debug \
-i /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/mapping \
-kpt /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/keypoints \
-desc /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/descriptors \
-gfeat /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/global_features/AP-GeM-LM18/global_features \
-matches /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_no_gv/matches \
-matches-gv /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_colmap_gv/matches \
--colmap-map /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/colmap-sfm/r2d2_500/AP-GeM-LM18_top5  \
--topk 5

• location

python3 kapture-localization/pipeline/kapture_pipeline_localize.py -v debug \
      -i /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/mapping \
      --query /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/query \
      -kpt /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/keypoints \
      -desc /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/descriptors \
      -gfeat /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/global_features/AP-GeM-LM18/global_features \
      -matches /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_no_gv/matches \
      -matches-gv /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_colmap_gv/matches \
      --colmap-map /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/colmap-sfm/r2d2_500/AP-GeM-LM18_top5 \
      -o /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/colmap-localization/r2d2_500/AP-GeM-LM18_top5/AP-GeM-LM18_top5/ \
      --topk 5 \
      --config 2

• retial

python3 kapture-localization/pipeline/kapture_pipeline_image_retrieval_benchmark.py -v debug \
      -i /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/mapping \
      --query /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/query \
      -kpt /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/keypoints \
      -desc /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/descriptors \
      -gfeat /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/global_features/AP-GeM-LM18/global_features \
      -matches /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_no_gv/matches \
      -matches-gv /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/local_features/r2d2_500/NN_colmap_gv/matches \
      --colmap-map /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/colmap-sfm/r2d2_500/AP-GeM-LM18_top5 \
      -o /home/jty/vs/kapture-localization/samples/virtual_gallery_tutorial/image_retrieval_benchmark/r2d2_500/AP-GeM-LM18_top5/AP-GeM-LM18_top5/ \
      --topk 5 \
      --config 2

注意的点

• 注意特征feature文件夹名称需要和对应的keypoints.txt或者descriptor.txt文件。不然会报错‘dosent match的问题’
• 自己提取的local feature，需要按照格式手动写成对应的格式，参考turtoial文件夹

1.2.2 Aachen-Day-Night-v1.1

参考： https://github.com/naver/kapture-localization/blob/main/doc/benchmark.adoc

• 数据组织格式如下

Aachen-Day-Night-v1.1
├─ mapping
│  └─ sensors
│     ├─ sensors.txt
│     ├─ trajectories.txt  
│     ├─ records_camera.txt 
│     └─ records_data/   
├─ query
│  └─ sensors
│     ├─ sensors.txt  
│     ├─ trajectories.txt 
│     ├─ records_camera.txt
│     └─ records_data/
├─ local_features
│  ├─ r2d2_WASF-N8_20k
│  │  ├─ keypoints/
│  │  │  ├─ db
│  │  │  ├─ keypoints.txt   # "r2d2, float32, 3"
│  │  │  ├─ query
│  │  │  └─ sequences
│  │  ├─ descriptors/
│  │  │  ├─ db
│  │  │  ├─ descriptors.txt     # this one should contain the line "r2d2, float32, 128, r2d2_WASF-N8_20k, L2", name is important
│  │  │  ├─ query
│  │  │  └─ sequences
│  │  ├─ NN_no_gv
│  │  │  └─ matches/ # it will be filled by the scripts
│  │  └─ NN_colmap_gv/ 
│  │     └─ matches/# it will be filled by the scripts
├─ global_features
│  ├─  AP-GeM-LM18
│  │  ├─ global_features
│  │  │  ├─ db
│  │  │  ├─ global_features.txt     # "dirtorch, float32, 2048, L2"
│  │  │  ├─ query
│  │  │  ├─ sequences
└─ pairsfile
   └─ mapping
      └─ Aachen_Day_Night_1_1_ir_benchmark_pairs.txt

• local 特征提取

python3 r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt --kapture-root /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/map_plus_query  --top-k 20000  --max-size 9999

## move （这个地方可以不用动了，因为原来这个地方有问题，我修改过了）
mkdir -p Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_20k
mv Aachen-Day-Night-v1.1/map_plus_query/reconstruction/keypoints Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_20k/
mv Aachen-Day-Night-v1.1/map_plus_query/reconstruction/descriptors Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_20k/

• mapping

python3 kapture-localization/pipeline/kapture_pipeline_mapping.py -v debug \
    -i /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/mapping \
    -kpt /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/keypoints \
    -desc /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/descriptors \
    --pairsfile-path /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/pairsfile/mapping/Aachen_Day_Night_1_1_ir_benchmark_pairs.txt \
    -matches /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_no_gv/matches \
    -matches-gv /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_colmap_gv/matches \
    --colmap-map /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/colmap-sfm/r2d2_WASF_N8_big/ir_benchmark/colmap

• location

python3 kapture-localization/pipeline/kapture_pipeline_localize.py -v debug \
      -i /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/mapping \
      --query /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/query_day \
      -kpt /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/keypoints \
      -desc /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/descriptors \
      -gfeat /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/global_features/AP-GeM-LM18/global_features \
      -matches /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_no_gv/matches \
      -matches-gv /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_colmap_gv/matches \
      --colmap-map /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/colmap-sfm/r2d2_WASF_N8_big/AP-GeM-LM18_top5 \
      -o /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/colmap-localization/r2d2_WASF_N8_big/AP-GeM-LM18_top5/AP-GeM-LM18_top5/ \
      --topk 5 \
      --config 2

• retial

python3 kapture-localization/pipeline/kapture_pipeline_image_retrieval_benchmark.py -v debug \
      -i /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/mapping \
      --query /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/query_night \
      -kpt /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/keypoints \
      -desc /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/descriptors \
      -gfeat /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/global_features/AP-GeM-LM18/global_features \
      -matches /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_no_gv/matches \
      -matches-gv /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/local_features/r2d2_WASF_N8_big/NN_colmap_gv/matches \
      --colmap-map /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/colmap-sfm/r2d2_WASF_N8_big/ir_benchmark/colmap \
      -o /home/jty/vs/kapture-localization/kapture_datasets/Aachen-Day-Night-v1.1/image_retrieval_benchmark/r2d2_WASF_N8_big/ir_benchmark/AP-GeM-LM18_top20 \
      --topk 20 \
      --config 2

2 Hierarchical-Localization

aachen

code

可以直接使用

/home/jty/geo/Hierarchical-Localization/tools/test.py

运行代码

python3 /home/jty/geo/Hierarchical-Localization/tools/test.py

• 设置

dataset = Path('datasets/aachen/')  # change this if your dataset is somewhere else
images = dataset / 'images/images_upright/'

pairs = Path('pairs/aachen/')
sfm_pairs = pairs / 'pairs-db-covis20.txt'  # top 20 most covisible in SIFT model
loc_pairs = pairs / 'pairs-query-netvlad50.txt'  # top 50 retrieved by NetVLAD

outputs = Path('outputs/aachen/')  # where everything will be saved
reference_sfm = outputs / 'sfm_superpoint+superglue'  # the SfM model we will build
results = outputs / 'Aachen_hloc_superpoint+superglue_netvlad50.txt'  # the result file

print(f'Configs for feature extractors:\n{pformat(extract_features.confs)}')
print(f'Configs for feature matchers:\n{pformat(match_features.confs)}')

feature_conf = extract_features.confs['superpoint_aachen']
matcher_conf = match_features.confs['superglue']

• 提取特征

extract_features.main(feature_conf, images, outputs)

• sfm重建

colmap_from_nvm.main(
    dataset / '3D-models/aachen_cvpr2018_db.nvm',
    dataset / '3D-models/database_intrinsics.txt',
    dataset / 'aachen.db',
    outputs / 'sfm_sift')

pairs_from_covisibility.main(
    outputs / 'sfm_sift', sfm_pairs, num_matched=20)

• 匹配

match_features.main(matcher_conf, sfm_pairs, feature_conf['output'], outputs)

• Triangulate a new SfM model from the given poses

colmap_from_nvm.main(
    dataset / '3D-models/aachen_cvpr2018_db.nvm',
    dataset / '3D-models/database_intrinsics.txt',
    dataset / 'aachen.db',
    outputs / 'sfm_empty',
    skip_points=True)

triangulation.main(
    reference_sfm,
    outputs / 'sfm_empty',
    images,
    sfm_pairs,
    outputs / f"{feature_conf['output']}.h5",
    outputs / f"{feature_conf['output']}_{matcher_conf['output']}_{sfm_pairs.stem}.h5",
    colmap_path='colmap')  # change if COLMAP is not in your PATH

• Match the query images

match_features.main(matcher_conf, loc_pairs, feature_conf['output'], outputs)

• Localize

localize_sfm.main(
    reference_sfm / 'model',
    dataset / 'queries/*_time_queries_with_intrinsics.txt',
    loc_pairs,
    outputs / f"{feature_conf['output']}.h5",
    outputs / f"{feature_conf['output']}_{matcher_conf['output']}_{loc_pairs.stem}.h5",
    results,
    covisibility_clustering=False)  # not required with SuperPoint+SuperGlue