【论文阅读】15-Real-Time Visibility-Based Fusion of Depth Maps
0 basic information
Merrell P , Akbarzadeh A , Wang L , et al. Real-Time Visibility-Based Fusion of Depth Maps[C]// IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007. IEEE, 2007.
1 aims and characteristic
- 解决depth map fusion问题:
based on visibility constraint
—2 method- 作用: depth accurate , data not redundancy
- Two stage – depth estimate & depth fusion
- 可单独进行(并行化),速度提升(GPU)
- 处理video – real-time
2 method
2.1 depth map estimate
2.1.1 depth 计算–plane-sweeping stereo
-
Referring image — neighboring images N
-
Divide into 2 group N_previous & N_follow
-
Each pixel in I_refer:
1)多个假想Plane ---多个假想深度值
2) 对于每一个假想平面,计算cost, 选择lowest cost 作为深度值
Cost计算:
N_previous每张图像 :
像素p 根据plane—homography — 对应像素p’ 在Square windows内—计算
absolute difference of intensity: c_pre_i
Cost-previous = sum(c_pre_i)
同理,
Cost-following = sum(c_fol_i)
当前plane对应的
depth-cost-plane = min ( Cost-previous ,Cost-following)
2.1.1 depth confidence计算
- 假设Cost(x,d)的噪声服从高斯分布(noise strength -sigma)
计算confidence的意义:衡量真正depth 所对应的cost不是所有cost中的最小值的可能性(负相关关系)
- 公式:
PS :
This equation produces ahigh confidence
when the cost has asingle sharp minimum
. The confidence islow
when the cost has ashallow minimum or several low minima
.
2.2 depth map fusion
- other views — center view ( referring view)
STEP1 ( common)
: depth map redering
other view + other depth map----- 3D point X -----projection ----center view depth map2 -D
- 几个概念:
R: distance measurement
D: camera project to form the depth map
hypothesis ( R& D 一般不同,存在误差)
几种visibility conflict:
Free-space : center 2( rendering) other A < A’ — close
Occlusion : other 2(rendering) center C’< C ---- far away
2.2.1 stability fusion
- 实质:穷举所有的occlusion + free-space 迭代得到fused depth
- 步骤:
Step2–Initial :
center depth = closest depth( D_refer & D_other_2_refer)Step3–计算stability:
公式 = the number of views (
occlusion ) - the number of views (free-space )
含义:衡量当前depth close
or far away
stability value :
0-- suitbale
正-- away
负 – closeStep4–调整depth:
move to the next closest,使得Stability 朝着0变化
Final/fused depth: 使得Stability-非负的最小depthStep5-- add confidence
The confidence of fused depth : add the confidence of the pixel in other views which supporting the fused depth
supporting 的判断标砖: Di vs fused depth 阈值
2.2.2 confidence fusion
- Based on confidence,depth先合并combine再 test
- 步骤
Step2 --Initial: the
highest confidence
–depth0Step3 —迭代combine (公式如下):
depth
:与depth0 closest( 阈值内)的depth1,二者加权(confidence)融合 PS :记录depth 迭代combine的次数K—the level
of supportingConfidence
: 采用相加操作
Step4:验证fused depth 的有效性-- Conflict Detection
Total supporting
: == k (上文) > C_threshold 不满足则舍弃该值Visibility constraints
:更新 confidence–计算公式如下( out of the supporting region – free -space &
occlusion )
判断最终的confidence ( 正 -keep/ 负- remove)
Step5 fused depth map 后处理 –hole
w*w window
- inlier depth( enough number ,否则,空洞保留)-中值滤波
w_smaller* w_smaller
—中值滤波—smooth
3 restruction surface
3.1 overlap depth
- 处理多张fused depth map overlap部分不统一问题
- 方法:
每一张fused map在生成最终的depth maps时,overlap区域的像素计算得到的new depth与前一张fused depth map对应值比较,若存在 free-space ,则reject;
反之二者merge into one vertex( 解决冗余问题与空洞问题)
3.2 from depth map to mesh
- 采用方法:quad-tree algorithm
- Top-down ( depth map 分辨率从低到高)
- 三角剖分
- 定性描述: bridge depth 不连续\包含low confidence 的点
- 定量化计算 simple planarity test
PS:具体原理未细究
4 result
- Dataset: 链接
- Speed( real -time video)
- Device information
- 参数设置
- Dataset information