【论文阅读】15-Real-Time Visibility-Based Fusion of Depth Maps

0 basic information

Merrell P , Akbarzadeh A , Wang L , et al. Real-Time Visibility-Based Fusion of Depth Maps[C]// IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007. IEEE, 2007.

1 aims and characteristic

1. 解决depth map fusion问题：

• based on visibility constraint—2 method
• 作用： depth accurate , data not redundancy

2. Two stage – depth estimate & depth fusion

• 可单独进行（并行化），速度提升（GPU）
• 处理video – real-time

2 method

2.1 depth map estimate

2.1.1 depth 计算–plane-sweeping stereo

1. Referring image — neighboring images N

2. Divide into 2 group N_previous & N_follow

3. Each pixel in I_refer:

1)多个假想Plane ---多个假想深度值
2) 对于每一个假想平面，计算cost， 选择lowest cost 作为深度值

Cost计算：

---

N_previous每张图像 :

像素p 根据plane—homography — 对应像素p’ 在Square windows内—计算 absolute difference of intensity： c_pre_i

Cost-previous = sum(c_pre_i)

---

同理，Cost-following = sum（c_fol_i）

---

当前plane对应的 depth-cost-plane = min （ Cost-previous ，Cost-following）

2.1.1 depth confidence计算

1. 假设Cost（x,d）的噪声服从高斯分布（noise strength -sigma）
2. 计算confidence的意义：衡量真正depth 所对应的cost不是所有cost中的最小值的可能性（负相关关系）
3. 公式：
   

PS :
This equation produces a high confidence when the cost has a single sharp minimum. The confidence is low when the cost has a shallow minimum or several low minima.

2.2 depth map fusion

1. other views — center view ( referring view)
2. STEP1 ( common): depth map redering

other view + other depth map----- 3D point X -----projection ----center view depth map2 -D

3. 几个概念：

R： distance measurement
D: camera project to form the depth map
hypothesis ( R& D 一般不同，存在误差)

几种visibility conflict:
Free-space : center 2( rendering) other A < A’ — close
Occlusion : other 2(rendering) center C’< C ---- far away

2.2.1 stability fusion

1. 实质：穷举所有的occlusion + free-space 迭代得到fused depth
2. 步骤：

• Step2–Initial :
  center depth = closest depth( D_refer & D_other_2_refer)

• Step3–计算stability:
  公式 = the number of views (
  occlusion ) - the number of views (free-space )
  含义：衡量当前depth close
  or far away
  stability value ：
  0-- suitbale
  正-- away
  负 – close

• Step4–调整depth：
  move to the next closest，使得Stability 朝着0变化
  Final/fused depth: 使得Stability-非负的最小depth

• Step5-- add confidence
  The confidence of fused depth : add the confidence of the pixel in other views which supporting the fused depth
  supporting 的判断标砖： Di vs fused depth 阈值

2.2.2 confidence fusion

1. Based on confidence，depth先合并combine再 test
2. 步骤

• Step2 --Initial： the highest confidence –depth0

• Step3 —迭代combine （公式如下）： depth：与depth0 closest( 阈值内)的depth1，二者加权（confidence）融合 PS ：记录depth 迭代combine的次数K—the level
  of supporting Confidence： 采用相加操作
  

• Step4：验证fused depth 的有效性-- Conflict Detection Total supporting: == k (上文) > C_threshold 不满足则舍弃该值 Visibility constraints：更新 confidence–计算公式如下（ out of the supporting region – free -space &
  occlusion ）
  
  
  判断最终的confidence ( 正 -keep/ 负- remove)

• Step5 fused depth map 后处理 –hole
  w*w window- inlier depth( enough number ,否则，空洞保留)-中值滤波
  w_smaller* w_smaller —中值滤波—smooth

3 restruction surface

3.1 overlap depth

1. 处理多张fused depth map overlap部分不统一问题
2. 方法：

每一张fused map在生成最终的depth maps时，overlap区域的像素计算得到的new depth与前一张fused depth map对应值比较，若存在 free-space ，则reject；
反之二者merge into one vertex( 解决冗余问题与空洞问题)

3.2 from depth map to mesh

1. 采用方法：quad-tree algorithm
2. Top-down ( depth map 分辨率从低到高)
3. 三角剖分

• 定性描述： bridge depth 不连续\包含low confidence 的点
• 定量化计算 simple planarity test
  PS：具体原理未细究

4 result

1. Dataset： 链接
2. Speed( real -time video)
3. Device information
4. 参数设置
5. Dataset information