Robust estimation

1. Formulation

1.1 Inlier set maximization

Inlier set maximisation, a.k.a. consensus maximisation, where one seeks the model with the most number of inliers.

T.-J. Chin, Z. Cai, and F. Neumann, “Robust fitting in computer vision: Easy or hard?” in European Conf. on Computer Vision (ECCV), 2018.
T. J. Chin and D. Suter. The maximum consensus problem: recent algorithmic advances. Synthesis Lectures on Computer Vision, 7(2):1–194, 2017. 3
Adaptive Trimming (ADAPT) + non-minimal solver V. Tzoumas, P. Antonante, and L. Carlone. Outlier-robust spatial perception: Hardness, general-purpose algorithms, and guarantees. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2019. 3
Graph-theoretic method+PGO J. G. Mangelson, D. Dominic, R. M. Eustice, and R. Vasudevan, “Pairwise consistent measurement set maximization for robust multi-robot map merging,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), 2018, pp. 2916–2923.
F. Wen, R. Ying, Z. Gong, and P. Liu, “Efficient algorithms for maximum consensus robust fitting,” IEEE Trans. Robotics, 2019.
Z. Cai, T.-J. Chin, and V. Koltun, “Consensus maximization tree search revisited,” in Intl. Conf. on Computer Vision (ICCV), 2019, pp. 1637–1645.
Ruwan Tennakoon, David Suter, Erchuan Zhang, Tat-Jun Chin, Alireza Bab-hadiashar. Consensus Maximisation Using Influences of Monotone Boolean Functions. CVPR 2021.
F. Wen, H. Wei, Y. Liu, and P. Liu, “Simultaneous consensus maximization and model fitting,” arXiv preprint arXiv:2008.01574, 2020.

1.1.1 Formulation

Given a set of outlier-contaminated measurements $\mathcal{D}={(a_i,b_i)}_{i=1}^N$, where $a_i\in {\mathbb{R}}^d$ and $b_i\in \mathbb{R}$, and an inlier threshold $ϵ\in {\mathbb{R}}^d$ that maximises
$${\Psi }_{ϵ}(x|\mathcal{D})=\sum _{i=1}^N\mathbb{I}(|a_i^{T}x-b_i|\le ϵ),$$
where $x\in {\mathbb{R}}^d$ is the parameter vector, the 0/1 valued indicator function $\mathbb{I}$ returns 1 if its input predicate is true, and 0 otherwise. The quantity $|a_i^{T}x-b_i|$ is the residual of the $i$-th measurement with respect to $x$, and the value given by ${\Psi }_{ϵ}(x|\mathcal{D})$ is the consensus of $x$ with respect to $\mathcal{D}$. Constant $ϵ$ is the predefined inlier threshold, and $d$ is called the problem dimension.

1.2 M-estimation

• Bundle adjustment (Zach. ECCV’14)
• Registration (Zhou et al. ECCV’16, Yang et al. RAL’20)
• Data clustering (Shah et al. Proceedings of the National Academy of Sciences of the United States of America’17)
• Line process (Black et al. IJCV’96)
• Geman-McClure+no global optimality guarantees: Heng Yang, Pasquale Antonante, Vasileios Tzoumas, and Luca Carlone. Graduated non-convexity for robust spatial perception: From non-minimal solvers to global outlier rejection. IEEE Robotics and Automation Letters (RA-L), 5(2):1127–1134, 2020. 2, 3, 4, 6, 8, 27
• Wahba+non-minimal solver+TLS cost+SDP+lagrangian duality+redundant constraints: H. Yang and L. Carlone. A quaternion-based certifiably optimal solution to the Wahba problem with outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.
• M. Bosse, G. Agamennoni, and I. Gilitschenski. Robust estimation and applications in robotics. Foundations and Trends in Robotics, 4(4):225–269, 2016. 3
• TLS estimation+image registration: E. Ask, O. Enqvist, and F. Kahl, “Optimal geometric fitting under the truncated L2-norm,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1722–1729.
• PGO+M-estimators+outlier mitigation: P. Agarwal, G. D. Tipaldi, L. Spinello, C. Stachniss, and W. Burgard. Robust map optimization using dynamic covariance scaling. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2013.
• PGO+M-estimators+outlier mitigation: E. Olson and P. Agarwal, “Inference on networks of mixtures for robust robot mapping,” in Robotics: Science and Systems (RSS), July 2012.
• PGO+M-estimators+outlier mitigation: J.J. Casafranca, L.M. Paz, and P. Pinies. A back-end l 1 norm ´ based solution for factor graph slam. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 17–23. IEEE, 2013.
• M-estimators: Michael J. Black and Anand Rangarajan. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. Intl. J. of Computer Vision, 19(1):57–91, 1996. 3, 4
• M-estimators: Kirk Mac Tavish and Timothy D. Barfoot. At all costs: A comparison of robust cost functions for camera correspondence outliers. In Computer and Robot Vision (CRV), 2015 12th Conference on, pages 62–69. IEEE, 2015.
• 3D registration+high outlier rate+graph matching+approximate vertex cover O. Enqvist, K. Josephson, and F. Kahl, “Optimal correspondences from pairwise constraints,” in Intl. Conf. on Computer Vision (ICCV), 2009, pp. 1295–1302.
• R. A. Rossi, D. F. Gleich, and A. H. Gebremedhin. Parallel maximum clique algorithms with applications to network analysis. SIAM Journal on Scientific Computing, 37(5):C589–C616, 2015.
• Point cloud registration+optimality certification+TLS cost+outlier-pruning scheme+non-minimal solver+SDP+graph algorithm: H. Yang, J. Shi, and L. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. IEEE Trans. Robotics (T-RO), 2020.
• Maximum clique algorithm+BnB A. Parra Bustos, T.-J. Chin, F. Neumann, T. Friedrich, and M. Katzmann. A practical maximum clique algorithm for matching with pairwise constraints. arXiv preprint arXiv:1902.01534, 2019.

1.3 Graph algorithms

• J. Shi, H. Yang, and L. Carlone, “ROBIN: a Graph-Theoretic Approach to Reject Outliers in Robust Estimation using Invariants,” in IEEE International Conference on Robotics and Automation (ICRA), 2021.
• Compatibility tests and BnB for data association in SLAM J. Neira and J. Tardós. Data association in stochastic mapping using the joint compatibility test. IEEE Trans. Robot. Automat., 17(6):890–897, December 2001.
• Maximum Common Subgraph algorithm+maximum clique: T. Bailey, E. M. Nebot, J. K. Rosenblatt, and H. F. Durrant-Whyte. Data association for mobile robot navigation: a graph theoretic approach. In IEEE Intl. Conf. on Robotics and Automation (ICRA), volume 3, pages 2512–2517, 2000.
• Graph-cut+RANSAC D. Barath and J. Matas. Graph-cut ransac. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6733–6741, 2018.
• Graph clique S. Perera and N. Barnes. Maximal cliques based rigid body motion segmentation with a rgb-d camera. In Asian Conference on Computer Vision, pages 120–133. Springer, 2012.
• Approximate spectral method+strongly-connected clusters M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 1482–1489. IEEE, 2005.
• Graph-theoretic method+PGO J. G. Mangelson, D. Dominic, R. M. Eustice, and R. Vasudevan, “Pairwise consistent measurement set maximization for robust multi-robot map merging,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), 2018, pp. 2916–2923.

2. Mathematical tools (Algorithms)

2.1 Sub-optimal methods

2.1.1 Fast heuristics

1. RANSAC: — Minimal solvers + consensus maximization + outlier-robust
Iteratively fit model on random minimal samples.
RANSAC (Fischler et al. Communications of the ACM’81)
MLESAC (Tordoff et al. TPAMI’05)
USAC (Raguram et al. TPAMI’13)
RANSAC: Eric Brachmann and Carsten Rother. Neural-guided RANSAC: Learning where to sample model hypotheses. In Int. Conf. Comput. Vis., pages 4322–4331, 2019.
R. Raguram, J.-M. Frahm, and M. Pollefeys. A comparative analysis of ransac techniques leading to adaptive real-time random sample consensus. In European Conference on Computer Vision, pages 500–513. Springer, 2008.
P-NAPSAC: Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas. MAGSAC++, a fast, reliable and accurate robust estimator. In Conference on Computer Vision and Pattern Recognition, pages 1304–1312, 2020.
NAPSAC: R. D. Myatt, Philip Torr, Slawomir Nasuto, John Bishop, and R. Craddock. NAPSAC: High noise, high dimensional robust estimation - it’s in the bag. In British Machine Vision Conference, 2002.
Ondrej Chum and Jiri Matas. Matching with prosacprogressive sample consensus. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 220–226. IEEE, 2005.
Ondrej Chum and Jirı Matas. Optimal randomized RANSAC. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(8):1472–1482, 2008.
Strength:
1.Good efficiency.
Weakness:
1.Often yield a much lower consensus than the maximum achievable;
2.Sometimes unstable, i.e., fail to gurantee the same result every time
3.No optimality guarantees
4.Running time grows exponentially with the outlier ratio

2. Locally Optimized RANSAC (LO-RANSAC):
Whenever RANSAC finds a better solution, perform model fitting on non-minimal inlier samples.
1.LO-RANSAC Ondrej Chum, Jiri Matas, and Josef Kittler. Locally optimized ransac. In DAGM. Springer, 2003.
Quoc Huy Tran, Tat-Jun Chin, Wojciech Chojnacki, and David Suter. Sampling minimal subsets with large spans for robust estimation. International journal of computer vision, 106(1):93–112, 2014.
2.Fixing LO-RANSAC (Lebeda et al. BMVC’12)
Strength:
Weakness:
1.Still random, less effective on challenging data.

3. Graduated non-convexity (GNC) — Non-minimal solvers + M-estimation + outlier-robust
Geman-McClure+no global optimality guarantees+GNC+Black- Rangarajan (BR) duality: Heng Yang, Pasquale Antonante, Vasileios Tzoumas, and Luca Carlone. Graduated non-convexity for robust spatial perception: From non-minimal solvers to global outlier rejection. IEEE Robotics and Automation Letters (RA-L), 5(2):1127–1134, 2020. 2, 3, 4, 6, 8, 27
Michael J. Black and Anand Rangarajan. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. Intl. J. of Computer Vision, 19(1):57–91, 1996. 3, 4
Pasquale Antonante, Vasileios Tzoumas, Heng Yang, and Luca Carlone. Outlier-robust estimation: Hardness, minimally-tuned algorithms, and applications. arXiv preprint arXiv:2007.15109, 2020. 3
M. Bosse, G. Agamennoni, and I. Gilitschenski. Robust estimation and applications in robotics. Foundations and Trends in Robotics, 4(4):225–269, 2016. 3
S. Gold, A. Rangarajan, C. Lu, S. Pappu, and E. Mjolsness, “New algorithms for 2D and 3D point matching,” Pattern Recognition, vol. 31, no. 8, pp. 1019–1031, 1998.
Point cloud registration+Geman-McClure cost+GNC: Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In European Conf. on Computer Vision (ECCV), pages 766–782. Springer, 2016. 3, 6
Strength:
1.Without requiring an initial guess
2. Good robustness (handle 70-80% outlier rate)
3. Accurate: > local solvers; Efficiency > global solvers
Weakness:
1.No optimality guarantees.

4. Local optimization – Outlier-robust
An initial guess is available
IRLS: A. Chatterjee and V. M. Govindu. Efficient and robust large-scale rotation averaging. In Intl. Conf. on Computer Vision (ICCV), pages 521–528, 2013.
Single rotation averaging: Richard Hartley, Khurrum Aftab, and Jochen Trumpf. L1 rotation averaging using the Weiszfeld algorithm. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3041–3048. IEEE, 2011.
Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 4104–4113, 2016. 3
Robust ICP+point cloud registration: S. Bouaziz, A. Tagliasacchi, and M. Pauly. Sparse iterative closest point. In ACM Symp. Geom. Process., pages 113–123. Eurographics Association, 2013. 3
PGO+M-estimators+outlier mitigation: P. Agarwal, G. D. Tipaldi, L. Spinello, C. Stachniss, and W. Burgard. Robust map optimization using dynamic covariance scaling. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2013.
D. Crandall, A. Owens, , N. Snavely, and D. Huttenlocher. Discrete-continuous optimization for largescale structure from motion. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3001–3008, 2011. 3
C. Zach, “Robust bundle adjustment revisited,” in European Conf. on Computer Vision (ECCV), 2014, pp. 772–787.
PGO+outlier rejection: N. Sünderhauf and P. Protzel, “Switchable constraints for robust pose graph SLAM,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2012.
PGO+outlier rejection: N. Sunderhauf and P. Protzel. Towards a robust back-end for pose graph slam. In IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 1254–1261. IEEE, 2012.
PGO+M-estimators+outlier mitigation: E. Olson and P. Agarwal, “Inference on networks of mixtures for robust robot mapping,” in Robotics: Science and Systems (RSS), July 2012.
PGO+outlier mitigation: M. Pfingsthorn and A. Birk, “Generalized graph SLAM: Solving local and global ambiguities through multimodal and hyperedge constraints,” Intl. J. of Robotics Research, vol. 35, no. 6, pp. 601–630, 2016.
Shape alignment: C. Wang, Y. Wang, Z. Lin, A. L. Yuille, and W. Gao, “Robust estimation of 3D human poses from a single image,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 2361–2368.
PGO+gradient method E. Olson, J. Leonard, and S. Teller, “Fast iterative alignment of pose graphs with poor initial estimates,” in Proc. IEEE Int. Conf. Robot. Autom., May 2006, pp. 2262–2269.
PGO+M-estimators+outlier mitigation: J.J. Casafranca, L.M. Paz, and P. Pinies. A back-end l 1 norm ´ based solution for factor graph slam. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 17–23. IEEE, 2013.
PGO+outlier mitigation+local solver: G. H. Lee, F. Fraundorfer, and M. Pollefeys. Robust pose-graph loop closures with expectation-maximization. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2013.
PGO+outlier rejection: L. Carlone, A. Censi, and F. Dellaert. Selecting good measurements via l1 relaxation: a convex approach for robust estimation over graphs. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2014.
PGO+outlier rejection: G. Zioutas and A. Avramidis. Deleting outliers in robust regression with mixed integer programming. Acta Mathematicae Applicatae Sinica, 21(2):323–334, 2005.
PGO+outlier rejection: Y. Latif, C. D. C. Lerma, and J. Neira. Robust loop closing over time. In Robotics: Science and Systems (RSS), 2012.
PGO+outlier rejection: M. C. Graham, J. P. How, and D. E. Gustafson. Robust incremental slam with consistency-checking. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 117–124, Sept 2015.
PGO+outlier rejection: L. Carlone, A. Censi, and F. Dellaert. Selecting good measurements via l1 relaxation: a convex approach for robust estimation over graphs. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2014.

2.1.2 Deterministic approaches

Optimization on relaxed objective functions.

1.Exact Penalty method: Huu Le, Tat-Jun Chin, and David Suter. An exact penalty method for locally convergent maximum consensus. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on. IEEE, 2017.
2.Smooth Surrogate method: Pulak Purkait, Christopher Zach, and Anders Eriksson. Maximum consensus parameter estimation by reweighted l1 methods. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 312–327. Springer, 2017.
3.ADMM-based method (Le et al. BMVC’18)
4.Biconvex programming: Zhipeng Cai, Tat-Jun Chin, Huu Le, and David Suter. Deterministic consensus maximization with biconvex programming. In Proceedings of the European Conference on Computer Vision (ECCV), pages 685–700, 2018.
5.Deterministic approximate methods for maximum consensus robust fitting (Le et al. TPAMI’19)
Strength:
1.Gurantee the same result every time, i.e., good robustness.
Weakness:
1.Require the tuning of smoothing parameters;
2.Fail to gurantee the same result every time.

2.2 Globally optimal methods

2.2.1 Analytical solutions

A classical approach involves the computation of all the stationary points (among which there is the global minimum).

• Outlier-free+2D mesh registration A. Censi, “An ICP variant using a point-to-line metric,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), Pasadena, CA, May 2008.
• Outlier-free+closed form solution+global and local solutions: Lipu Zhou, Shengze Wang, and Michael Kaess. A fast and accurate solution for pose estimation from 3d correspondences. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.
• F. M. Mirzaei and S. I. Roumeliotis. Globally optimal pose estimation from line correspondences. In IEEE Int. Conf. Robot. Autom. (ICRA), 2011, pages 5581–5588. Ieee, may 2011.
• F. M. Mirzaei and S. I. Roumeliotis. Optimal estimation of vanishing points in a Manhattan world. In IEEE Int. Conf. Comput. Vis. (ICCV), 2011, pages 2454–2461. IEEE, 2011.

2.2.2 Brand and Bound

BnB search (Bazin et al. ECCV’14, Bazin et al. ACCV’12, Hartley et al. IJCV’09)
BnB + MIP hybrid method (Chin et al. CVPR’16, Speciale et al. CVPR’17)
Essetnial matrix+BnB: Richard I Hartley and Fredrik Kahl. Global optimization through searching rotation space and optimal estimation of the essential matrix. In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
Essetnial matrix + Brute-force search: Enqvist et al. CVPR’11
Absolute pose + BnB search (Enqvist et al. ECCV’08)
Essetnial matrix + BnB search (Enqvist et al. BMVC’09, Yang et al. ECCV’14)
Fundamental matrix + BnB search (Zheng et al. CVPR’11)
Translation + BnB search: J. Fredriksson, V. Larsson, and C. Olsson. Practical robust two-view translation estimation. In CVPR, pages 2684–2690, 2015.
BnB search + unknown correspondences (Fredriksson et al. CVPR’16)
Fundamental matrix + BnB + MIP (Li. ICCV’09)
Mixed-integer program: G. Izatt, H. Dai, and R. Tedrake. Globally optimal object pose estimation in point clouds with mixedinteger programming. In Proc. of the Intl. Symp. of Robotics Research (ISRR), 2017. 2, 3
BnB+Wahba+consensus maximization Jean-Charles Bazin, Yongduek Seo, and Marc Pollefeys. Globally optimal consensus set maximization through rotation search. In Asian Conference on Computer Vision, pages 539–551. Springer, 2012. 2, 3, 6, 8, 18
Yanmei Jiao, Yue Wang, Bo Fu, Qimeng Tan, Lei Chen, Shoudong Huang, and Rong Xiong. Globally optimal consensus maximization for robust visual inertial localization in point and line map. arXiv preprint arXiv:2002.11905, 2020. 3
BnB+point cloud registration+Decouple scale, rotation, translation: Álvaro Parra Bustos and Tat-Jun Chin. Guaranteed outlier removal for point cloud registration with correspondences. IEEE Trans. Pattern Anal. Machine Intell., 40(12):2868–2882, 2018. 2, 3
BnB+rotation+translation: D. Campbell, L. Petersson, L. Kneip, and H. Li, “Globally-optimal inlier set maximisation for simultaneous camera pose and feature correspondence,” in Intl. Conf. on Computer Vision (ICCV), 2017, pp. 1–10.
BnB+MIP: H. Li. Consensus set maximization with guaranteed global optimality for robust geometry estimation. In ICCV, pages 1074–1080, Sept 2009.
BnB: Y. Zheng, S. Sugimoto, and M. Okutomi. Deterministically maximizing feasible subsystem for robust model fitting with unit norm constraint. In CVPR, pages 1825–1832, June 2011.
BnB+MIP: T.-J. Chin, Y. Heng Kee, A. Eriksson, and F. Neumann. Guaranteed outlier removal with mixed integer linear programs. In CVPR, June 2016.
BnB+3D registration: C. Olsson, F. Kahl, and M. Oskarsson, “The registration problem revisited: Optimal solutions from points, lines and planes,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1. IEEE, 2006, pp. 1206–1213.
Outlier-free+exponential-time methods (e.g., BnB)+mesh registration: C. Olsson, F. Kahl, and M. Oskarsson, “Branch-and-bound methods for euclidean registration problems,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 783–794, 2009.
Strength:
1.Guarantee the global optimality;
2.Good accuracy.
Weakness:
1.Effective on only small input sizes (small d, N and/or number of outliers o)
2.Run in worst-case exponential time

2.2.3 Convex relaxations

• PGO+outlier rejection: L. Carlone, A. Censi, and F. Dellaert. Selecting good measurements via l1 relaxation: a convex approach for robust estimation over graphs. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2014.
• M. Leordeanu and M. Hebert, “A spectral technique for correspondence problems using pairwise constraints,” in IEEE International Conference on Computer Vision, vol. 2, 2005, pp. 1482–1489.

2.2.3.1 Convex approximation (Convex hull)

Relaxing non-convex constraints to its convex hull.

• Rotation synchronization: J. Saunderson, P. A. Parrilo, and A. S. Willsky. Semidefinite descriptions of the convex hull of rotation matrices. arXiv Prepr. arXiv1403.4914, 2014.
• SLAM: D. M. Rosen, C. DuHadway, and J. J. Leonard. A convex relaxation for approximate global optimization in simultaneous localization and mapping. In Robot. Autom. (ICRA), 2015 IEEE Int. Conf., pages 5822–5829. IEEE, 2015.
• 2D/3D registration: Y. Khoo and A. Kapoor. Non-iterative rigid 2D/3D point-set registration using semidefinite programming. IEEE Trans. Image Process., 25(7): 2956–2970, 2016.

2.2.3.2 Shor’s relaxation

Weakness: noise level increases; the number of correspondences approaches the minimal cases

• R. Tron, D. M. Rosen, and L. Carlone. On the Inclusion of Determinant Constraints in Lagrangian Duality for 3D SLAM.

2.2.3.2.1 QCQP+SDP relaxation

• Y. Ding. On efficient semidefinite relaxations for quadratically constrained quadratic programming. 2007.
• Pose graph optimization+lagrangian duality+QCQP+SDP relaxations+non-minimal solver+outlier-frree L. Carlone, D. Rosen, G. Calafiore, J. Leonard, and F. Dellaert, “Lagrangian duality in 3D SLAM: Verification techniques and optimal solutions,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2015, pp. 125–132.
• Pose graph optimization+optimality certification+QCQP+SDP+SOS+lagrangian duality: L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar pose graph optimization: Duality, optimal solutions, and verification. IEEE Trans. Robotics (T-RO), 32(3):545–565, 2016.
• Pose Graph Optimization+Lagrangian duality+QCQP+SDP relaxation+outlier-free: J. Briales and J. Gonzalez-Jimenez. Fast global optimality verification in 3D SLAM. In International Conference on Intelligent Robots and Systems, 2016.
• Pose graph optimization+tightness analysis+QCQP+SDP+SOS+non-minimal solver+riemannian staircase: D.M. Rosen, L. Carlone, A.S. Bandeira, and J.J. Leonard. SE-Sync: a certifiably correct algorithm for synchronization over the Special Euclidean group. Intl. J. of Robotics Research, 2018.
• Pose Graph Optimization+QCQP+SDP relaxation: J. Briales and J. Gonzalez-Jim ´ enez. Initialization of 3D Pose Graph Optimization using Lagrangian duality. In Int. Conf. on Robotics and Automation (ICRA). IEEE, 2017.
• Point-to-plane correspondences+lagrangian duality: C. Olsson and A. Eriksson, “Solving quadratically constrained geometrical problems using lagrangian duality,” in 2008 19th International Conference on Pattern Recognition. IEEE, 2008, pp. 1–5.
• Rotation averaging+QCQP+tight SDP relaxation: N. Boumal. A Riemannian low-rank method for optimization over semidefinite matrices with block-diagonal constraints. arXiv preprint arXiv:1506.00575, 2015.
• Rotation averaging+QCQP+tight SDP relaxation: A. Bandeira, N. Boumal, and A. Singer. Tightness of the maximum likelihood semidefinite relaxation for angular synchronization. Mathematical Programming, 2016.
• Point cloud registration+TLS cost+outlier-pruning scheme+non-minimal solver+SDP: H. Yang and L. Carlone. A polynomial-time solution for robust registration with extreme outlier rates. In Robotics: Science and Systems (RSS), 2019.
• Tightness analysis+non-minimal solver+QCQP+SDR+GNC+Black- Rangarajan (BR) duality+outlier-robust: Ji Zhao. An Efficient Solution to Non-Minimal Case Essential Matrix Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/TPAMI.2020.3030161.
• 3D multimodal registration+SDP+SDR+non-minimal solver+outlier-free: Jesus Briales and Javier Gonzalez-Jimenez. Convex Global 3D Registration with Lagrangian Duality. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.

2.2.3.2.2 Redundant constraints

• Y. Nesterov, H. Wolkowicz, and Y. Ye. Semidefinite programming relaxations of nonconvex quadratic optimization. In Handb. semidefinite Program., pages 361–419. Springer, 2000.
• P. A. Parrilo and S. Lall. Semidefinite programming relaxations and algebraic optimization in control. Eur. J. Control, 9(2):307–321, 2003.
• J. P. Ruiz and I. E. Grossmann. Using redundancy to strengthen the relaxation for the global optimization of MINLP problems. Comput. Chem. Eng., 35(12):2729–2740, 2011.
• Wahba+non-minimal solver+TLS cost+SDP+lagrangian duality+redundant constraints: H. Yang and L. Carlone. A quaternion-based certifiably optimal solution to the Wahba problem with outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.
• Relative pose estimation+eigenvalue-based formulation+QCQP+SDP+Shor’s relaxation+SOS+non-minimal solver+outlier-free+redundant constraints: Jesus Briales, Laurent Kneip, and Javier Gonzalez-Jimenez. A certifiably globally optimal solution to the non-minimal relative pose problem. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. Problem: outlier-free

2.2.4 Certifiable approaches

Given an optimization problem $\mathbb{P}(\mathbb{D})$ that depends on input data $\mathbb{D}$, we say that an algorithm $\mathbb{A}$ is certifiable if, after solving $\mathbb{P}(\mathbb{D})$, algorithm $\mathbb{A}$ either provides a certificate for the quality of its solution (e.g., a proof of optimality, a finite bound on its sub-optimality, or a finite bound on the distance of the estimate from the optimal solution), or declares failure otherwise.

A.S. Bandeira. A note on probably certifiably correct algorithms. Comptes Rendus Mathematique, 354(3):329–333, 2016.
N. Boumal, V. Voroninski, and A. Bandeira. The non-convex Burer–Monteiro approach works on smooth semidefinite programs. In Advances in Neural Information Processing Systems (NIPS), pages 2757–2765. 2016.

2.2.4.1 Fast certifiable algorithms

Although tractable, solving the convex problems from scratch may not be the most efficient way to address them. Fast certifiable algorithms typically leverage the existence of an optimality certifier which, given a solution obtained by any means, may be able to certify its optimality. A straightforward approach to get that certificate is through the resolution of the dual problem from scratch, whose optimal cost value always provides a lower bound on the optimal objective for the original problem. In many real-world problem instances this bound is tight, meaning both cost values are the same up to some accuracy and one can certify optimality from it.
对偶问题产生的解是原问题的下界，如果松弛是紧的，那两者的代价值是相等的，以此来证实最优性， QCQP->SDP。

• Jesus Briales and Javier Gonzalez-Jimenez. Cartan-sync: Fast and
  global se (d)-synchronization. IEEE Robotics and Automation Letters,
  2(4):2127–2134, 2017.
• Pose graph optimization+tightness analysis+QCQP+SDP+SOS+non-minimal solver+riemannian staircase+outlier-free: D.M. Rosen, L. Carlone, A.S. Bandeira, and J.J. Leonard. SE-Sync: a certifiably correct algorithm for synchronization over the Special Euclidean group. Intl. J. of Robotics Research, 2018.

2.2.4.2 Faster certification algorithm

However, it is also possible to leverage the dual problem to derive a faster certification algorithm. This approach usually leads to a closed-form expression for dual candidates, that is, points that are feasible for the dual problem (within the domain of the dual function). This expression depends on the optimal solution for the original (primal) problem, and if the candidate is indeed feasible, it allows us to certify the optimality of the original solution. For QCQP, the feasibility of the dual candidates is reduced to a spectral analysis of the Hessian of the Lagrangian.
Candidate solution to the original problem can be leveraged to obtain a candidate dual certificate in closed-form, providing a much faster way to solve the dual problem. This enables a fast optimality verification approach with which we can augment fast iterative solvers with no guarantees into Fast Certifiable Algorithms
用对偶问题推导出更快速的证实算法，对偶候选解的闭式表达如果是可行的，以此来证实原问题的最优性，QCQP中对偶候选解的可行性分析可以转化为拉格朗日Hessian矩阵的谱分析。

• Multiple rotation averaging+tightness analysis+optimality certification+SDP+SOS+non-minimal solver+outlier-free:: A. Eriksson, C. Olsson, F. Kahl, and T.-J. Chin. Rotation averaging and strong duality. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
• Point set registration+tightness analysis+optimality certification+outlier-free: José Pedro Iglesias, Carl Olsson, and Fredrik Kahl. Global optimality for point set registration using semidefinite programming. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
• Point cloud registration+optimality certification+TLS cost+outlier-pruning scheme+non-minimal solver+SDP+graph algorithm: H. Yang, J. Shi, and L. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. IEEE Trans. Robotics (T-RO), 2020.
• Pose Graph Optimization+Lagrangian duality+QCQP+SDP relaxation+outlier-free: J. Briales and J. Gonzalez-Jimenez. Fast global optimality verification in 3D SLAM. In International Conference on Intelligent Robots and Systems, 2016.
• Pose graph optimization+lagrangian duality+QCQP+SDP relaxations+non-minimal solver: L. Carlone, D. Rosen, G. Calafiore, J. Leonard, and F. Dellaert, “Lagrangian duality in 3D SLAM: Verification techniques and optimal solutions,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2015, pp. 125–132.
• Pose graph estimation+lagrangian duality+convex relaxation+outlier-free: L. Carlone and F. Dellaert, “Duality-based verification techniques for 2D SLAM,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), 2015, pp. 4589–4596.
• Pose graph estimation: David M. Rosen. Scalable low-rank semidefinite programming for certifiably correct machine perception. In Intl. Workshop on the Algorithmic Foundations of Robotics (WAFR), 2020.

1. Without outliers
Fredrik Kahl and Didier Henrion. Globally optimal estimates for geometric reconstruction problems. Intl. J. of Computer Vision, 74(1):3–15, 2007.

Rotation averaging: Anders Eriksson, Carl Olsson, Fredrik Kahl, and Tat-Jun Chin. Rotation averaging with the chordal distance: Global minimizers and strong duality. IEEE Trans. Pattern Anal. Mach. Intell., pages 1–1, 2019.
Generalized essential matrix+non-minimal solvers+QCQP+SDR+redundant constraints+outlier-free: Ji Zhao, Wanting Xu, and Laurent Kneip. A certifiably globally optimal solution to generalized essential matrix estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
3D shape reconstruction from 2D landmarks: H. Yang and L. Carlone. In perfect shape: Certifiably optimal 3D shape reconstruction from 2D landmarks. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Thomas Probst, Danda Pani Paudel, Ajad Chhatkuli, and Luc Van Gool. Convex relaxations for consensus and non-minimal problems in 3D vision. In Intl. Conf. on Computer Vision (ICCV), 2019. 2
3D registration: Haggai Maron, Nadav Dym, Itay Kezurer, Shahar Kovalsky, and Yaron Lipman. Point registration via efficient convex relaxation. ACM Transactions on Graphics (TOG), 35(4):1–12, 2016. 1, 2, 3
3D registration+tightness analysis: Kunal N Chaudhury, Yuehaw Khoo, and Amit Singer. Global registration of multiple point clouds using semidefinite programming. SIAM Journal on Optimization, 25(1):468–501, 2015. 2, 3
Triangulation+tightness analysis+QCQP+Shor’s relaxation: Chris Aholt, Sameer Agarwal, and Rekha Thomas. A QCQP approach to triangulation. In European Conf. on Computer Vision (ECCV), pages 654–667. Springer, 2012.
Rotation averaging: Johan Fredriksson and Carl Olsson. Simultaneous multiple rotation averaging using lagrangian duality. In Asian Conf. on Computer Vision (ACCV), pages 245–258. Springer, 2012.
Absolute pose estimation+SDP+SOS+non-minimal solver: Sérgio Agostinho, João Gomes, and Alessio Del Bue. CvxPnPL: A unified convex solution to the absolute pose estimation problem from point and line correspondences. arXiv preprint arXiv:1907.10545, 2019. 2, 3
Matthew Giamou, Ziye Ma, Valentin Peretroukhin, and Jonathan Kelly. Certifiably globally optimal extrinsic calibration from per-sensor egomotion. IEEE Robotics and Automation Letters, 4(2):367–374, 2019. 2
Hand-eye calibration: Jan Heller, Didier Henrion, and Tomas Pajdla. Hand-eye and robot-world calibration by global polynomial optimization. In IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 3157–3164. IEEE, 2014. 2, 3
Emmett Wise, Matthew Giamou, Soroush Khoubyarian, Abhinav Grover, and Jonathan Kelly. Certifiably optimal monocular hand-eye calibration. arXiv preprint arXiv:2005.08298, 2020. 2
Optimality certification: David M. Rosen. Scalable low-rank semidefinite programming for certifiably correct machine perception. In Intl. Workshop on the Algorithmic Foundations of Robotics (WAFR), 2020. 3
Tightness analysis: Nadav Dym and Yaron Lipman. Exact recovery with symmetries for procrustes matching. SIAM Journal on Optimization, 27(3):1513–1530, 2017. 3
Pose graph optimization+optimality certification+QCQP+SDP+SOS+lagrangian duality: L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar pose graph optimization: Duality, optimal solutions, and verification. IEEE Trans. Robotics (T-RO), 32(3):545–565, 2016.
Pose graph optimization+sparse bounded-degree variant of SOS relaxation J. G. Mangelson, J. Liu, R. M. Eustice, and R. Vasudevan, “Guaranteed globally optimal planar pose graph and landmark SLAM via sparse-bounded sums-of-squares programming,” in IEEE Intl. Conf. on Robotics and Automation (ICRA). IEEE, 2019, pp. 9306–9312.
Relative pose estimation: Mercedes Garcia-Salguero, Jesus Briales, and Javier Gonzalez-Jimenez. Certifiable relative pose estimation. Image and Vision Computing, 109:104142, 2021.
Certifiable algorithm: Matthew Giamou, Ziye Ma, Valentin Peretroukhin, and Jonathan Kelly. Certifiably globally optimal extrinsic calibration from per-sensor egomotion. IEEE Robotics and Automation Letters, 4(2):367–374, 2019.

2. Outlier-robust
1) High-breakdown-point
Wahba+non-minimal solver+TLS cost+SDP+lagrangian duality+redundant constraints: H. Yang and L. Carlone. A quaternion-based certifiably optimal solution to the Wahba problem with outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.

TLS cost: Cindy Orozco Bohorquez, Yuehaw Khoo, and Lexing Ying. Maximizing robustness of point-set registration by leveraging non-convexity. arXiv preprint arXiv:2004.08772, 2020.
TLS cost+PGO+no global optimality guarantees: P. Lajoie, S. Hu, G. Beltrame, and L. Carlone. Modeling perceptual aliasing in SLAM via discretecontinuous graphical models. IEEE Robotics and Automation Letters (RA-L), 2019.
TLS cost+optimality certification+small scale: H. Yang and L. Carlone. One Ring to Rule Them All: Certifiably Robust Geometric Perception with Outliers. in Conference on Neural Information Processing Systems (NeurIPS), 2020
TLS cost: Min Yang, Linli Xu, Martha White, Dale Schuurmans, and Yao-liang Yu. Relaxed clipping: A global training method for robust regression and classification. In Advances in neural information processing systems, pages 2532–2540, 2010.
2) Low-breakdown-point
Yao-liang Yu, Özlem Aslan, and Dale Schuurmans. A polynomial-time form of robust regression. In Advances in Neural Information Processing Systems (NIPS), pages 2483–2491, 2012.
Ricardo A Maronna, R Douglas Martin, Victor J Yohai, and Matías Salibián-Barrera. Robust statistics: theory and methods (with R). John Wiley & Sons, 2019.
L1-norm+convex relaxation+semidefinite method: Lanhui Wang and Amit Singer. Exact and stable recovery of rotations for robust synchronization. Information and Inference: A Journal of the IMA, 2(2):145–193, 2013.
Huber+l1-norm+convex+non-minimal solver: L. Carlone and G. Calafiore. Convex relaxations for pose graph optimization with outliers. IEEE Robotics and Automation Letters (RA-L), 3(2):1160–1167, 2018.
Essential matrices+BnB: J. Yang, H. Li, and Y. Jia. Optimal essential matrix estimation via inlier-set maximization. In ECCV, pages 111–126, 2014.

2.2.3 Others

Tree search + pseudo-convex (Li. CVPR’07, Chin et al. CVPR’15)
Danda Pani Paudel and Luc Van Gool. Sampling algebraic varieties for robust camera autocalibration. In European Conference on Computer Vision, pages 275–292. Springer, Cham, 2018.
Danda Pani Paudel, Adlane Habed, Cedric Demonceaux, and Pascal Vasseur. Robust and optimal sum-of-squaresbased point-to plane registration of image sets and structured scenes. In Proceedings of the IEEE International Conference on Computer Vision, pages 2048–2056, 2015.
Pablo Speciale, Danda P Paudel, Martin R Oswald, Hayko Riemenschneider, Luc V Gool, and Marc Pollefeys. Consensus maximization for semantic region correspondences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7317–7326, 2018.
A star-search: T.-J. Chin, P. Purkait, A. Eriksson, and D. Suter. Efficient globally optimal consensus maximisation with tree search. In CVPR, pages 2413–2421, 2015.

求取两视图之间的相对平移
J. Fredriksson, O. Enqvist, and F. Kahl. Fast and reliable two-view translation estimation. In Computer Vision and Pattern Recognition (CVPR), 2014.

3. 特殊环境、有先验信息的情况

• Friedrich Fraundorfer, Petri Tanskanen, and Marc Pollefeys. A minimal case solution to the calibrated relative pose problem for the case of two known orientation angles. In European Conference on Computer Vision (ECCV), 2010.
• Oleg Naroditsky, Xun S. Zhou, Jean Gallier, Stergios I. Roumeliotis, and Kostas Daniilidis. Two efficient solutions for visual odometry using directional correspondence. Trans. Pattern Analysis and Machine Intelligence (PAMI), 2012.
• Chris Sweeney, John Flynn, and Matthew Turk. Solving for relative pose with a partially known rotation is a quadratic eigenvalue problem. International Conference on 3D Vision (3DV), 2014.
• Olivier Saurer, Pascal Vasseur, Remi Boutteau, Cedric Demonceaux, Marc Pollefeys, and Friedrich Fraundorfer. Homography based egomotion estimation with a common direction. Trans. Pattern Analysis and Machine Intelligence (PAMI), 2017.
• Yaqing Ding, Jian Yang, Jean Ponce, and Hui Kong. An efficient solution to the homography-based relative pose problem with a common reference direction. In International Conference on Computer Vision (ICCV), 2019.
• Yaqing Ding, Jian Yang, and Hui Kong. An efficient solution to the relative pose estimation with a common direction. In 2020 IEEE International Conference on Robotics and Automation, 2020.
• Ding, Y., Baráth, D., Yang, J., Kong, H., & Kukelova, Z. (2020). Globally Optimal Relative Pose Estimation with Gravity Prior. ArXiv, abs/2012.00458.

4. Minimal/Non-minimal case

4.1 Minimal case

Minimal solvers assume noiseless measurements and use the minimum number of measurements necessary to estimate parameters.

• Tomas Pajdla and Zuzana Kukelova. Minimal problems in computer vision. http://cmp.felk.cvut.cz/old_pages/mini/, 2019. 3
• Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla. Automatic generator of minimal problem solvers. In European Conf. on Computer Vision (ECCV), pages 302–315. Springer, 2008. 3
• Xiao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng. Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Machine Intell., 25(8):930–943, 2003. 3
• D. Nistér. An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Machine Intell., 26(6):756–770, 2004. 3
• Outlier-free+point cloud registration+3-point solver Berthold K. P. Horn. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer., 4(4):629–642, Apr 1987. 2, 3
• Outlier-free+12-point solver+mesh registration K. Khoshelham, “Closed-form solutions for estimating a rigid motion from plane correspondences extracted from point clouds,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 114, pp. 78 – 91, 2016.
• Line-to-plane: J. Briales and J. Gonzalez-Jimenez. A minimal solution for the calibration of a 2D laser-rangefinder and a camera based on scene corners. In Intell. Robot. Syst. (IROS), 2015 IEEE/RSJ Int. Conf., pages 1891–1896, sep 2015.
• Line-to-plane: J. Briales and J. Gonzalez-Jimenez. A Minimal Closed-form Solution for the Perspective Three Orthogonal Angles (P3oA) Problem: Application To Visual Odometry. J. Math. Imaging Vis., pages 1–18, 2015.

4.2 Non-minimal case

Non-minimal solvers account for measurement noise and estimate parameters via nonlinear least squares (NLS).

4.2.1 Closed form

• F. L. Markley. Attitude determination using vector observations and the singular value decomposition. The Journal of the Astronautical Sciences, 36(3):245–258, 1988. 3
• F. L. Markley and J. L. Crassidis. Fundamentals of spacecraft attitude determination and control, volume 33. Springer, 2014. 3
• Outlier-free+mesh registration A. Censi, “An ICP variant using a point-to-line metric,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), Pasadena, CA, May 2008.
• K. Somani Arun, Thomas S. Huang, and Steven D. Blostein. Least-squares fitting of two 3-D point sets. IEEE Trans. Pattern Anal. Machine Intell., 9(5):698–700, 1987. 1, 2

4.2.2 Polynomial equations from first-order optimality conditions

• F. Wientapper, M. Schmitt, M. Fraissinet-Tachet, and A. Kuijper, “A universal, closed-form approach for absolute pose problems,” Computer Vision and Image Understanding, vol. 173, pp. 57–75, 2018.
• L. Kneip, H. Li, and Y. Seo, “Upnp: An optimal o (n) solution to the absolute pose problem with universal applicability,” in European Conference on Computer Vision. Springer, 2014, pp. 127–142.
• Outlier-free+closed form solution+global and local solutions: Lipu Zhou, Shengze Wang, and Michael Kaess. A fast and accurate solution for pose estimation from 3d correspondences. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.

4.2.3 Nonconvex problem

• BnB+non-minimal solver: Hartley, R.I., Kahl, F.: Global optimization through searching rotation space and optimal estimation of the essential matrix. In: ICCV (2007)
• Outlier-free+exponential-time methods (e.g., BnB)+mesh registration: C. Olsson, F. Kahl, and M. Oskarsson, “Branch-and-bound methods for euclidean registration problems,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 783–794, 2009.
• Local solvers: S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and R. Szeliski. Bundle adjustment in the large. In European Conf. on Computer Vision (ECCV), pages 29–42, 2010. 3
• Outlier-free+shape alignment+approximate relaxations+non-minimal solvers X. Zhou, M. Zhu, S. Leonardos, and K. Daniilidis, “Sparse representation for 3D shape estimation: A convex relaxation approach,” IEEE Trans. Pattern Anal. Machine Intell., vol. 39, no. 8, pp. 1648–1661, 2017.
• Generalized essential matrix+non-minimal solvers+QCQP+SDR+redundant constraints+outlier-free: Ji Zhao, Wanting Xu, and Laurent Kneip. A certifiably globally optimal solution to generalized essential matrix estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.

4.2.4 Non-catalog

5. Robust pose estimation

5.1 Absolute pose estimation

Given a set of 3D points with known position, and corresponding 2D image points, determine the location and pose of the camera.

5.1.1 Formulation

Given 3D points $X_i$ and corresponding 2D image points, $x_i$, determine the camera matrix $P$ such that $x_i=P{X}_i$ for all $i$. An algebraic solution to this problem is given by the DLT algorithm (R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision – 2nd Edition. Cambridge University Press, 2003., chapter 7).

where each $P^{iT}$ is a 4-vector, the $i$-th row of $P$. Alternatively, one may choose to use only the first two equations:

$P$ has 12 entries and (ignoring scale) 11 degrees of freedom, it is necessary to have 11 equations to solve for $P$. Since each point correspondence leads to two equations, at a minimum $5\frac{1}{2}$ such correspondences are required to solve for $P$.
Given this minimum number of correspondences, the solution is exact, i.e. the space points are projected exactly onto their measured images. The solution is obtained by solving $Ap=0$ where A is an $11\times 12$ matrix in this case. In general $A$ will have rank 11, and the solution vector $p$ is the 1-dimensional right null-space of $A$.

• Joel A. Hesch and Stergios I. Roumeliotis. A direct leastsquares (dls) method for pnp. In International Conference on Computer Vision (ICCV), 2011.
• Yinqiang Zheng, Yubin Kuang, Shigeki Sugimoto, Kalle Astrom, and Masatoshi Okutomi. Revisiting the pnp problem: A fast, general and optimal solution. In International Conference on Computer Vision (ICCV), 2013.
• Gaku Nakano. Globally optimal dls method for pnp problem with cayley parameterization. In British Machine Vision Conference (BMVC), 2015.
• Gaku Nakano. A versatile approach for solving pnp, pnpf, and pnpfr problems. In European Conference on Computer Vision (ECCV), 2016.
• Ping Wang, Guili Xu, Yuehua Cheng, and Qida Yu. A simple, robust and fast method for the perspective-n-point problem. Pattern Recognition Letters, 2018.
• Lipu Zhou and Michael Kaess. An efficient and accurate algorithm for the perspecitve-n-point problem. In International Conference on Intelligent Robots and Systems (IROS), 2019.
• George Terzakis and Manolis Lourakis. A consistently fast and globally optimal solution to the perspective-n-point problem. In European Conference on Computer Vision (ECCV), 2020.
• G. Schweighofer and A. Pinz. Globally optimal O(n) solution to the PnP problem for general camera models. In British Machine Vision Conf. (BMVC), pages 1–10, 2008.
• Minimal solver+ absolute pose problem: Z. Kukelova, M. Bujnak, and T. Pajdla. Real-time solution to the absolute pose problem with unknown radial distortion and focal length. In International Conference on Computer Vision (ICCV), 2013.
• Minimal solver+ absolute pose problem: V. Larsson, T. Sattler, Z. Kukelova, and M. Pollefeys. Revisiting radial distortion absolute pose. In International Conference on Computer Vision (ICCV), 2019.

5.1.2 Algorithms

5.1.2.1 Algebraic approaches

Optimize an algebraic cost function
1.Minimal case (Haralick et al. IJCV’94)
2.Four points (Quan et al. TPAMI’99)
3.DLT (R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision – 2nd Edition. Cambridge University Press, 2003., chapter 7)
Weakness:
1.Solution may get trapped in a local minimum.

5.1.2.2 RANSAC

Hypothesize several poses from the sampled correspondences, and retrieve the optimal pose fitting most inliers.
1.Fischler et al. 1981
2.Lepetit et al. IJCV’08
3.Zheng et al. ICCV’13
Weakness:
1.Fail to handle high outlier ratios, i.e., limited robustness;
2.Inefficiency, i.e., number of iterations significantly increases.

5.1.2.3 Inlier set maximization

Searches over the parameter space to avoid data sampling, guaranteeing the global optimality in terms of maximizing the number of inliers.
1.Li et al. ICCV’09
2.Larsson et al. BMVC’16
3.Speciale et al. CVPR’14
Strength:
1.Guarantee the global optimality.
2. High robustness.
Weakness:
1.Inefficiency.

5.1.2.4 Expectation-maximization

Estimate both parameters and inliers, and solve it in an iterative way.
1.Li et al. IROS’18
2.Ferraz et al. CVPR’14
PGO+outlier mitigation+local solver: G. H. Lee, F. Fraundorfer, and M. Pollefeys. Robust pose-graph loop closures with expectation-maximization. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2013.
Strength:
1.Good efficiency.
Weakness:
1.Require a reliable initial pose and hardly converges.
2.Fail to handle high outlier ratios, e.g. >50%

5.1.2.5 Robust losses

Define a cost function and minimize it, given an initial pose.
1.Moreno et al. ICRA’13
2.Gomez-Ojeda et al. ICRA’16
Strength:
1.Good efficiency.
Weakness:
Prone to local minimum;
Fail to handle high outlier ratios, i.e., limited robustness.

5.1.2.6 Globally optimal methods

1.$L_2$ norm reprojection error (Olsson et al. ICPR’06)
2.$L_{\infty }$ norm reprojection error + BnB (Hartley et al. ICCV’07, Enqvist et al. ECCV’08)
Closed form: L. Kneip, H. Li, and Y. Seo, “Upnp: An optimal o (n) solution to the absolute pose problem with universal applicability,” in European Conference on Computer Vision. Springer, 2014, pp. 127–142.

5.2 Relative pose estimation

The relative pose (or relative orientation) problem is to find the relative pose of two cameras, given a set of image correspondences, determined by unknown 3D points. Often the solution to this problem involves finding the positions of the 3D points as well.

• Olivier D Faugeras and Steve Maybank. Motion from point matches:
  multiplicity of solutions. International Journal of Computer Vision,
  4(3):225–246, 1990.

5.2.1 Formulation

Given image correspondences $x_i\leftrightarrow$ $x_i^{{}^{\prime }}$, find two camera matrices $P$ and $P^{{}^{\prime }}$, along with 3D points $X_i$, such tah $x_i=P{X}_i$ and $x_i^{{}^{\prime }}=P^{{}^{\prime }}{X}_i$. This is the problem often solved by computing the fundamental or essential matrix: acommonly used algorithm is the 8-point algorithm (R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision – 2nd Edition. Cambridge University Press, 2003., chapter 11).
The fundamental matrix is defined by the equation $$x^{\prime T}Fx=0$$ for any pair of matching points in two images. In particular, writing $x=(x,y,1{)}^T$ and $x^{{}^{\prime }}=(x^{{}^{\prime }},y^{{}^{\prime }},1{)}^T$ each point match gives rise to one linear equation in the unknown entries of $F$

Denote by $f$ the 9-vector made up of the entries of $F$ in row-major order. Then the above equation can be expressed as a vector inner product

From a set of $n$ point matches, we obtain a set of linear equations of the form

This is a homogeneous set of equations, and $f$ can only be determined up to scale. $F$ has 9 entries and (ignoring scale) 8 degrees of freedom, it is necessary to have 8 equations to solve for $F$. Since each point correspondence leads to only one equation, at a minimum 8 such correspondences are required to solve for $F$. For a solution to exist, matrix $A$ must have rank at most 8, and if the rank is exactly 8, then the solution is unique (up to scale), and can be found by linear methods – the solution is the generator of the right null-space of $A$. The algorithm just described is the essence of a method called the 8-point algorithm for computation of the fundamental matrix.

• fundamental matrix & 8 point method: R. I. Hartley, “In defence of the 8-point algorithm,” in International Conference on Computer Vision, 1995, pp. 1064–1070.
• fundamental matrix & 8 point method: Olivier D Faugeras and Steve Maybank. Motion from point matches: multiplicity of solutions. International Journal of Computer Vision, 4(3):225–246, 1990.
• DLT: Seong Hun Lee and Javier Civera. Geometric interpretations of the normalized epipolar error. arXiv preprint arXiv:2008.01254, 2020.
• Laurent Kneip, Roland Siegwart, and Marc Pollefeys. Finding the exact rotation between two images independently of the translation. In Eur. Conf. Comput. Vis., pages 696–709, 2012.
• Not global optimum R. Hartley, “Minimizing algebraic error in geometric estimation problem,” in IEEE International Conference on Computer Vision, 1998, pp. 469–476.
• Viktor Larsson, Kalle Astrom, and Magnus Oskarsson. Efficient solvers for minimal problems by syzygy-based reduction. In Computer Vision and Pattern Recognition (CVPR), 2017.
• Viktor Larsson, Magnus Oskarsson, Kalle Astrom, Alge Wallis, Zuzana Kukelova, and Tomas Pajdla. Beyond grobner bases: Basis selection for minimal solvers. In Computer Vision and Pattern Recognition (CVPR), 2018.

minimizing an algebraic error

• G. Chesi, “Camera displacement via constrained minimization of the algebraic error,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 370–375, 2009.
• Tsuyoshi Migita and Takeshi Shakunaga. Evaluation of epipole estimation methods with/without rank-2 constraint across algebraic/geometric error functions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7, 2007.

5.2.2 Related work

5.2.2.1 Minimal case

The essential matrix has five degrees of freedom (three from 3D rotation, three from 3D translation and one less from the scale ambiguity) and therefore, only five correspondences (except for degenerate cases) are required for its estimation.

• Essential matrix+5 point method: D. Nister, “An efficient solution to the five-point relative pose problem,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 756–770, 2004.
• Minimal solver: Zuzana Kukelova and Tomas Pajdla. Two minimal problems for cameras with radial distortion. In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
• Minimal solver: Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla. Polynomial eigenvalue solutions to the 5-pt and 6-pt relative pose problems. In BMVC, volume 2, page 2008, 2008.
• Minimal solver: Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla. Polynomial eigenvalue solutions to minimal problems in computer vision. Trans. Pattern Analysis and Machine Intelligence (PAMI), 2012.
• Minimal solver: Kaveh Fathian, J. P. Ramirez-Paredes, Emily A. Doucette, J. W. Curtis, and Nicholas R. Gans. QuEst: A quaternionbased approach for camera motion estimation from minimal feature points. IEEE Robotics and Automation Letters, 3(2):857–864, 2018.
• Minimal solver: Hongdong Li and Richard Hartley. Five-point motion estimation made easy. In IEEE Conf. Comput. Vis. Pattern
  Recog., volume 1, pages 630–633, 2006.
• Minimal solver: Henrik Stewenius, Christopher Engels, and David Nister. Recent developments on direct relative orientation. ISPRS Journal of Photogrammetry and Remote Sensing, 60(4):284–294, 2006.
• Minimal solver: Richard Hartley and Hongdong Li. An efficient hidden variable approach to minimal-case camera motion estimation. Trans. Pattern Analysis and Machine Intelligence (PAMI), 2012.

5.2.2.2 8-point algorithm

8pt algorithm can be considered as the state-of-the-art initialization for further refinements.

• Essential matrix+8-point algorithm H Christopher Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 293(5828):133–135, 1981.
• Normalized 8-point algorithm: Richard I. Hartley. In defense of the eight-point algorithm. IEEE Trans. Pattern Anal. Mach. Intell., 19(6):580–593, 1997.

5.2.2.3 Iterative optimization on the essential matrix manifold

Minimal solvers or the 8-point algorithm typically provide suboptimal solutions for the non-minimal N-point problem and therefore it is a common practice to refine these initial estimates by local, iterative methods.
In this context, the essential matrix manifold has been characterized via different, yet (almost) equivalent formulations.

• Outlier-robust: Tom Botterill, Steven Mills, and Richard Green. Refining essential matrix estimates from ransac. In Proceedings Image and Vision Computing
• U. Helmke, K. Huper, P. Y. Lee, and J. Moore, “Essential matrix estimation using Gauss-Newton iterations on a manifold,” International Journal of Computer Vision, vol. 74, no. 2, pp. 117–136, 2007.
• 5 point initialization Vincent Lui and Tom Drummond. An iterative 5-pt algorithm for fast and robust essential matrix estimation. In BMVC, 2013.
• 8 point initialization+iterative method: Y. Ma, J. Kosecka, and S. Sastry, “Optimization criteria and geometric algorithms for motion and structure estimation,” International Journal of Computer Vision, vol. 44, no. 3, pp. 219–249, 2001.
• Quotient Riemannian manifold: Roberto Tron and Kostas Daniilidis. The space of essential matrices as a riemannian quotient manifold. SIAM Journal on Imaging Sciences, 10(3):1416–1445, 2017.
• Z. Zhang, “Determining the epipolar geometry and its uncertainty: A review,” International Journal of Computer Vision, vol. 27, pp. 161–195, 1998.
• K. Kanatani and Y. Sugaya, “Unified computation of strict maximum likelihood for geometric fitting,” Journal of Mathematical Imaging and Vision, vol. 38, pp. 1–13, 2010.

5.2.2.1 Globally optimal methods

Despite its attractive as fast solvers, the above-mentioned proposals do not guarantee nor certify if the retrieved solution is optimal.

• Essetnial matrix+BnB+L$\infty$ cost: Richard I Hartley and Fredrik Kahl. Global optimization through searching rotation space and optimal estimation of the essential matrix. In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.

• Fundamental matrix + BnB search (Li. ICCV’09)

• Brute-force search + essetnial matrix (Enqvist et al. CVPR’11)

• Eigenvalue-based formulation+BnB+non-minimal+LM+BnB: L. Kneip and S. Lynen, “Direct optimization of frame-to-frame rotation,” in IEEE International Conference on Computer Vision, 2013, pp. 2352–2359.

• Essential matrix +inlier-set maximization+BnB: J. Yang, H. Li, and Y. Jia. Optimal essential matrix estimation via inlier-set maximization. In ECCV, pages 111–126, 2014.

• BnB + essetnial matrix(Enqvist et al. BMVC’09, Yang et al. ECCV’14)

• BnB + unknown correspondences (Fredriksson et al. CVPR’16)

• Fundamental matrix determinant as 0 (Y. Zheng, S. Sugimoto, and M. Okutomi, “A branch and contract algorithm for globally optimal fundamental matrix estimation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 2953–2960.
  F. Bugarin, A. Bartoli, D. Henrion, J.-B. Lasserre, J.-J. Orteu, and T. Sentenac, “Rank-constrained fundamental matrix estimation by polynomial global optimization versus the eight-point algorithm,” Journal of Mathematical Imaging and Vision, vol. 53, no. 1, pp. 42–60, 2015.)

• Fundamental matrix + constrained polynomial optimization (G. Chesi, A. Garulli, A. Vicino, and R. Cipolla, “Estimating the fundamental matrix via constrained least-squares: A convex approach,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 3, pp. 397–401, 2002.)

• POP+SDR: Fredrik Kahl and Didier Henrion. Globally optimal estimates for geometric reconstruction problems. Intl. J. of Computer Vision, 74(1):3–15, 2007. 2

• Relative pose estimation+eigenvalue-based formulation+SDP+SOS+non-minimal solver+outlier-free: Jesus Briales, Laurent Kneip, and Javier Gonzalez-Jimenez. A certifiably globally optimal solution to the non-minimal relative pose problem. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. Problem: outlier-free

• Tightness analysis+non-minimal solver+QCQP+SDR+GNC+Black- Rangarajan (BR) duality+outlier-robust: Ji Zhao. An Efficient Solution to Non-Minimal Case Essential Matrix Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/TPAMI.2020.3030161.

• Relative pose estimation: Mercedes Garcia-Salguero, Jesus Briales, and Javier Gonzalez-Jimenez. Certifiable relative pose estimation. Image and Vision Computing, 109:104142, 2021.

• Triangulation+tightness analysis+QCQP+Shor’s relaxation: Chris Aholt, Sameer Agarwal, and Rekha Thomas. A qcqp approach to triangulation. In European Conference on Computer Vision, pages 654–667. Springer, 2012.

5.3 Multiple view geometry

• Triangulation: R. Hartley and F. Schaffalitzky. L∞ minimization in geometric reconstruction problems. In Computer Vision and Pattern Recognition (CVPR), 2004.
• F. Kahl. Multiple view geometry and the L∞-norm. In International Conference on Computer Vision (ICCV), pages 1002–1009, Beijing, China, 2005.
• Known-rotation problem: F. Kahl and R. Hartley. Multiple-view geometry under the L∞-norm. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 30:1603–17, 09 2008.
• Q. Ke and T. Kanade. Quasiconvex optimization for robust geometric reconstruction. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 29(10):1834–1847, 2007.
• 1D cameras O. Enqvist, F. Kahl, C. Olsson, and K. Astr ˚ om. Global op- ¨ timization for one-dimensional structure and motion problems. SIAM J. Imaging Sci., 3(4):1075–1095, 2010.
• Triangulation of planar structures+non-sequential method C. Olsson and A. Eriksson. Triangulating a plane. In Scandinavian Conference on Image Analysis (SCIA), 2011.
• Known-rotation problem: K. Sim and R. Hartley. Recovering camera motion using l∞ minimization. In Computer Vision and Pattern Recognition (CVPR), 2006.
• Known-rotation problem: Q. Zhang, T.-J. Chin, and H.M. Le. A fast resectionintersection method for the known rotation problem. In Computer Vision and Pattern Recognition (CVPR), 2018.
• Non-sequential method: P. Moulon, P. Monasse, and R. Marlet. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. In International Conference on Computer Vision (ICCV), 2013.
• Non-sequential method: A.P. Bustos, T.-J. Chin, A. Eriksson, and I. Reid. Visual SLAM: Why bundle adjust? In International Conference on Robotics and Automation (ICRA), 2019.
• D. Nister. Reconstruction from uncalibrated sequences with a hierarchy of trifocal tensors. In European Conference on Computer Vision (ECCV), 2000.
• R. Hartley, F. Kahl, C. Olsson, and Y. Seo. Verifying global minima for L2 minimization problems in multiple view geometry. International Journal of Computer Vision (IJCV), 101(2):288–304, 2012.
• Outlier rejection: Sim, K., Hartley, R.: Removing outliers using the $L_{\infty }$ norm. In: CVPR (2006)
• Outlier rejection: Li, H.: A practical algorithm for $L_{\infty }$ triangulation with outliers. In: CVPR (2007)
• Computational geometry+outlier rejection: C. Olsson, O. Enqvist, and F. Kahl. A polynomial-time bound for matching and registration with outliers. In CVPR, pages 1–8, 2008.
• Outlier rejection+robust relaxations: C. Olsson, A. Eriksson, and R. Hartley. Outlier removal using duality. In Computer Vision and Pattern Recognition (CVPR), 2010.
• Outlier rejection+robust relaxations: Y. Seo, H. Lee, and S.W. Lee. Outlier removal by convex optimization for L∞-infinity approaches. In Advances in Image and Video Technology, 2009.
• Outlier rejection+robust relaxations: J. Yu, A. Eriksson, T.-J. Chin, and D. Suter. An adversarial optimization approach to efficient outlier removal. In International Conference on Computer Vision (ICCV), 2011.
• Outlier rejection+robust relaxations: Q. Zhang, T.-J. Chin, and D. Suter. Quasiconvex plane sweep for triangulation with outliers. In International Conference on Computer Vision (ICCV), 2017.
• Radial constraint: S. Thirthala and M. Pollefeys. Radial multi-focal tensors. International Journal of Computer Vision (IJCV), 96:195–211, 2012.
• Convex relaxation+sensitive to noise: J.-H. Kim, Y. Dai, H. Li, X. Du, and J. Kim. Multi-view 3D reconstruction from uncalibrated radially-symmetric cameras. In International Conference on Computer Vision (ICCV), 2013.
• F. Camposeco, T. Sattler, and M. Pollefeys. Non-parametric structure-based calibration of radially symmetric cameras. In International Conference on Computer Vision (ICCV), 2015.
• Multi-camera calibration: Y. Lin, V. Larsson, M. Geppert, Z. Kukelova, M. Pollefeys, and T. Sattler. Infrastructure-based multi-camera calibration using radial projections. In European Conference on Computer Vision (ECCV), 2020.
• V. Larsson, N. Zobernig, K. Taskin, and M. Pollefeys. Calibration-free structure-from-motion with calibrated radial trifocal tensors. In European Conference on Computer Vision (ECCV), 2020.
• S. Agarwal, N. Snavely, and S.M. Seitz. Fast algorithms for L∞-problems in multiview geometry. In Computer Vision and Pattern Recognition (CVPR), 2008.
• Z. Dai, Y. Wu, F. Zhang, and H. Wang. A novel fast method for L∞ problems in multiview geometry. In European Conference on Computer Vision (ECCV), 2012.
• Carl Olsson, Viktor Larsson, Fredrik Kahl. A Quasiconvex Formulation for Radial Cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14576-14585.

6. 其他类型的correspondence

6.1 Affine correspondences

6.2 Semantic correspondences

• D. P. Paudel, A. Habed, and L. V. Gool. Optimal transformation estimation with semantic cues. In ICCV, pages 4658–4667, 2017.
• Pablo Speciale, Danda P Paudel, Martin R Oswald, Hayko Riemenschneider, Luc V Gool, and Marc Pollefeys. Consensus maximization for semantic region correspondences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7317–7326, 2018.
• H. Bristow, J. Valmadre, and S. Lucey. Dense semantic correspondence where every pixel is a classifier. In ICCV, pages 4024–4031, 2015.
• D. Glasner, S. N. P. Vitaladevuni, and R. Basri. Contourbased joint clustering of multiple segmentations. In CVPR, 2011.
• K. Han, R. S. Rezende, B. Ham, K. K. Wong, M. Cho, C. Schmid, and J. Ponce. Scnet: Learning semantic correspondence. CoRR, abs/1705.04043, 2017.
• A. Toshev, J. Shi, and K. Daniilidis. Image matching via saliency region correspondences. In CVPR, 2007.

6.3 未分类

• BnB+3D registration: C. Olsson, F. Kahl, and M. Oskarsson, “The registration problem revisited: Optimal solutions from points, lines and planes,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1. IEEE, 2006, pp. 1206–1213.
• Point-to-plane correspondences+lagrangian duality: C. Olsson and A. Eriksson, “Solving quadratically constrained geometrical problems using lagrangian duality,” in 2008 19th International Conference on Pattern Recognition. IEEE, 2008, pp. 1–5.
• Outlier-free+exponential-time methods (e.g., BnB)+mesh registration: C. Olsson, F. Kahl, and M. Oskarsson, “Branch-and-bound methods for euclidean registration problems,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 783–794, 2009.
• F. Wientapper and A. Kuijper, “Unifying algebraic solvers for scaled euclidean registration from point, line and plane constraints,” in Asian Conference on Computer Vision. Springer, 2016, pp. 52–66.
• 3D multimodal registration+SDP+SDR+non-minimal solver+outlier-free: Jesus Briales and Javier Gonzalez-Jimenez. Convex Global 3D Registration with Lagrangian Duality. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
• F. Wientapper, M. Schmitt, M. Fraissinet-Tachet, and A. Kuijper, “A universal, closed-form approach for absolute pose problems,” Computer Vision and Image Understanding, vol. 173, pp. 57–75, 2018.
• Plane-to-plane: A. Segal, D. Haehnel, and S. Thrun, “Generalized-icp.” in Robotics: science and systems, vol. 2, no. 4. IEEE, 2015, pp. 742–749.
• Outlier-free+mesh registration A. Censi, “An ICP variant using a point-to-line metric,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), Pasadena, CA, May 2008.
• Point-to-plane: J. Serafin and G. Grisetti, “Nicp: Dense normal based point cloud registration,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 742–749.
• Rotation+translation+scale: D. P. Paudel, A. Habed, C. Demonceaux, and P. Vasseur. Robust and optimal sum-of-squares-based point-to-plane registration of image sets and structured scenes. In ICCV, pages 2048–2056, 2015.
• Point-to-plane+line-to-plane+local method: Y. Chen and G. Medioni. Object modelling by registration of multiple range images. Image Vis. Comput., 10(3):145–155, 1992.
• Line-to-plane: F. Vasconcelos, J. P. Barreto, and U. Nunes. A Minimal Solution for the Extrinsic Calibration of a Camera and a Laser-Rangefinder. Pattern Anal. Mach. Intell. IEEE Trans., 34(11):2097–2107, nov 2012.
• Line-to-plane: J. Briales and J. Gonzalez-Jimenez. A minimal solution for the calibration of a 2D laser-rangefinder and a camera based on scene corners. In Intell. Robot. Syst. (IROS), 2015 IEEE/RSJ Int. Conf., pages 1891–1896, sep 2015.
• Line-to-plane: J. Briales and J. Gonzalez-Jimenez. A Minimal Closed-form Solution for the Perspective Three Orthogonal Angles (P3oA) Problem: Application To Visual Odometry. J. Math. Imaging Vis., pages 1–18, 2015.
• Point-to-plane: W. Grimson and T. Lozano-Perez. Model-based recognition and localization from sparse range or tactile data. Int. J., 1984.
• Point-to-plane: S. Ramalingam, Y. Taguchi, T. K. Marks, and O. Tuzel. P2Π: A minimal solution for registration of 3D points to 3D planes. In Eur. Conf. Comput. Vis., pages 436–449. Springer, 2010.
• Point-to-plane: S. Ramalingam and Y. Taguchi. A theory of minimal 3D point to 3D plane registration and its generalization. Int. J. Comput. Vis., 2013.
• Outlier-free+closed form solution+global and local solutions+point-to-point+point-to-line+point-to-plane: Lipu Zhou, Shengze Wang, and Michael Kaess. A fast and accurate solution for pose estimation from 3d correspondences. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.

8. Non-central camera system

8.1 Problem formulation

8.2 Relative pose

• Linear solver+17 correspondences: Hongdong Li, Richard Hartley, and Jae-hak Kim. A linear approach to motion estimation using generalized camera models. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2008.
• Minimal solver+6 correspondences: Henrik Stewenius, Magnus Oskarsson, , Kalle Astrom, and David Nister. Solutions to minimal generalized relative pose problems. In Workshop on Omnidirectional Vision in conjunction with ICCV, 2005.
• **SOCP: ** Jae Hak Kim, Richard Hartley, Jan Michael Frahm, and Marc Pollefeys. Visual odometry for non-overlapping views using second-order cone programming. In Asian Conference on Computer Vision, pages 353–362, 2007.
• Branch-and-bound: Jae Hak Kim, Hongdong Li, and Richard Hartley. Motion estimation for nonoverlapping multicamera rigs: Linear algebraic and L∞ geometric solutions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6):1044–1059, 2010.
• **5+1 point: ** Brian Clipp, Jae-Hak Kim, Jan-Michael Frahm, Marc Pollefeys, and Richard Hartley. Robust 6DOF motion estimation for non-overlapping, multi-camera systems. In IEEE Workshop on Applications of Computer Vision, pages 1–8, 2008.
• Antipodal epipolar constraint: John Lim, Nick Barnes, and Hongdong Li. Estimating relative camera motion from the antipodal-epipolar constraint. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(10):1907–1914, 2010.
• Minimal solver+non-holonomic motion: Gim Hee Lee, Friedrich Faundorfer, and Marc Pollefeys. Motion estimation for self-driving cars with a generalized camera. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2746–2753, 2013.
• Minimal solver: Gim Hee Lee, Marc Pollefeys, and Friedrich Fraundorfer. Relative pose estimation for a multi-camera system with known vertical direction. In IEEE Conference on Computer Vision and Pattern Recognition, pages 540–547, 2014.
• Minimal solver: Liu Liu, Hongdong Li, Yuchao Dai, and Quan Pan. Robust and efficient relative pose with a multi-camera system for autonomous driving in highly dynamic environments. IEEE Transactions on Intelligent Transportation Systems, 19(8):2432–2444, 2018.
• Non-minimal solver+local optimization+eigenvalue-based: Laurent Kneip and Hongdong Li. Efficient computation of relative pose for multi-camera systems. In IEEE Conference on Computer Vision and Pattern Recognition, pages 446–453, 2014.
• Non-minimal solver+local optimization+alternating minimization: Joao Campos, Joao R. Cardoso, and Pedro Miraldo. POSEAMM: A unified framework for solving pose problems using an alternating minimization method. In IEEE Interna- tional Conference on Robotics and Automation, pages 3493–3499, 2019.
• Generalized essential matrix+non-minimal solvers+QCQP+SDR+redundant constraints+outlier-free: Ji Zhao, Wanting Xu, and Laurent Kneip. A certifiably globally optimal solution to generalized essential matrix estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.

8.3 Generalized relative pose and scale

• Chris Sweeney, Laurent Kneip, Tobias Hollerer, and Matthew Turk. Computing similarity transformations from only image correspondences. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3305–3313, 2015.
• Laurent Kneip, Chris Sweeney, and Richard Hartley. The generalized relative pose and scale problem: View graph fusion via 2D-2D registration. In IEEE Winter Conference on Applications of Computer Vision, pages 1–9, 2016.

8.4 未分类

• Tim Kazik, Laurent Kneip, Janosch Nikolic, Marc Pollefeys, and Roland Siegwart. Real-time 6D stereo visual odometry with non-overlapping fields of view. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1529–1536, 2012.

9. Wahba problem/rotation search

9.1 Problem formulation

Given two sets of vectors $a_i,b_i\in {\mathbb{R}}^3,i=1,...,N$, the the Wahba problem is formulated as a least squares problem
$$\underset{R\in SO(3)}{\min }\sum _{i=1}^N{w_i}^2||b_i-R{a}_i|{|}^2$$
which computes the best rotation $R$ that aligns vectors $a_i$ and $b_i$, and where ${\left\{w_i^2\right\}}_{i=1}^N$ are (known) weights associated to each pair of measurements. Here $SO(3)\stackrel{.}{=}\left\{R\in {\mathbb{R}}^{3\times 3}:R^{T}R=R{R}^T=I_3,det(R)=1\right\}$ is the 3D Special Orthogonal Group containing proper 3D rotation matrices and $I_d$ denotes the identity matrix of size $d$. This problem is known to be a maximum likelihood estimator for the unknown rotation when the ground-truth correspondences $(a_i,b_i)$ are known and the observations are corrupted with zero-mean isotropic Gaussian noise. In other words, this problem computes an accurate estimate for $R$ when the observations can be written as $b_i=R{a}_i+{ϵ}_i(i=1,...,N)$, where ${ϵ}_i$ is isotropic Gaussian noise.

9.2 Outlier-free

• Wahba problem: Grace Wahba. A least squares estimate of satellite attitude. SIAM review, 7(3):409–409, 1965. 1, 2
• **Orthogonal Procrustes: ** John Gower and Garmt B. Dijksterhuis. Procrustes problems. Procrustes Problems, Oxford Statistical Science Series, 30, 01 2005. 2
• **Orthogonal Procrustes: ** Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31:1–10, 1966. 1, 2
• Outlier-free+point cloud registration+3-point solver+quaternion+closed-form solutions Berthold K. P. Horn. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer., 4(4):629–642, Apr 1987. 2, 3
• Quaternion: F. Landis Markley and John L. Crassidis. Fundamentals of spacecraft attitude determination and control, volume 33. Springer, 2014. 1, 2
• Rotation matrix: Berthold K. P. Horn, Hugh M. Hilden, and Shahriar Negahdaripour. Closed form solution of absolute orientation using orthonormal matrices. J. Opt. Soc. Amer., 5(7):1127–1135, 1988. 1, 2, 6
• Rotation matrix: F. Landis Markley. Attitude determination using vector observations and the singular value decomposition. The Journal of the Astronautical Sciences, 36(3):245–258, 1988. 1, 2
• Rotation matrix: James Richard Forbes and Anton H. J. de Ruiter. LinearMatrix-Inequality-Based solution to Wahba problem. ´ Journal of Guidance, Control, and Dynamics, 38(1):147– 151, 2015. 2
• Rotation matrix: James Saunderson, Pablo A. Parrilo, and Alan S. Willsky. Semidefinite descriptions of the convex hull of rotation matrices. SIAM J. OPTIM., 25(3):1314–1343, 2015. 2
• Rotation matrix: Kourosh Khoshelham. Closed-form solutions for estimating a rigid motion from plane correspondences extracted from point clouds. ISPRS Journal of Photogrammetry and Remote Sensing, 114:78 – 91, 2016. 2
• Image stitching: Jean-Charles Bazin, Yongduek Seo, Richard Hartley, and Marc Pollefeys. Globally optimal inlier set maximization with unknown rotation and focal length. In European Conf. on Computer Vision (ECCV), pages 803–817, 2014. 1, 2
• Motion estimation and 3D reconstruction: Gérard Blais and Martin D. Levine. Registering multiview range data to create 3d computer objects. IEEE Trans. Pattern Anal. Machine Intell., 17(8):820–824, 1995. 1, 2
• Motion estimation and 3D reconstruction: Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5556–5565, 2015. 1, 2
• Local iterative optimization algorithm+anisotropic Gaussian noise: Yang Cheng and John L. Crassidis. A total least-squares estimate for attitude determination. In AIAA Scitech 2019 Forum, page 1176, 2019. 1, 2
• **SDP relaxation+bounded noise: ** Shakil Ahmed, Eric C. Kerrigan, and Imad M. Jaimoukha. A semidefinite relaxation-based algorithm for robust attitude estimation. IEEE Transactions on Signal Processing, 60(8):3942–3952, 2012. 2

9.3 Robust Wahba

9.3.1 Local Methods

• M-estimators: Michael J. Black and Anand Rangarajan. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. Intl. J. of Computer Vision, 19(1):57–91, 1996. 3, 4
• M-estimators: Kirk Mac Tavish and Timothy D. Barfoot. At all costs: A comparison of robust cost functions for camera correspondence outliers. In Computer and Robot Vision (CRV), 2015 12th Conference on, pages 62–69. IEEE, 2015.

9.3.2 Global Methods

• BnB: Richard I. Hartley and Fredrik Kahl. Global optimization through rotation space search. Intl. J. of Computer Vision, 82(1):64–79, 2009.
• BnB+consensus maximization Jean-Charles Bazin, Yongduek Seo, and Marc Pollefeys. Globally optimal consensus set maximization through rotation search. In Asian Conference on Computer Vision, pages 539–551. Springer, 2012. 2, 3, 6, 8, 18
• Consensus maximization+computational geometry: Carl Olsson, Olof Enqvist, and Fredrik Kahl. A polynomialtime bound for matching and registration with outliers. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1–8, 2008. 3
• Consensus maximization+truncated least squares (TLS) cost+computational geometry+triangulation problems: Olof Enqvist, Erik Ask, Fredrik Kahl, and Kalle Åström. Robust fitting for multiple view geometry. In European Conf. on Computer Vision (ECCV), pages 738–751. Springer, 2012.
• Wahba+non-minimal solver+TLS cost+SDP+lagrangian duality+redundant constraints: H. Yang and L. Carlone. A quaternion-based certifiably optimal solution to the Wahba problem with outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.

9.3.3 Outlier-removal Methods

• Sub-optimal: Álvaro Parra Bustos and Tat-Jun Chin. Guaranteed outlier removal for rotation search. In Intl. Conf. on Computer Vision (ICCV), pages 2165–2173, 2015. 1, 2, 3, 6, 8, 18

10. 3D registration

10.1 Simultaneous pose and correspondence

10.1.1 Local methods

• Point cloud registration+ICP+local method Paul J. Besl and Neil D. McKay. A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Machine Intell., 14(2), 1992. 1, 2

• ICP+robust cost function: S. Granger and X. Pennec, “Multi-scale EM-ICP: A fast and robust approach for surface registration,” in European Conf. on Computer Vision (ECCV), 2002.

• ICP+robust cost function: L. Maier-Hein, A. M. Franz, T. R. dos Santos, M. Schmidt, M. Fangerau, H. P. Meinzer, and J. M. Fitzpatrick, “Convergent iterative closest-point algorithm to accomodate anisotropic and inhomogenous localization error,” IEEE Trans. Pattern Anal. Machine Intell., vol. 34, no. 8, pp. 1520–1532, 2012.

• ICP+robust cost function: D. Chetverikov, D. Stepanov, and P. Krsek, “Robust euclidean alignment of 3D point sets: the trimmed iterative closest point algorithm,” Image and Vision Computing, vol. 23, no. 3, pp. 299–309, 2005.

• ICP: Dmitry Chetverikov, Dmitry Svirko, Dmitry Stepanov, and Pavel Krsek. The trimmed iterative closest point algorithm. In Pattern Recognition, 2002. Proceedings. 16th International Conference on, volume 3, pages 545–548. IEEE, 2002.

• ICP: Per Bergstrom and Ove Edlund. Robust registration of surfaces using a refined iterative closest point algorithm with a trust region approach. Numerical Algorithms, 74(3):755–779, 2017.

• ICP: Jeff M Phillips, Ran Liu, and Carlo Tomasi. Outlier robust icp for minimizing fractional rmsd. In 3-D Digital Imaging and Modeling, 2007. 3DIM’07. Sixth International Conference on, pages 427–434. IEEE, 2007

• ICP+robust cost function: S. Kaneko, T. Kondo, and A. Miyamoto, “Robust matching of 3D contours using iterative closest point algorithm improved by Mestimation,” Pattern Recognition, vol. 36, no. 9, pp. 2041–2047, 2003.

• ICP+probabilistic interpretations+point matching+GMM: A. Myronenko and X. Song, “Point set registration: Coherent point drift,” IEEE Trans. Pattern Anal. Machine Intell., vol. 32, no. 12, pp. 2262–2275, 2010.

• ICP+probabilistic interpretations: B. Jian and B. C. Vemuri, “Robust point set registration using gaussian mixture models,” IEEE Trans. Pattern Anal. Machine Intell., vol. 33, no. 8, pp. 1633–1645, 2011.

• 3D registration+SDP relaxation+local method: H. M. Le, T.-T. Do, T. Hoang, and N.-M. Cheung, “SDRSAC: Semidefinite-based randomized approach for robust point cloud registration without correspondences,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 124–133.

• Aiger, D., Mitra, N.J., Cohen-Or, D.: 4-points congruent sets for robust pairwise surface registration. ACM Transactions on Graphics 27(3) (2008)

• Nicolas Mellado, Dror Aiger, and Niloy J Mitra. Super 4pcs fast global pointcloud registration via smart indexing. In Computer Graphics Forum, volume 33, pages 205–215. Wiley Online Library, 2014.

• Drost, B., Ulrich, M., Navab, N., Ilic, S.: Model globally, match locally: Efficient and robust 3D object recognition. In: CVPR (2010)

• **Point cloud registration+Riemannian optimization: ** W. Clark, M. Ghaffari, and A. Bloch, “Nonparametric continuous sensor registration,” arXiv preprint arXiv:2001.04286, 2020.

• ICP variant: R. Sandhu, S. Dambreville, and A. Tannenbaum. Point set registration via particle filtering and stochastic dynamics. IEEE Trans. Pattern Anal. Machine Intell., 32 (8):1459–1473, 2010.

• ICP variant: C. Zhang, S. Du, J. Liu, and J. Xue. Robust 3d point set registration using iterative closest point algorithm with bounded rotation angle. Signal Processing, 120:777–788, 2016.

• Robust ICP+point cloud registration: S. Bouaziz, A. Tagliasacchi, and M. Pauly. Sparse iterative closest point. In ACM Symp. Geom. Process., pages 113–123. Eurographics Association, 2013. 3

• Probabilistic interpretations+point matching+GMM+L2-norm: B. Jian and B. C. Vemuri. A robust algorithm for point set registration using mixture of gaussians. In Intl. Conf. on Computer Vision (ICCV), volume 2, pages 1246–1251. IEEE, 2005.

• Probabilistic interpretations: D. Campbell and L. Petersson. An adaptive data representation for robust point-set registration and merging. In Intl. Conf. on Computer Vision (ICCV), pages 4292–4300, 2015.

• Plane-to-plane: A. Segal, D. Haehnel, and S. Thrun, “Generalized-icp.” in Robotics: science and systems, vol. 2, no. 4. IEEE, 2015, pp. 742–749.

• Outlier-free+mesh registration A. Censi, “An ICP variant using a point-to-line metric,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), Pasadena, CA, May 2008.

• Point-to-plane: J. Serafin and G. Grisetti, “Nicp: Dense normal based point cloud registration,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 742–749

• S. Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In 3DIM, 2001.

• ICP: Z. Zhang, “Iterative point matching for registration of free-form curves and surfaces,” Int. J. Comput. Vis., vol. 13, no. 2, pp. 119–152, 1994.

• D. Landry, F. Pomerleau, and P. Giguere, “CELLO-3D: Estimating the covariance of ICP in the real world,” Proc. IEEE Int. Conf. Robot. Autom., vol. 2019-May, pp. 8190–8196, 2019.

• W. Tabib, C. O Meadhra, and N. Michael, “On-Manifold GMM Registration,” IEEE Robot. Autom. Lett., vol. 3, no. 4, pp. 3805–3812, 2018.

• S. A. Parkison, M. Ghaffari, L. Gan, R. Zhang, A. K. Ushani, and R. M. Eustice, “Boosting Shape Registration Algorithms via Reproducing Kernel Hilbert Space Regularizers,” IEEE Robot. Autom. Lett., vol. 4, no. 4, pp. 4563–4570, 2019.

• M. Magnusson, A. Lilienthal, and T. Duckett, “Scan registration for autonomous mining vehicles using 3D-NDT,” Journal of Field Robotics, vol. 24, no. 10, pp. 803–827, 10 2007.

10.1.2 Global method

• 3D registration+high outlier rate+graph matching+approximate vertex cover O. Enqvist, K. Josephson, and F. Kahl, “Optimal correspondences from pairwise constraints,” in Intl. Conf. on Computer Vision (ICCV), 2009, pp. 1295–1302.

• BnB+Increase the search speed: H. Li and R. Hartley. The 3d-3d registration problem revisited. In International Conference on Computer Vision (ICCV), 2007.

• 3D registration+BnB: Á. Parra Bustos, T. J. Chin, and D. Suter, “Fast rotation search with stereographic projections for 3d registration,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 3930– 3937. BnB $\to$ run in worst-case exponential time

• BnB: T. M. Breuel. Implementation techniques for geometric branch-and-bound matching methods. Comput. Vis. Image Underst., 90(3):258–294, 2003.

• BnB+Increase the search speed+point cloud registration: Jiaolong Yang, Hongdong Li, Dylan Campbell, and Yunde Jia. Go-ICP: A globally optimal solution to 3D ICP pointset registration. IEEE Trans. Pattern Anal. Machine Intell., 38(11):2241–2254, Nov. 2016. least squares objective $\rightarrow $ non-robust outliers

• **MIP: ** G. Izatt, H. Dai, and R. Tedrake, “Globally optimal object pose estimation in point clouds with mixed-integer programming,” in Proc. of the Intl. Symp. of Robotics Research (ISRR), 2017.

• SDP relaxation: H. Maron, N. Dym, I. Kezurer, S. Kovalsky, and Y. Lipman, “Point registration via efficient convex relaxation,” ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1–12, 2016.

• BnB: D. Campbell and L. Petersson. Gogma: Globally-optimal gaussian mixture alignment. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 5685–5694, 2016.

• X. Li, Y. Liu, Y. Wang, C. Wang, M. Wang, and Z. Song. Fast and globally optimal rigid registration of 3d point sets by transformation decomposition. arXiv preprint arXiv:1812.11307, 2019.

• C. Papazov and D. Burschka, “Stochastic global optimization for robust point set registration,” Computer Vision and Image Understanding, vol. 115, no. 12, pp. 1598–1609, 2011.

• Point cloud registration+Geman-McClure cost+GNC: Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In European Conf. on Computer Vision (ECCV), pages 766–782. Springer, 2016. 3, 6

• BnB+rotation+translation: D. Campbell, L. Petersson, L. Kneip, and H. Li, “Globally-optimal inlier set maximisation for simultaneous camera pose and feature correspondence,” in Intl. Conf. on Computer Vision (ICCV), 2017, pp. 1–10.

• R. B. Rusu, N. Blodow, and M. Beetz, “Fast Point Feature Histograms (FPFH) for 3D registration,” in Proc. IEEE Int. Conf. Robot. Autom. IEEE, 5 2009, pp. 3212–3217.

• H. Lei, G. Jiang, and L. Quan, “Fast Descriptors and Correspondence Propagation for Robust Global Point Cloud Registration,” IEEE Trans. Image Process., vol. 26, no. 8, pp. 1–1, 2017.

• Fourier domain: H. Bulow and A. Birk, “Spectral 6DOF registration of noisy 3D range data with partial overlap,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 4, pp. 954–969, 2013.

• Fourier-Mellin transform: ——, “Scale-Free Registrations in 3D: 7 Degrees of Freedom with Fourier Mellin SOFT Transforms,” Int. J. Comput. Vis., vol. 126, no. 7, pp. 731–750, 7 2018.

• Correlation-based registration approaches in the Fourier domain+2D B. Srinivasa Reddy and B. N. Chatterji, “An FFT-based technique for translation, rotation, and scale-invariant image registration,” IEEE Transactions on Image Processing, vol. 5, no. 8, pp. 1266–1271, 1996.

• Correlation-based registration approaches in the Fourier domain+3D C. Wang, X. Jing, and C. Zhao, “Local Upsampling Fourier Transform for accurate 2D/3D image registration,” Computers and Electrical Engineering, vol. 38, no. 5, pp. 1346–1357, 2012.

• Point cloud registration+Decouple scale, rotation, translation: A. Makadia, A. Patterson, and K. Daniilidis, “Fully automatic registration of 3d point clouds,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), vol. 1, 2006, pp. 1297–1304.

• Bernreiter, L., Ott, L., Nieto, J., Siegwart, R., & Cadena, C. (2021). PHASER: A Robust and Correspondence-Free Global Pointcloud Registration. IEEE Robotics and Automation Letters, 6, 855-862.

10.1.3 未分类

• Fredriksson et al. CVPR’16
• Dellaert, F., Seitz, S.M., Thorpe, C.E., Thrun, S.: Structure from motion without correspondence. In: CVPR (2000)
• Makadia, A., Geyer, C., Daniilidis, K.: Correspondence free structure from motion. IJCV (2007)
• P. Ji, H. Li, M. Salzmann, and Y. Dai. Robust motion segmentation with unknown correspondences. In European Conference on Computer Vision (ECCV). 2014.
• J. Bazin, H. Li, I. S. Kweon, C. Demonceaux, P. Vasseur, and K. Ikeuchi. A branch-and-bound approach to correspondence and grouping problems. Pattern Analysis and Machine Intelligence (PAMI), 2013.
• J. Maciel and J. Costeira. Robust point correspondence by concave minimization. Image and Vision Computing (IVC), 2002.
• Graph matching: Marius Leordeanu and Martial Hebert. A spectral technique for correspondence problems using pairwise constraints. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, volume 2, pages 1482–1489. IEEE, 2005.
• Graph matching: D Khue Le-Huu and Nikos Paragios. Alternating direction graph matching. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4914–4922. IEEE, 2017.
• Concave optimization: J. Maciel and J. Costeira, “A global solution to sparse correspondence problems,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 2, pp. 187–199, Feb. 2003.
• BnB: F. Pfeuffer, M. Stiglmayr and K. Klamroth, “Discrete and geometric branch and bound algorithms for medical image registration,” Ann. Operations Res., vol. 196, no. 1, pp. 737–765, 2012.
• BnB: J.-C. Bazin, H. Li, I. S. Kweon, C. P. Vasseur and K. Ikeuchi, “A branch-and-bound approach to correspondence and grouping problems,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 7, pp. 1565–1576, Jul. 2013.
• W. Lian and L. Zhang, “Robust point matching revisited: A concave optimization approach,” in Proc. 12th Eur. Conf. Comput. Vis., 2012, 259–272.
• W. Lian and L. Zhang, “Point matching in the presence of outliers in both point sets: A concave optimization approach,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 352–359.
• M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 1482–1489. IEEE, 2005.
• Uncertainty estimation+sampling-based approaches: J. Nieto, T. Bailey, and E. Nebot, “Scan-SLAM: Combining EKF-SLAM and Scan Correlation,” in Field and Service Robotics. Springer Berlin Heidelberg, 2006, vol. 25, pp. 167–178.
• Uncertainty estimation+closed-form solutions: S. M. Prakhya, L. Bingbing, Y. Rui, and W. Lin, “A closed-form estimate of 3D ICP covariance,” in Proc. 14th IAPR Int. Conf. Mach. Vis. Appl., no. May. IEEE, 5 2015, pp. 526–529.

10.2 Object-model registration

室内场景重建质量的limitations:

1. 几何完整性
2. 小细节
   室内场景重建质量的挑战：
   scan相机观测物体的噪声很多，而且观测是局部的，也就是不完整的

解决方案：object-model 全局位姿优化(registration) or 3D model retrieval，也就是配准相机观测的物体和CAD模型

难点：

1. model上的点非常有结构和光滑，但是scan相机观测物体的噪声很多，而且观测是局部的，也就是不完整的
2. high-level几何结构比较相似，但是low-level几何特征就可以相差很多

• U. Castellani and A. Bartoli. 3D shape registration. 3D Imaging, Analysis, and Applications, 2012.
• Rotation matrix+closed-form solutions: K. Somani Arun, Thomas S. Huang, and Steven D. Blostein. Least-squares fitting of two 3-D point sets. IEEE Trans. Pattern Anal. Machine Intell., 9(5):698–700, 1987.
• Guo, Y., Bennamoun, M., Sohel, F.A., Lu, M., Wan, J.: 3D object recognition in cluttered scenes with local surface features: A survey. PAMI 36(11) (2014)
• Mian, A.S., Bennamoun, M., Owens, R.: Three-dimensional model-based object recognition and segmentation in cluttered scenes. PAMI 28(10) (2006)
• Shin, J., Triebel, R., Siegwart, R.: Unsupervised discovery of repetitive objects. In: ICRA (2010)
• Papazov, C., Haddadin, S., Parusel, S., Krieger, K., Burschka, D.: Rigid 3D geometry matching for grasping of known objects in cluttered scenes. International Journal of Robotics Research 31(4) (2012)
• Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H.J., Davison, A.J.: SLAM++: Simultaneous localisation and mapping at the level of objects. In: CVPR (2013)
• Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: ICRA (2009)
• Granger, S., Pennec, X.: Multi-scale EM-ICP: A fast and robust approach for surface registration. In: ECCV (2002)
• Bylow, E., Sturm, J., Kerl, C., Kahl, F., Cremers, D.: Real-time camera tracking and 3D reconstruction using signed distance functions. In: RSS (2013)
• Point set registration+tightness analysis+optimality certification+outlier-free: José Pedro Iglesias, Carl Olsson, and Fredrik Kahl. Global optimality for point set registration using semidefinite programming. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
• 3D registration+tightness analysis: Kunal N Chaudhury, Yuehaw Khoo, and Amit Singer. Global registration of multiple point clouds using semidefinite programming. SIAM Journal on Optimization, 25(1):468–501, 2015.
• 3D multimodal registration+SDP+SOS+non-minimal solver+anisotropic Gaussian noise: Jesus Briales and Javier Gonzalez-Jimenez. Convex Global 3D Registration with Lagrangian Duality. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. Problem: outlier-frree
• F. Wientapper and A. Kuijper, “Unifying algebraic solvers for scaled euclidean registration from point, line and plane constraints,” in Asian Conference on Computer Vision. Springer, 2016, pp. 52–66.
• BnB+3D registration: C. Olsson, F. Kahl, and M. Oskarsson, “The registration problem revisited: Optimal solutions from points, lines and planes,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1. IEEE, 2006, pp. 1206–1213.
• Outlier-free+exponential-time methods (e.g., BnB)+mesh registration: C. Olsson, F. Kahl, and M. Oskarsson, “Branch-and-bound methods for euclidean registration problems,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 783–794, 2009.
• F. Wientapper, M. Schmitt, M. Fraissinet-Tachet, and A. Kuijper, “A universal, closed-form approach for absolute pose problems,” Computer Vision and Image Understanding, vol. 173, pp. 57–75, 2018.
• Outlier-free+closed form solution+global and local solutions: Lipu Zhou, Shengze Wang, and Michael Kaess. A fast and accurate solution for pose estimation from 3d correspondences. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.
• S. Belongie, J. Malik and J. Puzicha, “Shape matching and object recognition using shape contexts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 4, pp. 509–522, Apr. 2002.

10.3 Outdoor-indoor registration

• A. Cohen, J. L. Schonberger, P. Speciale, T. Sattler, J.-M. Frahm, and M. Pollefeys. Indoor-outdoor 3d reconstruction alignment. In European Conference on Computer Vision (ECCV), 2016.

10.4 Day-night registration

• Image-based localization: H. Zhou, T. Sattler, and D. W. Jacobs. Evaluating local features for day-night matching. In Computer Vision - ECCV 2016 Workshops - Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III, pages 724–736, 2016.
• Aligning structure-from-motion models: F. Radenovic, J. L. Schonberger, D. Ji, J. Frahm, O. Chum, and J. Matas. From dusk till dawn: Modeling in the dark. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5488–5496, 2016.
• Image matching: M. Bansal and K. Daniilidis. Joint spectral correspondence for disparate image matching. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23-28, 2013, pages 2802–2809, 2013.
• Image matching: D. C. Hauagge and N. Snavely. Image matching using local symmetry features. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 206–213, 2012.
• Image matching: J. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal imageto-image translation. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 465–476, 2017.
• Video registration: S. Alletto, G. Serra, and R. Cucchiara. Video registration in egocentric vision under day and night illumination changes. Computer Vision and Image Understanding, 157:274–283, 2017.

10.5 Non-rigid registration

• Nadav Dym, Haggai Maron, and Yaron Lipman. Ds++: a flexible, scalable and provably tight relaxation for matching problems. ACM Transactions on Graphics (TOG), 36(6):184, 2017.
• M Hullin, R Klein, T Schultz, and A Yao. Efficient lifted relaxations of the quadratic assignment problem. 2017.
• Itay Kezurer, Shahar Z Kovalsky, Ronen Basri, and Yaron Lipman. Tight relaxation of quadratic matching. In Computer Graphics Forum, volume 34, pages 115–128. Wiley Online Library, 2015.
• Non-rigid point matching+mixed linear assignment—least square
  problem: H. Chui and A. Rangarajan, “A new algorithm for non-rigid point matching,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2000, vol. 2, pp. 44–51.
• Non-rigid point matching+mixed linear assignment—least square problem: H. Chui and A. Rangarajan, “A new point matching algorithm for non-rigid registration,” Comput. Vis. Image Understanding, vol. 89, no. 2/3, pp. 114–141, 2003.
• Point matching: M. Sofka, G. Yang and C. V. Stewart, “Simultaneous covariance driven correspondence (CDC) and transformation estimation in the expectation maximization framework,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1–8.
• J. Maciel and J. Costeira, “A global solution to sparse correspondence problems,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 2, pp. 187–199, Feb. 2003.
• H. Jiang, M. S. Drew and Z.-N. Li, “Matching by linear programming and successive convexification,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 959–975, Jun. 2007.
• H. Jiang, T.-P. Tian and S. Sclaroff, “Scale and rotation invariant matching using linearly augmented trees,” IEEE Conf. Comput. Vis. Pattern Recog., vol. 37, no. 12, pp. 2558–2572, Dec. 2015.
• H. Jiang and S. X. Yu, “Linear solution to scale and rotation invariant object matching,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 2474–2481.
• Affine invariant+linear embedding algorithm: H. Li, X. Huang and L. He, “Object matching using a locally affine invariant and linear programming techniques,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 2, pp. 411–424, Feb. 2013.
• J. Ma, J. Zhao, J. Tian, Z. Tu and A. L. Yuille, “Robust estimation of nonrigid transformation for point set registration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 2147–2154.
• Weighted sum of Dirac measures: J. Glaunes, A. Trouve and L. Younes, “Diffeomorphic matching of distributions: A new approach for unlabelled point-sets and submanifolds matching,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2004, pp. 712–718.
• GMM: Tsin, Y., Kanade, T.: A correlation-based approach to robust point set registration. In: ECCV (2004).
• Log-exponential function: D. Breitenreicher and C. Schnorr, “Model-based multiple rigid object detection and registration in unstructured range data,” Int. J. Comput. Vis., vol. 92, no. 1, pp. 32–52, Mar. 2011.
• Moments of distributions J. Ho, A. Peter, A. Rangarajan and M.-H. Yang, “An algebraic approach to affine registration of point sets,” in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 1335–1340.
• Y. Deng, A. Rangarajan, S. Eisenschenk and B. C. Vemuri, “A Riemannian framework for matching point clouds represented by the Schrodinger distance transform,” in € Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 3756–3761.
• Graph matching: M. Leordeanu, M. Hebert and R. Sukthankar, “An integer projected fixed point method for graph matching and MAP inference,” in Proc. Neural Inf. Process. Syst., 2009, pp. 1114–1122.
• Graph matching: M. Cho, J. Lee and K. M. Lee, “The reweighted random walks matching algorithm,” in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 492–505.
• Graph matching: M. Cho, J. Sun, O. Duchenne and J. Ponce, “Finding matches in a haystack: A max-pooling strategy for graph matching in the presence of outliers,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 2091–2098.
• Graph matching: O. Duchenne, F. Bach, I.-S. Kweon and J. Ponce, “A tensor-based algorithm for high-order graph matching,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 12, pp. 2383–2395, Dec. 2011.
• Tensor block coordinate ascend method: Q. Ngoc, A. Gautier and M. Hein, “A flexible tensor block coordinate ascent scheme for hypergraph matching,” in Proc. Comput. Vis. Pattern Recog., 2015, pp. 5270–5278.
• Relaxation labeling techniques J.-H. Lee and C.-H. Won, “Topology preserving relaxation labeling for nonrigid point matching,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 2, pp. 427–432, Feb. 2011.
• Relaxation labeling techniques Y. Zheng and D. Doermann, “Robust point matching for nonrigid shapes by preserving local neighborhood structures,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 643–649, Apr. 2006.
• DP: T. S. Caetano and T. Caelli, “A unified formulation of invariant point pattern matching,” in Proc. IEEE 18th Int. Conf. Pattern Recog., 2006, pp. 121–124.
• Fan-shaped graphs: W. Lian and L. Zhang, “Rotation invariant non-rigid shape matching in cluttered scenes,” in Proc. 11th Eur. Conf. Comput. Vis.: Part V, 2010, pp. 506–518.
• W. Lian, L. Zhang and D. Zhang, “Rotation-invariant nonrigid point set matching in cluttered scenes,” IEEE Trans. Image Process., vol. 21, no. 5, pp. 2786–2797, May 2012.

11. 3D shape reconstruction

11.1 Problem formulation

Given $N$ pixel measurements $Z=[z_1,...,z_N]\in {\mathbb{R}}^{2\times N}$ (the 2D landmarks), landmarks), generated from the projection of points belonging to an unknown 3D shape $S\in {\mathbb{R}}^{3\times N}$ onto an image. Further assume the shape $S$ that can be represented as a linear combination of $K$ predefined basis shape $B_k\in {\mathbb{R}}^{3\times N}$, $i.e.S=\sum _{k=1}^K{c}_k{B}_k$, where { c k } k = 1 K \left \{ c_k \right \}_{k=1}^K {ck​}k=1K​ are (unknown) shape coefficients. Then, the generative model of the 2D landmarks reads:
$$z_i=\Pi R\left(\sum _{k=1}^K{c}_k{B}_k\right)+t+{ϵ}_i,i=1,...,N,$$ where $B_{ki}$ denotes $i$-th 3D point on the $k$-th basis shape, ${ϵ}_i\in {\mathbb{R}}^2$ models the measurement noise, and $\Pi$ is the (known) weak perspective projection matrix:
$$\Pi =\left[\begin{array}{c}{s}_{x}00\\ 0s_{y}0\end{array}\right],$$ with $s_x$ and $s_y$ being constants, $R\in SO(3)$ and $t\in {\mathbb{R}}^2$ model the (unknown) rotation and translation of the shape $S$ relative to the camera (only a 2D translation can be estimated). The shape reconstruction problem consists in the joint estimation of the shape parameters { c k } k = 1 K \left \{ c_k \right \}_{k=1}^K {ck​}k=1K​ and the camera pose $(R,t)$.

• 3D shape reconstruction from 2D landmarks: H. Yang and L. Carlone. In perfect shape: Certifiably optimal 3D shape reconstruction from 2D landmarks. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
• 3D pose and shape estimation from 3D keypoint measurements: Shi, J., Yang, H., & Carlone, L. (2021). Optimal Pose and Shape Estimation for Category-level 3D Object Perception. ArXiv, abs/2104.08383.
• Y. Li, L. Gu, and T. Kanade, “Robustly aligning a shape model and its application to car alignment of unknown pose,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 9, pp. 1860–1876, 2011.

12. Rotation averaging/synchronization

Rotation averaging, a.k.a. multiple rotation averaging or SO(3) synchronisation, is the problem of estimating absolute rotations (orientations w.r.t. a common coordinate system) from a set of distinct estimated relative rotation measurements.

12.1 Problem formulation

The input to rotation averaging is a set of noisy relative rotations $\left\{{\tilde{R}}_{ij}\right\}$, where each ${\tilde{R}}_{ij}$ is a measurement of the orientation difference between cameras $i$ and $j$ which overlap in view. From the relative rotations, rotation averaging aims to recover the absolute rotations ${\left\{{\tilde{R}}_i\right\}}_{i=1}^N$ which represent the orientations of the cameras. In the ideal case where there is no noise in the relative rotations $\left\{{\tilde{R}}_{ij}\right\}$, $${\tilde{R}}_{ij}=R_j{R}_i^T$$ The input relative rotations $\left\{{\tilde{R}}_{ij}\right\}$ define a camera graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where V = { 1 , . . . , n } \mathcal{V}=\left \{ 1,...,n \right \} V={1,...,n} is the set of cameras, and $(i,j)\in \mathcal{E}$ is an edge in $\mathcal{G}$ if the relative rotation ${\tilde{R}}_{ij}$ between cameras $i$ and $j$ is measured.
However, in the presence of noise, Rotation averaging is usually posed as a nonlinear optimization problem in a least-metric sense with nonconvex domain
$$\underset{R_1,...,R_n\in SO(3)}{\min }\sum _{(i,j)\in \mathcal{E}}d(R_j{R}_i^T,{\tilde{R}}_{ij}{)}^p$$ where $d:SO(3)\times SO(3)\mapsto \mathbb{R}$ is a distance function that measures the deviation from ${\tilde{R}}_{ij}=R_j{R}_i^T$ based on measured and estimated quantities.

12.2 Single rotation averaging

• Single rotation averaging: Richard Hartley, Khurrum Aftab, and Jochen Trumpf. L1 rotation averaging using the Weiszfeld algorithm. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3041–3048. IEEE, 2011.
• Single rotation averaging: Seong Hun Lee and Javier Civera. Robust single rotation averaging. CoRR, abs/2004.00732, 2020.
• • Single rotation averaging: R. Hartley, J. Trumpf, and Y. Dai. Rotation averaging and weak convexity. In International Symposium on Mathematical Theory of Networks and Systems, 2010.

12.2 Multiple rotation averaging

• Multiple rotation averaging: Johan Fredriksson and Carl Olsson. Simultaneous multiple rotation averaging using lagrangian duality. In Asian Conf. on Computer Vision (ACCV), pages 245–258. Springer, 2012.
• Multiple rotation averaging: Mica Arie-Nachimson, Shahar Z Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, and Ronen Basri. Global motion estimation from point matches. In International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, 2012.
• Y. Zhong and N. Boumal. Near-optimal bounds for phase synchronization. ArXiv e-prints, Mar. 2017.
• Multiple rotation averaging+tightness analysis+optimality certification+SDP+SOS+non-minimal solver+outlier-free: A. Eriksson, C. Olsson, F. Kahl, and T.-J. Chin. Rotation averaging and strong duality. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
• Multiple Rotation averaging: Anders Eriksson, Carl Olsson, Fredrik Kahl, and Tat-Jun Chin. Rotation averaging with the chordal distance: Global minimizers and strong duality. IEEE Trans. Pattern Anal. Mach. Intell., pages 1–1, 2019.
• Survey: R. Hartley, J. Trumpf, Y. Dai, and H. Li. Rotation averaging. IJCV, 103(3):267–305, 2013.
• Multiple Rotation averaging+IRLS: A. Chatterjee and V. M. Govindu. Efficient and robust large-scale rotation averaging. In Intl. Conf. on Computer Vision (ICCV), pages 521–528, 2013.
• Multiple Rotation averaging: V. Govindu. Combining two-view constraints for motion estimation. In IEEE Conference on Computer Vision and Pattern Recognition, 2001.
• Multiple Rotation averaging+linear method+non-sequential method: D. Martinec and T. Pajdla, “Robust rotation and translation estimation in multiview reconstruction,” in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2007, pp. 1–8.
• Nicolas Boumal, Amit Singer, P-A Absil, and Vincent D Blondel. Cramer Rao bounds for synchronization of rotations. Information and Inference: A Journal of the IMA, 3(1):1–39, 2014.
• Kyle Wilson, David Bindel, and Noah Snavely. When is rotations averaging hard? In ECCV, 2016.
• Venu Madhav Govindu. Combining two-view constraints for motion estimation. In CVPR, 2001.
• Maher Moakher. Means and averaging in the group of rotations. SIAM SIMAX, 24(1):1–16, 2002.
• Venu Madhav Govindu. Lie-algebraic averaging for globally consistent motion estimation. In CVPR, 2004.
• Roberto Tron, Bijan Afsari, and Rene Vidal. Intrinsic consensus on SO(3) with almost-global convergence. In IEEE CDC, 2012.
• Spectral decomposition methods: Federica Arrigoni, Beatrice Rossi, and Andrea Fusiello. Spectral synchronization of multiple views in SE(3). SIAM SIMAX, 9(4):1963–1990, 2016.
• Multiple Rotation averaging+spectral decomposition methods: Federica Arrigoni, Beatrice Rossi, Pasqualina Fragneto, and Andrea Fusiello. Robust synchronization in so (3) and se (3) via low-rank and sparse matrix decomposition. CVIU, 174:95–113, 2018.
• F. Arrigoni, L. Magri, B. Rossi, P. Fragneto, and A. Fusiello. Robust absolute rotation estimation via low-rank and sparse matrix decomposition. In International Conference on 3D Vision, 2014.
• L1-norm+convex relaxation+semidefinite method: Lanhui Wang and Amit Singer. Exact and stable recovery of rotations for robust synchronization. Information and Inference: A Journal of the IMA, 2(2):145–193, 2013.
• Survey+semidefinite method: Amit Singer. Angular synchronization by eigenvectors and semidefinite programming. ACHA, 30(1):20–36, 2011.
• Roberto Tron, Xiaowei Zhou, and Kostas Daniilidis. A survey on rotation optimization in structure from motion. In CVPRW, 2016.
• Multiple Rotation averaging: Frank Dellaert, David M Rosen, Jing Wu, Robert Mahony, and Luca Carlone. Shonan rotation averaging: Global optimality by surfing SO§n. In ECCV, 2020.
• Riemannian staircase: Yulun Tian, Kasra Khosoussi, David M Rosen, and Jonathan P How. Distributed certifiably correct pose-graph optimization. arXiv preprint arXiv:1911.03721, 2019.
• Alvaro Parra, Shin-Fang Chng, Tat-Jun Chin, Anders Eriksson, Ian Reid (2021). Rotation Coordinate Descend for Fast Globally Optimal Rotation Averaging. Computer Vision and Pattern Recognition (CVPR).
• F. Kahl and R. Hartley. Multiple-view geometry under the $L_{\infty }$-norm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(9):1603 1617, 2008.
• L. Carlone, R. Tron, K. Daniilidis, and F. Dellaert. Initialization techniques for 3D SLAM: A survey on rotation estimation and its use in pose graph optimization. In International Conference on Robotics and Automation, 2015.
• V. Govindu. Robustness in motion averaging. In European Conference on Computer Vision, 2006.
• Non-sequential method: O. Enqvist, F. Kahl, and C. Olsson. Non-sequential structure from motion. In International Workshop on Omnidirectional Vision, Camera Networks and Non-Classical Cameras, 2011.
• Rotation averaging+QCQP+tight SDP relaxation: N. Boumal. A Riemannian low-rank method for optimization over semidefinite matrices with block-diagonal constraints. arXiv preprint arXiv:1506.00575, 2015.
• Rotation averaging+QCQP+tight SDP relaxation: A. Bandeira, N. Boumal, and A. Singer. Tightness of the maximum likelihood semidefinite relaxation for angular synchronization. Mathematical Programming, 2016.
• Multiple Rotation averaging: Avishek Chatterjee and Venu Madhav Govindu. Robust relative rotation averaging. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):958–972, 2018.
• Multiple rotation averaging: Pulak Purkait, Tat-Jun Chin, and Ian Reid. NeuRoRA: Neural robust rotation averaging. CoRR, abs/1912.04485, 2020.
• Jesus Briales and Javier Gonzalez-Jimenez. Cartan-sync: Fast and
  global se (d)-synchronization. IEEE Robotics and Automation Letters,
  2(4):2127–2134, 2017.

13. Pose graph estimation/optimization

Camera orientations and positions are jointly optimized.

• Pose graph optimization+optimality certification+QCQP+SDP+SOS++lagrangian duality: L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar pose graph optimization: Duality, optimal solutions, and verification. IEEE Trans. Robotics (T-RO), 32(3):545–565, 2016.
• Pose graph estimation+lagrangian duality+convex relaxation+outlier-free: L. Carlone and F. Dellaert, “Duality-based verification techniques for 2D SLAM,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), 2015, pp. 4589–4596.
• PGO+outlier rejection: N. Sünderhauf and P. Protzel, “Switchable constraints for robust pose graph SLAM,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2012.
• PGO+outlier rejection: N. Sunderhauf and P. Protzel. Towards a robust back-end for pose graph slam. In IEEE Intl. Conf. on Robotics and Automation (ICRA), pages 1254–1261. IEEE, 2012.
• PGO+outlier rejection: G. Zioutas and A. Avramidis. Deleting outliers in robust regression with mixed integer programming. Acta Mathematicae Applicatae Sinica, 21(2):323–334, 2005.
• PGO+outlier rejection: Y. Latif, C. D. C. Lerma, and J. Neira. Robust loop closing over time. In Robotics: Science and Systems (RSS), 2012.
• PGO+outlier rejection: M. C. Graham, J. P. How, and D. E. Gustafson. Robust incremental slam with consistency-checking. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 117–124, Sept 2015.
• PGO+outlier rejection: L. Carlone, A. Censi, and F. Dellaert. Selecting good measurements via l1 relaxation: a convex approach for robust estimation over graphs. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2014.
• Consensus algorithm: R. Tron and R. Vidal. Distributed 3-D localization of camera sensor networks from 2-D image measurements. IEEE Transactions on Automatic Control, 59(12):3325–3340, 2014.
• PGO+gradient method E. Olson, J. Leonard, and S. Teller, “Fast iterative alignment of pose graphs with poor initial estimates,” in Proc. IEEE Int. Conf. Robot. Autom., May 2006, pp. 2262–2269.
• Pose graph optimization+tightness analysis+QCQP+SDP+SOS+non-minimal solver+riemannian staircase: D.M. Rosen, L. Carlone, A.S. Bandeira, and J.J. Leonard. SE-Sync: a certifiably correct algorithm for synchronization over the Special Euclidean group. Intl. J. of Robotics Research, 2018.
• Pose graph optimization+lagrangian duality+QCQP+SDP relaxations+non-minimal solver+outlier-frree L. Carlone, D. Rosen, G. Calafiore, J. Leonard, and F. Dellaert, “Lagrangian duality in 3D SLAM: Verification techniques and optimal solutions,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2015, pp. 125–132.
• Pose graph optimization+sparse bounded-degree variant of SOS relaxation J. G. Mangelson, J. Liu, R. M. Eustice, and R. Vasudevan, “Guaranteed globally optimal planar pose graph and landmark SLAM via sparse-bounded sums-of-squares programming,” in IEEE Intl. Conf. on Robotics and Automation (ICRA). IEEE, 2019, pp. 9306–9312.
• Huber+l1-norm+convex+non-minimal solver: L. Carlone and G. Calafiore. Convex relaxations for pose graph optimization with outliers. IEEE Robotics and Automation Letters (RA-L), 3(2):1160–1167, 2018.
• L. Carlone, R. Tron, K. Daniilidis, and F. Dellaert. Initialization techniques for 3D SLAM: A survey on rotation estimation and its use in pose graph optimization. In International Conference on Robotics and Automation, 2015.
• Pose Graph Optimization+Lagrangian duality+QCQP+SDP relaxation+outlier-free: J. Briales and J. Gonzalez-Jimenez. Fast global optimality verification in 3D SLAM. In International Conference on Intelligent Robots and Systems, 2016.
• Lagrangian duality+QCQP+SDP relaxation: J. Briales and J. Gonzalez-Jimenez. Initialization of 3D Pose Graph Optimization using Lagrangian duality. In Int. Conf. on Robotics and Automation (ICRA). IEEE, 2017.
• **PGO+M-estimators+outlier mitigation: ** P. Agarwal, G. D. Tipaldi, L. Spinello, C. Stachniss, and W. Burgard. Robust map optimization using dynamic covariance scaling. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2013.
• PGO+M-estimators+outlier mitigation: E. Olson and P. Agarwal, “Inference on networks of mixtures for robust robot mapping,” in Robotics: Science and Systems (RSS), July 2012.
• PGO+M-estimators+outlier mitigation+local solver: J.J. Casafranca, L.M. Paz, and P. Pinies. A back-end l 1 norm ´ based solution for factor graph slam. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 17–23. IEEE, 2013.
• PGO+outlier mitigation+local solver: G. H. Lee, F. Fraundorfer, and M. Pollefeys. Robust pose-graph loop closures with expectation-maximization. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2013.
• PGO+outlier mitigation: M. Pfingsthorn and A. Birk, “Generalized graph SLAM: Solving local and global ambiguities through multimodal and hyperedge constraints,” Intl. J. of Robotics Research, vol. 35, no. 6, pp. 601–630, 2016.

14. 未归类

• Global translations: Kyle Wilson and Noah Snavely. Robust global translations with 1DSfM. In Eur. Conf. Comput. Vis., pages 61–75, 2014.
• Global translations: V. M. Govindu, “Combining two-view constraints for motion estimation,” in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2001, pp. 218–225.
• V. M. Govindu, “Lie-algebraic averaging for globally consistent motion estimation,” in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2004, pp. I-684–I-691.
• G. C. Sharp, S. W. Lee, and D. K. Wehe, “Multiview registration of 3D scenes by minimizing error between coordinate frames,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1037–1050, Aug. 2004.
• M. B. Horowitz, N. Matni, and J. W. Burdick, “Convex relazations of se(2) and se(3) for visual pose estimation,” in Proc. IEEE Int. Conf. Robot. and Autom., pp. 1148–1154, 2014.
• Point cloud registration+Decouple scale, rotation, translation: Y. Liu, C. Wang, Z. Song, and M. Wang, “Efficient global point cloud registration by matching rotation invariant features through translation search,” in European Conf. on Computer Vision (ECCV), September 2018.
• 3D registration+BnB: A. Parra Bustos, T.-J. Chin, F. Neumann, T. Friedrich, and M. Katzmann, “A practical maximum clique algorithm for matching with pairwise constraints,” arXiv preprint arXiv:1902.01534, 2019.
• Physics-based: V. Golyanik, S. Aziz Ali, and D. Stricker, “Gravitational approach for point set registration,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5802–5810.
• Physics-based: P. Jauer, I. Kuhlemann, R. Bruder, A. Schweikard, and F. Ernst, “Efficient registration of high-resolution feature enhanced point clouds,” IEEE Trans. Pattern Anal. Machine Intell., vol. 41, no. 5, pp. 1102– 1115, 2018.
• Physics-based: H. Yang, “A dynamical perspective on point cloud registration,” arXiv preprint arXiv:2005.03190, 2020.
• C. S. Chen, Y. P. Hung, and J. B. Cheng. RANSAC-based DARCES: A new approach to fast automatic registration of partially overlapping range images. IEEE Trans. Pattern Anal. Machine Intell., 21(11):1229–1234, 1999.
• P. Meer, D. Mintz, A. Rosenfeld, and D. Y. Kim. Robust regression methods for computer vision: A review. Intl. J. of Computer Vision, 6(1):59–70, Apr 1991.
• K. Mac Tavish and T. D. Barfoot. At all costs: A comparison of robust cost functions for camera correspondence outliers. In Computer and Robot Vision (CRV), 2015 12th Conference on, pages 62–69. IEEE, 2015.
• Global optimization: R. Hartley and F. Kahl. Optimal Algorithms in Multiview Geometry. In Comput. Vis. ACCV 2007, pages 13–34. Springer Berlin Heidelberg, Berlin, Heidelberg.
• Triangulate the points: Lai Kang, Lingda Wu, and Yee-Hong Yang. Robust multiview L2 triangulation via optimal inlier selection and 3D structure refinement. Pattern Recognition, 47(9):2974–2992, 2014.
• Triangulate the points: Seong Hun Lee and Javier Civera. Robust uncertainty-aware multiview triangulation. CoRR, abs/2008.01258, 2020.

Datasets

Relative rotation: Kyle Wilson and Noah Snavely. Robust global translations with 1DSfM. In Eur. Conf. Comput. Vis., pages 61–75, 2014.

• internal camera calibration and radial distortion data
• SIFT feature tracks and their image coordinates
• estimated relative rotations
• reconstruction made with Bundler, consisting of the camera poses and a sparse set of 3D points

Relative pose estimation: Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multiview stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3260–3269, 2017.
indoor and outdoor scenes
ground-truth camera poses
intrinsic camera calibration

Relative pose estimation: Malaga J.-L. Blanco-Claraco, F.-A. Moreno-Duenas, and J. G. Jimenez. The malaga urban dataset: High-rate stereo and lidar in a realistic urban scenario. International Journal of Robotics Research (IJRR), 2014.
https://www.mrpt.org/MalagaUrbanDataset

Relative pose estimation: KITTI Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012.
http://www.cvlibs.net/datasets/kitti

Relative pose estimation: ETH3D Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with highresolution images and multi-camera videos. In Computer Vision and Pattern Recognition (CVPR), 2017.
https://www.eth3d.net/