本专栏参考崔华坤的《VINS论文推导及代码解析》的PDF,按照这个框架,对VINS-Mono的原理进行详解,并附加代码解析~
总体内容如下:
VINS-Mono原理推导与代码解析(五)初始化
VINS-Mono原理推导与代码解析(六)边缘化Maginalization和FEJ
VINS-Mono原理推导与代码解析(七)闭环检测与优化
VINS-Mono原理推导与代码解析(八)其他补充~
1.状态向量
状态向量共包括滑动窗口内的n+1个所有相机的状态(位置、速度、朝向、加速度bias和陀螺仪bias)、Camera到IMU的外参、m+1个3D点的逆深度:
2.目标函数
(19)
三个残差项分别是边缘化的先验信息、IMU测量残差、视觉的重投影残差。这三个残差都是用码是距离来表示的。
根据《十四讲》中高斯-牛顿法,若要计算目标函数的最小值,可以理解为,当优化变量有一个增量后,目标函数值最小,以IMU残差为例,可写成如下形式:
其中为误差项关于所有状态向量(即优化变量)X的Jacobian,将上式展开并令关于的导数为0,可得增量的计算公式:
那么,对于公式(19)的整体目标函数的整体增量方程可写成:
上式中,为IMU预积分噪声项的协方差,为visual观测的噪声协方差。当IMU的噪声协方差越大时,其信息矩阵将越小,意味着该IMU观测越不可信,换句话说,因IMU噪声较大,越不可信IMU预积分数据,而更加相信visual观测。这里的IMU和visual协方差的绝对值没有意义,考虑的是两者的相对性。
将上式继续化简为:
其中,和为Hessian矩阵,上述方程称之为增量方程。
上面的Jacobian虽然是误差项r关于状态向量X的一阶导,但是在具体求解时,通常采用扰动的方式进行计算,如下所示,其中是状态向量的微小扰动,并非上面求解的增量:
3.IMU约束
(1)残差
IMU残差为两帧之间的PVQ和bias的变化量的差
(20)
其中各增量关于bias的Jacobian可从公式(16)的大Jacobian中的相应位置获得。
(16)
然后看看代码中对IMU残差定义的实现,在vins_estimator/src/factor/integration_base.h的evaluate中
Eigen::Matrix<double, 15, 1> evaluate(const Eigen::Vector3d &Pi, const Eigen::Quaterniond &Qi, const Eigen::Vector3d &Vi, const Eigen::Vector3d &Bai, const Eigen::Vector3d &Bgi,
const Eigen::Vector3d &Pj, const Eigen::Quaterniond &Qj, const Eigen::Vector3d &Vj, const Eigen::Vector3d &Baj, const Eigen::Vector3d &Bgj)
{
Eigen::Matrix<double, 15, 1> residuals;
Eigen::Matrix3d dp_dba = jacobian.block<3,3>(O_P, O_BA);
Eigen::Matrix3d dp_dbg = jacobian.block<3,3>(O_P, O_BG);
Eigen::Matrix3d dq_dbg = jacobian.block<3,3>(O_R, O_BG);
Eigen::Matrix3d dv_dba = jacobian.block<3,3>(O_V, O_BA);
Eigen::Matrix3d dv_dbg = jacobian.block<3,3>(O_V, O_BG);
Eigen::Vector3d dba = Bai - linearized_ba;
Eigen::Vector3d dbg = Bgi = linearized_bg;
Eigen::Quaterniond corrected_delta_q = delta_q * Utility::deltaQ(dq_dbg * dbg);
Eigen::Vector3d corrected_delta_v = delta_v + dv_dba * dba + dv_dbg * dbg;
Eigen::Vector3d corrected_delta_p = delta_p + dp_dba * dba + dp_dbg * dbg;
// 残差
residuals.block<3,1>(O_P, 0) = Qi.inverse() * (0.5 * G * sum_dt * sum_dt + Pj - P_i - Vi * sum_dt) - corrected_delta_p;
residuals.block<3,1>(O_R, 0) = 2 * (corrected_delta_q.inverse() * (Qi.inverse() * Qj)).vec();
residuals.block<3,1>(O_V, 0) = Qi.inverse() * (G * sum_dt + Vj - Vi) - corrected_delta_v;
residuals.block<3,1>(O_BA, 0) = Baj - Bai;
residuals.block<3,1>(O_BG, 0) = Bgj - Bgi;
return residuals;
}
(2)优化变量
(3)Jacobian
计算Jacobian时,残差对应的求偏导对象为上面的优化变量,但是计算时采用扰动方式计算,即扰动为。
然后看看代码实现部分~在vins_estimator/src/factor/imu_factor.h中
if(jacobians[0])
{
Eigen::Map<Eigen::Matrix<double, 15, 7, Eigen::RowMahjor>> jacobian_pose_i(jacobians[0]);
jacobian_pose_i.setZero();
jacobian_pose_i.block<3,3>(O_P, O_P) = -Qi.inverse().toRotationMatirx();
jacobian_pose.i.block<3,3>(O_P, O_R) = Utility::skewSymmetric(Qi.inverse() * (0.5 * G * sum_dt * sum_dt * Pj - Pi - Vi * sum_dt));
#if 0
jacobian_pose_i.block<3, 3>(O_R, O_R) = -(Qj.inverse() * Qi).toRotationMatrix();
#else
Eigen::Quaterniond corrected_delta_q = pre_integration->delta_q * Utility::deltaQ(dq_dbg * (Bgi - pre_integration->linearized_bg));
jacobian_pose_i.block<3, 3>(O_R, O_R) = -(Utility::Qleft(Qj.inverse() * Qi) * Utility::Qright(corrected_delta_q)).bottomRightCorner<3, 3>();
#endif
jacobian_pose_i.block<3,3>(O_V, O_R) = Utility::skewSymmetric(Qi.inverse() * (G * sum_dt + Vj - Vi));
jacobian_pose_i = sqrt_info * jacobian_pose_i;
}
if(jacobians[1])
{
Eigen::Map<Eigen::Matrix<double, 15, 9, Eigen::RowMajor>> jacobian_speedbias_i(jacobians[1]);
jacobian_speedbias_i.setZero();
jacobian_speedbias_i.block<3, 3>(O_P, O_V - O_V) = -Qi.inverse().toRotationMatrix() * sum_dt;
jacobian_speedbias_i.block<3, 3>(O_P, O_BA - O_V) = -dp_dba;
jacobian_speedbias_i.block<3, 3>(O_P, O_BG - O_V) = -dp_dbg;
#if 0
jacobian_speedbias_i.block<3, 3>(O_R, O_BG - O_V) = -dq_dbg;
#else
jacobian_speedbias_i.block<3, 3>(O_R, O_BG - O_V) = -Utility::Qleft(Qj.inverse() * Qi * pre_integration->delta_q).bottomRightCorner<3, 3>() * dq_dbg;
#endif
jacobian_speedbias_i.block<3, 3>(O_V, O_V - O_V) = -Qi.inverse().toRotationMatrix();
jacobian_speedbias_i.block<3, 3>(O_V, O_BA - O_V) = -dv_dba;
jacobian_speedbias_i.block<3, 3>(O_V, O_BG - O_V) = -dv_dbg;
jacobian_speedbias_i.block<3, 3>(O_BA, O_BA - O_V) = -Eigen::Matrix3d::Identity();
jacobian_speedbias_i.block<3, 3>(O_BG, O_BG - O_V) = -Eigen::Matrix3d::Identity();
jacobian_speedbias_i = sqrt_info * jacobian_speedbias_i;
}
if(jacobians[2])
{
Eigen::Map<Eigen::Matrix<double, 15, 7, Eigen::RowMajor>> jacobian_pose_j(jacobians[2]);
jacobian_pose_j.setZero();
jacobian_pose_j.block<3, 3>(O_P, O_P) = Qi.inverse().toRotationMatrix();
#if 0
jacobian_pose_j.block<3, 3>(O_R, O_R) = Eigen::Matrix3d::Identity();
#else
Eigen::Quaterniond corrected_delta_q = pre_integration->delta_q * Utility::deltaQ(dq_dbg * (Bgi - pre_integration->linearized_bg));
jacobian_pose_j.block<3, 3>(O_R, O_R) = Utility::Qleft(corrected_delta_q.inverse() * Qi.inverse() * Qj).bottomRightCorner<3, 3>();
#endif
jacobian_pose_j = sqrt_info * jacobian_pose_j;
}
if(jacobians[3])
{
Eigen::Map<Eigen::Matrix<double, 15, 9, Eigen::RowMajor>> jacobian_speedbias_j(jacobians[3]);
jacobian_speedbias_j.setZero();
jacobian_speedbias_j.block<3, 3>(O_V, O_V - O_V) = Qi.inverse().toRotationMatrix();
jacobian_speedbias_j.block<3, 3>(O_BA, O_BA - O_V) = Eigen::Matrix3d::Identity();
jacobian_speedbias_j.block<3, 3>(O_BG, O_BG - O_V) = Eigen::Matrix3d::Identity();
jacobian_speedbias_j = sqrt_info * jacobian_speedbias_j;
}
代码中residual还乘以一个信息矩阵sqrt_info,这是因为真正的优化项是Mahalanobis距离:,P是协方差,又因为Ceres只接受最小二乘优化,也就是,所有把做LLT分解,即,则有:
令作为新的优化误差,这样就能用Ceres求解了。Mahalanobis距离其实相当于残差加权,协方差大的加权小,协方差小的加权大,着重优化那些比较确定的残差。
(4)协方差
(17)
IMU协方差P为公式(17)推导的IMU预积分中迭代出来的IMU增量误差的协方差。
4.视觉约束
(1)残差
视觉残差是重投影误差,对于第l个路标点P,将P从第一次观看到它的第i个相机坐标系,转换到当前的第j个相机坐标系下的像素坐标系,可定义视觉误差项为:
(25)
其中,为第l个路标点在第j个相机归一化相机坐标系中的观察到的坐标:
(26)
是估计第l个路标点在第j个相机归一化相机坐标系中可能坐标:
(27)
(2)优化变量
(3)Jacobian
根据视觉残差公式,可以得到相对于各优化变量的Jacobian。
代码实现在vins_estimator/src/factor/projection_factor.cpp中
if(jacobians[0])
{
Eigen::Map<Eigen::Matrix<double, 2, 7, Eigen::RowMajor>> jacobian_pose_i(jacobians[0]);
Eigen::Matrix<double, 3, 6> jaco_i;
jaco_i.leftCols<3>() = ric.transpose() * Rj.transpose();
jaco_i.rightCols<3>() = ric.transpose() * Rj.transpose() * Ri * -Utility::skewSymmetric(pts_imu_i);
jacobian_pose_i.leftCols<6>() = reduce * jaco_i;
jacobian_pose_i.rightCols<1>().setZero();
}
if(jacobians[1])
{
Eigen::Map<Eigen::Matrix<double, 2, 7, Eigen::RowMajor>> jacobian_pose_j(jacobians[1]);
Eigen::Matrix<double, 3, 6> jaco_j;
jaco_j.leftCols<3>() = ric.transpose() * -Rj.transpose();
jaco_j.rightCols<3>() = ric.transpose() * Utility::skewSymmetric(pts_imu_j);
jacobian_pose_j.leftCols<6>() = reduce * jaco_j;
jacobian_pose_j.rightCols<1>().setZero();
}
if(jacobians[2])
{
Eigen::Map<Eigen::Matrix<double, 2, 7, Eigen::RowMajor>> jacobian_ex_pose(jacobians[2]);
Eigen::Matrix<double, 3, 6> jaco_ex;
jaco_ex.leftCols<3>() = ric.transpose() * (Rj.transpose() * Ri - Eigen::Matrix3d::Identity());
Eigen::Matrix3d tmp_r = ric.transpose() * Rj.transpose() * Ri * ric;
jaco_ex.rightCols<3>() = -tmp_r * Utility::skewSymmetric(pts_camera_i) + Utility::skewSymmetric(tmp_r * pts_camera_i) +
Utility::skewSymmetric(ric.transpose() * (Rj.transpose() * (Ri * tic + Pi - Pj) - tic));
jacobian_ex_pose.leftCols<6>() = reduce * jaco_ex;
jacobian_ex_pose.rightCols<1>().setZero();
}
if(jacobians[3])
{
Eigen::Map<Eigen::Vector2d> jacobian_feature(jacobians[3]);
#if 1
jacobian_feature = reduce * ric.transpose() * Rj.transpose() * Ri * ric * pts_i * -1.0 / (inv_dep_i * inv_dep_i);
#else
jacobian_feature = reduce * ric.transpose() * Rj.transpose() * Ri * ric * pts_i;
#endif
}
(4)协方差
视觉约束的噪声协方差与标定相机内参时的重投影误差有关,这里取1.5个像素,对应到归一化相机平面上的协方差矩阵需除以焦距f,则信息矩阵等于协方差矩阵的逆,为: