EVO: Python package for the evaluation of odometry and SLAM

This packages provides executables and a small library for handling, evaluating and comparing the trajectory output of odometry and SLAM algorithms.

Supported trajectory formats:

‘TUM’ trajectory files
‘KITTI’ pose files
‘EuRoC MAV’ (.csv groundtruth and TUM trajectory file)
ROS bagfile with geometry_msgs/PoseStamped, geometry_msgs/TransformStamped, geometry_msgs/PoseWithCovarianceStamped or nav_msgs/Odometry topics

Why?

evo has several advantages over other public benchmarking tools:

• common tools for different formats
• algorithmic options for association, alignment, scale adjustment for monocular SLAM etc.
• flexible options for output, plotting or export (e.g. LaTeX plots or Excel tables)
• a powerful, configurable CLI that can cover many use cases
• modular core and tools libraries for custom extensions
• faster than other established Python-based tools (see here)

What it’s not: a 1-to-1 re-implementation of a particular evaluation protocol tailored to a specific dataset.

Installation / Upgrade

Installation is easy-peasy if you’re familiar with this: https://xkcd.com/1987/#

Python 3.4+ and Python 2.7 are both supported. If you want to use the ROS bagfile interface, first check which Python version is used by your ROS installation and install accordingly. You might also want to use a virtual environment.

From PyPi

If you just want to use the executables of the latest release version, the easiest way is to run:

准备工作：python多版本管理，把python升级

pip install evo --upgrade --no-binary evo

This will download the package and its dependencies from PyPi and install or upgrade them. Depending on your OS, you might be able to use pip2 or pip3 to specify the Python version you want. Tab completion for Bash terminals is supported via the argcomplete package on most UNIX systems - open a new shell after the installation to use it (without --no-binary evo the tab completion might not be installed properly). If you want, you can subscribe to new releases via https://libraries.io/pypi/evo.

From Source

Run this in the repository’s base folder:

pip install . --upgrade --no-binary evo
Dependencies
Python packages

evo has the following dependencies that are automatically resolved during installation:

numpy, matplotlib, scipy, pandas, seaborn, natsort, argcomplete, colorama, pygments, enum34 (only Python 2.7)

PyQt5 / PyQt4 (optional)

PyQt5 will give you the enhanced GUI for plot figures from the “Qt5Agg” matplotlib backend (otherwise: “TkAgg”). If PyQt5 is already installed when installing this package, it will be used as a default (see evo_config show). To change the plot backend afterwards, run evo_config set plot_backend Qt5Agg. PyQt4 can be set equivalently.

ROS (optional)

All ROS-related features like reading bagfiles require a ROS installation, see here. We tested this package with ROS Melodic, Kinetic and Indigo.

Command Line Interface
After installation with setup.py or from pip, the following executables can be called globally from your command-line:

Metrics:

evo_ape - absolute pose error
evo_rpe - relative pose error
Tools:

evo_traj - tool for analyzing, plotting or exporting one or more trajectories
evo_res - tool for comparing one or multiple result files from evo_ape or evo_rpe
evo_fig - (experimental) tool for re-opening serialized plots (saved with --serialize_plot)
evo_config - tool for global settings and config file manipulation
Call the commands with --help to see the options, e.g. evo_ape --help. Tab-completion of command line parameters is available on UNIX-like systems.

More documentation Check out the Wiki on GitHub.
https://michaelgrupp.github.io/evo/

Formats

Trajectories
Executables that load trajectories support different formats. You can specify the format you want to use with the mode switch (2nd argument), for example: evo_traj tum … or evo_traj kitti … This document provides information about these file formats.

Tip: if you don’t know which format you should use for your SLAM algorithm, use the TUM format. It’s simple, but can be converted to any other format later and doesn’t require ROS.

bag - ROS bagfile
Note: requires a ROS installation

Currently, bagfiles with topics that contain geometry_msgs/PoseStamped, geometry_msgs/TransformStamped, geometry_msgs/PoseWithCovarianceStamped and nav_msgs/Odometry messages are supported. evo only reads the trajectory, thus covariance or twist aren’t used.

More infos:
http://wiki.ros.org/Bags
http://docs.ros.org/api/geometry_msgs/html/msg/PoseStamped.html
http://docs.ros.org/api/geometry_msgs/html/msg/PoseWithCovarianceStamped.html
http://docs.ros.org/api/geometry_msgs/html/msg/TransformStamped.html
http://docs.ros.org/api/nav_msgs/html/msg/Odometry.html
or run: rosmsg show geometry_msgs/PoseStamped

euroc - EuRoC MAV dataset format
Every sequence of the EuRoC MAV dataset has .csv files with various ground truth information. For us, only the first columns with the ground truth’s trajectory are important (timestamps, positions and orientations).

You can find these files in the sequence folder: /mav0/state_groundtruth_estimate0/data.csv

This format only makes sense for these ground truth files because they have additional data (which we don’t need). If you use the EuRoC dataset, you usually need to save the trajectory in another format like the TUM format (e.g. ORB-SLAM does it like that).

In euroc mode, evo’s metrics require you to give the reference in the EuRoC format and the estimated trajectory in the TUM format.

More infos: http://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets#the_euroc_mav_dataset

kitti - KITTI dataset pose format
This is actually not a real trajectory format because it has no timestamps - it only contains the poses in a text file. This means that you have to be careful when you want to compare two files in this format with a metric because the number of poses must be exactly the same.

Every row of the file contains the first 3 rows of a 4x4 homogeneous pose matrix (SE(3) matrix) flattened into one line, with each value separated by a space. For example, this pose matrix:

a b c d
e f g h
i j k l
0 0 0 1
will appear in the file as the row:

a b c d e f g h i j k l
More infos: http://www.cvlibs.net/datasets/kitti/eval_odometry.php

tum - TUM RGB-D dataset trajectory format
Every row has 8 entries containing timestamp (in seconds), position and orientation (as quaternion) with each value separated by a space:

timestamp x y z q_x q_y q_z q_w
More infos: https://vision.in.tum.de/data/datasets/rgbd-dataset/file_formats

Saving / Exporting to other formats
In general, you can export trajectories to other formats with evo_traj using the --save_as_<other_format> options.

The following combinations are possible:

–save_as_bag --save_as_kitti --save_as_tum
bag yes yes yes
euroc yes yes yes
kitti no (no timestamps) yes no (no timestamps)*
tum yes yes yes
*… but you can use this script together with the timestamp files of the KITTI dataset

exported ROS bag files will contain geometry_msgs/PoseStamped messages

Example:

export a EuRoC groundtruth file to a TUM trajectory
evo_traj euroc data.csv --save_as_tum
(will be saved as data.tum)

export TUM trajectories to KITTI format
evo_traj tum traj_1.txt traj_2.txt traj_3.txt --save_as_kitti
(will be saved as *.kitti)

export TUM trajectories to ROS bagfile
evo_traj tum traj_1.txt traj_2.txt traj_3.txt --save_as_bag
(will be saved as .bag with topics traj_1, traj_2 and traj_3)

and so on…
There is no --save_as_euroc option because the EuRoC format only makes sense for the ground truth of the EuRoC dataset.

Note: Prior to version 1.0.5, there was a bug with the quaternions if TUM files were exported. Make sure your version is at least v1.0.5 with evo pkg --version or upgrade.

Result zip files for evo_res
These files generated by a metric with the --save_results option contain the data that is needed to compare different results with evo_res. They are just .zip files that contain a few .json files and .npz files for storing larger numpy arrays.

If you want, you can also store a backup of the trajectories that were used to generate the result in the .zip file. To do this automatically whenever you use --save_results, change the settings:

evo_config set save_traj_in_zip true
Note that this will increase the size of the files.

https://github.com/MichaelGrupp/evo.wiki.git