Davison's MonoSLAM测试经验

Testing Davison's MonoSLAM

半个月前从Davison的主页上下载了monoSLAM的源码，在ubuntu 6.10(gcc 4.1.2)上 编译通过，并运行正常，最近又将它移植到windows平台，主要是考虑以后的兼容使用。其中有一些心得和经验，记录于此。

1. support usb camera under linux
Davison的代码只支持1394 camera，在linux下我先为其添加了USB camera支持. 这个工作其实也挺简单，参考 VW34/VWFirewire和 VW34/VWGLOW/Interface/*代码，写了个VWUSBCam接口，其中调用v4l2(2.6 kernel支持)API，这里又参考了spcaview中的代码，再修改一下makefile就行了，也就用了一天时间，麻烦的事情在于makefile的修改，automake生成的东东总是显得过于繁杂，手动修改起来比较烦。
sequence的工程机理是这样的：底层有一个循环，进行IO操作，不断地读取文件或摄像机数据；一旦得到数据后，发送消息给上层，上层调用响应消息的回调函数，在其中进行数据处理。 这样的循环式操作有一个弊端：CPU占用率一直比较高。特别是读取视频时，没有数据时也会一直在"空转"。不像MS的DirectShow，底层的WDM检测到数据信号时，自动发送消息通知上层，即中断式的操作。不过对于linux下的v4l2编程，目前还没有看到过支持中断方式视频捕获。

2.port to windows
移植往往是比较麻烦的工作，linux与windows的平台差异比较大。不过Oxford的VW4是cross platform的，并提供了VC7的project，monoslamglow和scenelib也是完全platform independent，并全用c++实现。这样的工作还是挺好的，我比较喜欢这样的方式，以前写的一些程序也都是这么处理的。
不过在移植过程中还是有一些问题。我使用的是VC2005，可能是目前最好的 编译器与IDE了，对C++标准支持得很好，毕竟是Lippman领导设计的。这使得monoslam中大量的模板 编译能够顺利通过。
首先需要 编译glut和glow，是opengl的一些扩展工具，我使用的分别是glut-3.7.6和glow_104， 编译中没有问题。
其次需要 编译wxWidgets，目前有2.8.3，第一次尝试搞错了一些设计，出了错，便选用了2.6.4，也是一样的问题。后来分析了一下错误，再结合readme，明白了原因：需要设置一些宏，打开或关闭一些选项。顺利通过后也懒得换高版本了，2.6.4够用了。
现在便可 编译VW34了，直接用其中的VC7 solution，并设置好前面几个库的路径， 编译也可以通过，但在后来 编译MonoSLAMGlow时，却有unresolved link error，显然是找不到库的模块。回头再看 VW34，发现了是因为有一些.cpp文件没有添加进solution，没有 编译进lib，还有就是文件读取的几个template没实例化，手动加上就可以了(虽说简单，但找到错误却不容易，一开始没想到是这些问题~~)。在这里，gcc与vc2005似乎有比较大的差别，gcc似乎不合乎标准。
最后 编译monoslamglow和scenelib。这两个没有现成的工程，monoslamglow其实就是scenelib中API的简单调用，负责一些界面处理，主要的vSLAM工作在scenelib中完成。直接将这两个模块放到了同一个project中，修改了一些头文件路径和gcc、vc2005的语言差别，最后也能 编译完成，但冒出来很多multiple symbols error，都是来自于 VW34，像是 VW34中已经定义了一些标准C++的库函数，如basic_string, vector之类，细查一下又没有，比较奇怪。google了一下，从VXL(一个比较好的computer vision library， VW34中的VNL便是它的一个模块)的文档中找到了答案：C++ 编译器 编译选项设置不一致。solution如下：

Project->Settings->C/C++ Tab->Category: Code Generation->Use run-time library: Multithreaded DLL

于是 编译就通过了，porting就初步完成了。

3. fix bugs
前面完成了移植工作，这只是一部分工作而已。程序在windows上读取Davison提供的test sequence，一会便出错了，而且是比较莫名的错误：基于wxWidget的窗口一重绘便出错。wxWidget的问题？若如此，便很麻烦了，不确定是wxWidget还是opengl或是基于opengl的glut、glow有bug，没精力去研读几十M的代码的~~
仍然 编译了个debug版本，初步调试了一下，发现只是在窗口重绘时出问题。像一般的界面程序一样，重绘是基于消息的，消息又与回调函数挂钩。在调试中发现调用回调函数的对象指针失效了，这是失败的原因！深入VW32的底层代码，跟踪程序的过程，了解了它的处理方式：从一个带虚函数的EventHandler类派生出窗口子类，再利于EventHandler保存子类的句柄，回调时，由该句柄调用多态调用子类的重绘函数。一切看起来很合理，充分利用了C++的多态性。但EventHandler却在重绘时失效了。不可理解。 好看月前仔细阅读了，了解了C++一些底层的对象模型处理过程，对于虚拟继承内部错综复杂的关系也比较清楚了。很快便想到：基类指针失效应该是由于指针在对象转型时offset调整出了问题。这部分是 编译器的工作，我们无法干预。这便又是gcc与vc2005的一个显著不同！看了一下MonoSLAMGlow类的声明：多重继承！4个类的多重继承！令人生畏的多重继承！其中EventHandler被放在第二个！这些类都有虚函数，很显然，也就会有多个虚函数表，指针调整时也会比较复杂。从理论上来说，EventHandler应该设计成抽象类，并作为公有虚拟基类，这样便不会有上述问题。我采用了一个比较简单的修改方案：将EventHandler设为第一个继承类，因为不是虚拟基类，它的地址便是MonoSLAMGlow的地址(vc2005中是这么处理的)，不再需要指针调整，快速、方便而又正确！就是这么一个问题，真是得感谢Lippman，难怪侯捷竭力推荐他的这本书，很有道理。
紧接着便是第二个错误：运行一会程序会崩溃了。使用debug版本观察了一下，是vector出错了，看似是指针无效。这个错误很难找，整个工程大量使用了vector，无法定位是哪里出了问题。前段时间发现这个问题时，觉得一时难以下手，也正好在忙别的事，就此搁下了。昨晚又拿起来分析了一下，初步判定是monoslamglow中的问题。今天花了一天时间调试，才找到这个问题所在。真是个非常隐晦而又阴暗的错误！当然，也怪我一时疏忽了，话说过来，这种bug若非亲历并尝过艰辛，是不可能想到并很快发现的，虽然我早就知道bug原因和解决方法~~建议使用STL的程序员都好好读读scott meyer的effective stl，唉，其实我已经读过两遍了~~
这个bug便是vector::erase便得基于vector的所有iterator都失效了！很简单的理由，在effective stl中反复说了好几遍。比如程序中有这样的一段代码：
for(vector::iterator feat = feature_init_info_vector.begin();
feat != feature_init_info_vector.end();feat ++)
{
if(feat->...){}
else{
delete_partially_initialised_feature(feat); // 调用vector::erase
}
}
看似没有问题，delete_partially_initialised_feature erase的是feat的一个副本，feat已经更新了。看过effective stl的人也都会这么说：这是meyer推荐的一个技巧啊，后置++返回旧值。其实不然，erase后，vector内存已经重新分配，以前的iterator都已经失效，因此更新后的feat实际是无效的，在下一次for比较中就会崩溃，错误提示为：incompatible iterator！一个我刚开始始终看不懂的错误。
修改方法就比较简单了，修改delete_partially_initialised_feature的原型，返回iterator,然后
feat = delete_partially_initialised_feature(feat)
即可。
注：程序中有三处这样的错误，需要一一修改。
gcc对于这个问题也没有出错。gcc使用的是sgi的stl，不知道底层是如何处理的，按理说不应如此。还可以说明一点的是：Davison(也有可能是其他人)的c++和stl水平不怎么样，不应该犯这样的错误的，嘿嘿！

update 1 : 2007-04-12
yet another bug of the same kind : incompatible iterator !
in Scene_single.cpp, line 301, delete_feature erases an item from feature vector and invalids all the iterators kept previously in the context, so in the next loop, the "incompatible iterator" error will jump out while comparing the iterators !
solution :
I subsitude the original prototype of delete_feature below

bool delete_feature()
with
std::vector::iterator delete_feature()

in fact, the latter conforms to the convention in STL, such as vector::erase, list::erase, etc.

update 2 : 2007-04-19
add usb camera support on both linux and windows !
It's quite interesting to run the demo with real environment.

4. conclusion
移植与修改这个程序还是很有意思的，有利于提高自己的程序水平。特别是使我们看到了大师提出的经典与意见是很有用的，经典著作不能不读！
另外，对于gcc，我也觉得比较有意思，上面发现的一些错误在gcc中完全正常，虽说有些地方不符合标准，但确实好用。还是比较喜欢gcc的。
写了这么多，觉得可以写封信给Davison了，report bugs to him and ask him to read some classic books on stl :)

5. forwards
以后再为monoslam加一个windows上的USB摄像头接口，那就可以做实验了。这倒是比较容易的，对于摄像头采集还是很熟的，今天就到此为止吧。
我还是比较喜欢在linux下写程序，工具比较丰富，也比较好玩，如emacs，eclipse之类的，只是不如windows中方便，因为vs2005的功能实在太强大了！不过支持开源、支持linux! 发贴者  xuning

==========英文原文==========

Two months ago, I tried Davision's MonoSALM which is very interesting. For research convenience, I successfully ported the project to both linux(ubuntu 6.10 & 7.04, gcc-4.1.2) and windows(WinXP,VS2005) platform with USB camera support. Similar work has been done by Bob Mottram, where he translated the project into c#.

As some other researchers may be interested in Davison's MonoSLAM, and just like me, want to experience the demo under both windows and linux with USB Camera support. So at here, I took down some of the key tips of my porting experience.

1.USB camera support
Davison's original demo only support 1394 camera under linux, but I didn't have one. So first I added USB camera support in order to experience the demo.

1.1 Under linux -- V4L2 Support
The following work was done under ubuntu 6.10 & 7.04(gcc-4.1.2), and you many have to change it for some other linux versions.

Download  v4l2 driver  according to the kernel version and install it properly. Try spacaview(availabel on the same website) to test the USB camera. A list of supported USB camera can be fond  here , which includes almost all the popular USB camera.

Spcaview is a smart but powerful tool to control the USB camera under linux, and it provides a good encapsulation to V4L2 APIs. I extraced some of the source code and built them into a library(libspcav4l) for easy usage in other applications. My transported project uses this library too.

The VW34 library used by MonoSLAM provides an abstraction to the manipulation of image source, either from file sequences or video capture device. Its general mechanism looks like this :
1) A looper runs continuously on the backgournd to read image file or capture video data from the caputre device. Once it gets the image data, it then notify the upper layer(image processing layer) to fetch the data.
2) The upper layer responds to the notification from the lower loop and process the image data in a callback function registered during the initialization phase.

One of the drawback of this mechanism is that, the looper runs continuously and takes up a lot of CPU resources. In fact, the approach adopted by windows' WDM driver is much more reasonable and elegant : whenever the video data arrives, WDM notifiles the upper layer to fetch it. That is a interruption approach, not loop approach.

The related code can be found at VW34/VWGLOW/Interface folder. Refering to VW34/VWFirewire, I added an interface named VWUSBCam, which calls functions in libspcav4l and added VWUSBCam interface to MonoSLAMGlow/monoslamglow*. Now the USB camera works for MonoSLAM!


1.2 Under Windows -- DirectShow Support
DShow provides a routine to easily manipulate the usb camera. Same as under linux, I added usb camera support with DShow to VWUSBCam interface.


2. Poring to Windows
It's usually troublesome to port a linux application to windows, especially when the application uses many libraries under linux. Fortunately, Oxford's VW34 is designed to be platform independent, and that saved me a lot of time. monoslamglow and scenelib are purely coded with c++, which is also cross-platform.

STL can be seen everywhere in Davison's MonoSLAM. This is good, for STL is efficient and elegant. But some fatal bugs lurk in the code due to the misusing of STL, for example vector::erase. Scott Meyer's Effective STL is really a good reference to write both effective and robust code with STL.

As VC6 does not fully support templates, so I chose VS2005 to compile the projects. Some additinal libraries should be prepared first. They are :

glut-3.7.6 : opengl wraper
wxWidgets-2.6.4 : wxWidgets support

Both the two libraries are platform independent and you can get them freely.I compiled and build them into static libraries.

other libraries used in MonoSLAM :

glow_104
scenelib
vw34

All the libraries are compiled into static libraries. The compiler may complaint "unresolved link" or some other errors due to the project setting or syntax problem. Just follow these prompts and fix them one by one.
What I want to mention is that, VW34 wrapps VNL which is part of VXL library, and after successfully compiling of the source code, plenty of "multiple symbols error" prompts may jump out while linking. I checked the VXL's FAQ and found the solution :
change the project setting in VS2005 as following :
Project->Settings->C/C++ Tab->Category: Code Generation->Use run-time library: Multithreaded DLL

Now I have a copy of MonoSLAM runs on windows with USB camera support. Next I fixed some treacherous bugs mainly due to the misusage of STL.

3. Fix bugs
Here I listed several most treacherous bugs of MonoSLAM. The amazing thing is that, these bugs keep quiet under linux, due to difference between gcc & vs2005 compiler, as well as the difference between implementations of SGI's and Microsoft's STL.
But, in my view, they are really bugs, and linux just "hides" them! So I'll list them up and give the solutions respectively.

3.1. Multiple Inheritance
The first problem encountered is that, once the main GUI window redrawed, it crashed.
It is very curisou, and may be cased by the bugs in wxWidget, opengl, glut or glow. And I had on idea of it ~~

As usual, I debuged the code, and looked deep into the whole mechanism of the paint procedure of monoslamglow. Same as other GUI applications, the mechanism is message driven and call a callback function to redraw the window. After carefully tracking all the variables, I found the handler of the callback object changed during the process which crashed the application!

The inheritance tree looks like this :

MonoSLAMGlow
<-- GlowWindow <-- VWEvents::EventHandler <-- GlowPushButtonReceiver <-- GlowCheckBoxReceiver Multiple inheritance is complicated, especially when vitual inheritance is involved. You can get a idea of how complicate it is from Lippman's  Inside the c++ object model .

All the four base class has virtual functions. The handler to callback object invalidated when converting to VWEvent::EventHandler polymorphicly. Lippman pointed out that the polymorphism of c++ itself is very complicate which involves adjusting pointer's address and the actual implementation is compiler-dependend.
I found that the adjusted address of pointer to the derived class had an addrss offset compared to its actual address. So, I adjusted the inheritance order of the base class, that was, put VWEvent::EventHandler at the first place, and the GUI application worked fine!

3.2. STL's pitfalls : vector::erase
Scott Meyer pointed out some dark corners of the c++ language and STL's pitfalls in effective stl . And it happened that Davison's MonoSLAM was caught by one of the pitfalls, that was vector::erase !

After erasing an object from a vector, all the iterators previously pointing to the object in the vector will be invalidate.

The above rule was emphasized again and again by Meyer. Though simple, yet easy to fall into this pitfall, as MonoSLAM did.

At first, MonoSLAM crashed for the error "incompatible iterator!" after running for a short while. Curious error, isn't it ? Experience told me that something may be wrong with stl's container, such as vector, map, set, list, etc. But locating the error was rather difficult because stl's container appeared everywhere in the code. After a careful debugging and scrutinizing the code, I found following code :

for(vector::iterator feat = feature_init_info_vector.begin();
feat != feature_init_info_vector.end();feat ++)
{
if(feat->...){}
else{
delete_partially_initialised_feature(feat); // call vector::erase inside
}
//...
}

similar codes could be found at other places.

It seemed pretty good from logical sense: go through all the objects in the vector and delete unqualified ones.
But it was wrong, according to the above rule!

Once feature_init_info_vector deleted an object, it would reallocate memory and copy the remaining object into new memory. So the feat iterator invalidated after calling delete_partially_initialised_feature, in other words, feat pointed to a old, invalidated memory object, not that in the new allocated feature_init_info_vector! That's why the application complained for "incompatible iterator"!

The solution is very easy, with Meyer's advice, return a iterator pointing to the new allocated feature_init_info_vector and assign it to feat. Ok, that's done !

4. Conclusion
Above is my experience with porting MonoSLAM to windows with USB camera support. Currently, it works well under both linux and windows. There may be some other hidden bugs, I'll fixed them when they emerges.