Debugging Freeswitch

Contents [hide] 1 Introduction 2 Troubleshooting 2.1 Handy Troubleshooting Links 2.2 UUID Stamp at each DEBUG line 2.3 Chasing down XML errors 2.4 Recompiling with debug symbols on 2.5 Creating core files 2.6 Loading FreeSWITCH in GDB 2.6.1 Getting a Backtrace 2.6.2 Finding session pointers 2.7 Simple bash script to make debug easy 3 gdb issues 4 See Also Introduction This page is about using a debugger with FreeSWITCH. If you are new to troubleshooting, asking for help, or filing bug reports, please read this first. It will save you a lot of time. For an introduction to please see troubleshooting see Troubleshooting FreeSWITCH Troubleshooting Handy Troubleshooting Links Troubleshooting FreeSWITCH Packet Capturing The Dialplan SIP Debugging Connection Refused UUID Stamp at each DEBUG line The update in the code to enable a UUID Stamp at each debug line (provided kindly by Mathieu Rene and referred to in the "Finding session pointers" section below) has been committed to the GIT head (as of 19/Nov/2009), and you can enable this by a simple option at the logfile.conf.xml: Chasing down XML errors So you have FreeSWITCH with all of those included files and you get the "error near line X". With all of the included files where does this line number come from. By default it is ~freeswitch/log/freeswitch.xml.fsxml Do not edit this file while FreeSWITCH is running. It is memory mapped. freeswitch@localhost.localdomain>reloadxml API CALL [reloadxml()] output: +OK [[error near line 1622]: unexpected closing tag ] freeswitch@localhost.localdomain> Recompiling with debug symbols on export CFLAGS="-g -ggdb" export MOD_CFLAGS="-g -ggdb" ./configure Creating core files For core files to be created when a crash occurs, set the core ulimit to unlimited before starting FreeSWITCH $ ulimit -c unlimited $ ./freeswitch The core file should be located in the directory where FreeSWITCH was started (i.e., if you were in the /tmp directory when you typed the command to start FreeSWITCH, then there should be a file called something like /tmp/core.xxx). NOTE: On OS X, core files are dumped to a hidden directory called /cores by default, not the current directory! Loading FreeSWITCH in GDB From /usr/local/freeswitch: $ gdb bin/freeswitch To load with a core file $ gdb bin/freeswitch core.xxx Getting a Backtrace Once you load the core file into GDB you can collect a backtrace typing bt bt full thread apply all bt thread apply all bt full Collect the output when reporting bugs to Jira Finding session pointers Sometimes hunting stuff down can be a royal pain. The following code, submitted by Mathieu Rene, will print out all the session information (pointer information, etc.) for each session in a core file. After loading gdb with a core file (see above) do this in gdb: set $x=session_manager.session_table->table->first while($x != 0x0) printf "uuid %s is at %p/n", $x->pKey, $x->data set $x = $x->next end NB: $x->data is the switch_core_session_t*, you can dereference the channel with ->channel and can print some other stuff you might be looking for. This also work to print the content of a hashtable (use hash_table_name->table->first on the first set) Simple bash script to make debug easy In the source directory there is a support-d sub-directory. Inside this directory is a handy shell script called fscore_pb. It will post a backtrace to pastebin and report the pastebin URL to you on the command line. Usage: support-d/fscore_pb [corefile] The corefile is optional. If not specified, the script will take a gcore of a running instance of FreeSWITCH. If you've had a segfault then specify the core file name. For example, if you had a segfault and the core file produced was named core.12345 then use this command: support-d/fscore_pb core.12345 The script will collect the appropriate trace (and some other information), post it to pastebin.freeswitch.org, and then give you the URL to give to the developers. gdb issues On Ubuntu (Dapper/Hardy/Jaunty) gdb does not show symbols for some modules (e.g. mod_portaudio), while CentOS 5.3 havan't got such issue. The reason of the issue is that gdb in Ubuntu (probably Debian is affected too) is buggy. As a quick fix you can install development version of gdb from sources (install flex, bison and texinfo and get sources from git repository) which solves the issue. gdb trunk 20090626 is proven to work. See Also Reporting Bugs Error Codes Submitting Patches