The LTTng Documentation
Copyright © 2014 The LTTng Project
This work is licensed under a Creative Commons Attribution 4.0 International License.
Welcome!
Welcome to the LTTng Documentation!
The Linux Trace Toolkit: next generationis an open source system software package for correlated tracing of theLinux kernel, user applications and libraries. LTTng consists of kernelmodules (for Linux kernel tracing) and dynamically loaded libraries (foruser application and library tracing). It is controlled by a sessiondaemon, which receives commands from a command line interface.
Convention
Function and argument names, variable names, command names,file system paths, file names and other precise strings are writtenusing amonospaced typeface in this document. Anitalic word within such a block is aplaceholder, usually described in the following sentence.
Practical tips and sidenotes are given throughout the document using ablue background:
Tip:Make sure you read the tips.
Terminal boxes are used to show command lines:
echo This is a terminal box
Typical command prompts, like $ and #, are not shown in terminalboxes to make copy/paste operations easier, especially for multilinecommands which may be copied and pasted as is in a user's terminal.Commands to be executed as a root user begin with sudo.
Target audience
The material of this documentation is appropriate for intermediate toadvanced software developers working in a Linux environment who areinterested in efficient software tracing. LTTng may also be worth atry for students interested in the inner mechanics of their systems.
Readers who do not have a programming background may wish to skipeverything related to instrumentation, which requires, most of thetime, some C language skills.
Note to readers:This is an opendocumentation: its source is available in apublic Gitrepository. Should you find any error in the contents of this text,any grammatical mistake or any dead link, we would be very grateful ifyou would fill a GitHub issue for it or, even better, contribute a patchto this documentation using a GitHub pull request.
Chapter descriptions
What follows is a list of brief descriptions of this documentation'schapters. The latter are ordered in such a way as to make the readingas linear as possible.
- Nuts and bolts explains therudiments of software tracing and the rationale behind theLTTng project.
- Installing LTTng is divided intosections describing the steps needed to get a working installationof LTTng packages for common Linux distributions and from itssource.
- Getting started is a very concise guide toget started quickly with LTTng kernel and user space tracing. Thischapter is recommended if you're new to LTTng or software tracingin general.
- Understanding LTTng deals with somecore concepts and components of the LTTng suite. Understandingthose is important since the next chapter assumes you're familiarwith them.
- Using LTTng is a complete user guide of theLTTng project. It shows in great details how to instrument userapplications and the Linux kernel, how to control tracing sessionsusing the
lttngcommand line tool and miscellaneous practical usecases. - Reference contains references of LTTng components,like links to online manpages and various APIs.
We recommend that you read the above chapters in this order, althoughsome of them may be skipped depending on your situation. You may skipNuts and bolts if you're familiar with tracingand LTTng. Also, you may jump overInstalling LTTng if LTTng is already properlyinstalled on your target system.
Acknowledgements
A few people made the online LTTng Documentation possible.
Philippe Proulx wrote and formatted most of the text.Daniel U. Thibault, from theDRDC,wrote an open guide calledLTTng: The Linux Trace Toolkit NextGeneration — A Comprehensive User's Guide (version 2.3edition) which was mostly used to complete parts of theUnderstanding LTTng chapter and for a fewpassages here and there.The wholeEfficiOSteam (Christian Babeux, Antoine Busque, Julien Desfossez,Mathieu Desnoyers, Jérémie Galarneau and David Goulet) made essentialreviews of the whole document.
We sincerely thank everyone who helped make this documentation whatit is. We hope you enjoy reading it as much as we did writing it.
What's new in LTTng 2.6?
Most of the changes of LTTng 2.6 are bug fixes, making the toolchainmore stable than ever before. Still, LTTng 2.6 adds some interestingfeatures.
LTTng 2.5 already supported the instrumentation and tracing ofJava applications throughjava.util.logging(JUL). LTTng 2.6 goes one step further by supportingApache log4j 1.2.The new log4j domain is selected using the--log4j option in variouscommands of thelttng tool.
LTTng-modules has supported system call tracing for a long time,but until now, it was only possible to record either all of them,or none of them. LTTng 2.6 allows the user to record only a specificsubset of system call events, e.g.:
lttng enable-event --kernel --syscall open,fork,chdir,pipe
Finally, the lttng command line tool cannot only communicate withhumans as it used to do, but also with machines thanks to its newmachine interface feature.
To learn more about the new features of LTTng 2.6, seethis release announcement.
Nuts and bolts
What is LTTng? As its name suggests, theLinux Trace Toolkit: next generation is a modern toolkit fortracing Linux systems and applications. So your first question mightrather be:what is tracing?
As the history of software engineering progressed and led to whatwe now take for granted—complex, numerous andinterdependent software applications running in parallel onsophisticated operating systems like Linux—the authors of suchcomponents, or software developers, began feeling a naturalurge of having tools to ensure the robustness and good performanceof their masterpieces.
One major achievement in this field is, inarguably, theGNU debugger(GDB), which is an essential tool for developers to find and fixbugs. But even the best debugger won't help make your software runfaster, and nowadays, faster software means either more work done bythe same hardware, or cheaper hardware for the same work.
A profiler is often the tool of choice to identify performancebottlenecks. Profiling is suitable to identifywhere performance islost in a given software; the profiler outputs a profile, astatistical summary of observed events, which you may use to knowwhich functions took the most time to execute. However, a profilerwon't reportwhy some identified functions are the bottleneck.Also, bottlenecks might only occur when specific conditions are met.For a thorough investigation of software performance issues, a historyof execution, with historical values of chosen variables, isessential. This is where tracing comes in handy.
Tracing is a technique used to understand what goes on in a runningsoftware system. The software used for tracing is called atracer,which is conceptually similar to a tape recorder. When recording,specific points placed in the software source code generate eventsthat are saved on a giant tape: atrace file. Both user applicationsand the operating system may be traced at the same time, opening thepossibility of resolving a wide range of problems that are otherwiseextremely challenging.
Tracing is often compared to logging. However, tracers and loggersare two different types of tools, serving two different purposes. Tracers aredesigned to record much lower-level events that occur much morefrequently than log messages, often in the thousands per second range,with very little execution overhead. Logging is more appropriate forvery high-level analysis of less frequent events: user accesses,exceptional conditions (e.g., errors, warnings), databasetransactions, instant messaging communications, etc. More formally,logging is one of several use cases that can be accomplished withtracing.
The list of recorded events inside a trace file may be read manuallylike a log file for the maximum level of detail, but it is generallymuch more interesting to perform application-specific analyses toproduce reduced statistics and graphs that are useful to resolve agiven problem. Trace viewers and analysers are specialized tools whichachieve this.
So, in the end, this is what LTTng is: a powerful, open source set oftools to trace the Linux kernel and user applications. LTTng iscomposed of several components actively maintained and developed byitscommunity.
Excluding proprietary solutions, a few competing software tracersexist for Linux.ftraceis the de facto function tracer of the Linux kernel.straceis able to record all system calls made by a user process.SystemTapis a Linux kernel and user space tracer which uses custom user scriptsto produce plain text traces.sysdigalso uses scripts, written in Lua, to trace and analyze the Linuxkernel.
The main distinctive features of LTTng is that it produces correlatedkernel and user space traces, as well as doing so with the lowestoverhead amongst other solutions. It produces trace files in theCTFformat, an optimized file format for production and analyses ofmulti-gigabyte data. LTTng is the result of close to 10 years ofactive development by a community of passionate developers. It iscurrently available on all major desktop, server, and embedded Linuxdistributions.
The main interface for tracing control is a single command line toolnamed lttng. The latter can create several tracing sessions,enable/disable events on the fly, filter them efficiently with customuser expressions, start/stop tracing and do much more. Traces can berecorded on disk or sent over the network, kept totally or partially,and viewed once tracing is inactive or in real-time.
Install LTTng now and start tracing!
Installing LTTng
LTTng is a set of software components which interact to allowinstrumenting the Linux kernel and user applications and controllingtracing sessions (starting/stopping tracing, enabling/disabling events,etc.). Those components are bundled into the following packages:
- LTTng-tools: Libraries and command line interface to controltracing sessions
- LTTng-modules: Linux kernel modules allowing Linux to betraced using LTTng
- LTTng-UST: User space tracing library
Most distributions mark the LTTng-modules and LTTng-UST packages asoptional. In the following sections, we always provide the steps toinstall all three, but be aware that LTTng-modules is only required ifyou intend to trace the Linux kernel and LTTng-UST is only required ifyou intend to trace user space applications.
This chapter shows how to install the above packages on a Linuxsystem. The easiest way is to use the package manager of the system'sdistribution (desktop orembedded). Support is also available forenterprise distributions, such asRed Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES).Otherwise, you canbuild the LTTng packages from source.
Desktop distributions
Official and unofficial LTTng packages are available for the majorLinux desktop distributions:Ubuntu,Fedora,Debian,openSUSE(and other RPM-based distributions) andArch Linux.LTTng is regularly tested on those. Should any issue arise whenfollowing the procedures below, please inform thecommunity about it.
Ubuntu
The following steps apply to Ubuntu ≥ 12.04. Forprevious releases, you will need to build and install LTTngfrom source, as no Ubuntu packages wereavailable before version 12.04.
Two repository types can provide LTTng packages for Ubuntu: officialrepositories andPPA.
Official repositories
To install LTTng from the official Ubuntu repositories, simply use apt-get:
sudo apt-get install lttng-tools sudo apt-get install lttng-modules-dkms sudo apt-get install liblttng-ust-dev
PPA
TheLTTng PPAoffers the latest stable versions of LTTng packages. To get packagesfrom the PPA, follow these steps:
sudo apt-add-repository ppa:lttng/ppa sudo apt-get update sudo apt-get install lttng-tools sudo apt-get install lttng-modules-dkms sudo apt-get install liblttng-ust-dev
Fedora
Starting from Fedora 17, LTTng-tools and LTTng-UST packages are officiallyavailable usingyum:
sudo yum install lttng-tools sudo yum install lttng-ust sudo yum install lttng-ust-devel
LTTng-modules still needs to be built and installed from source. For that, make sure that thekernel-devel package is already installed beforehand:
sudo yum install kernel-devel
Proceed on to fetch LTTng-modules' source.Build and install it as follows:
KERNELDIR=/usr/src/kernels/$(uname -r) make sudo make modules_install
Debian
Debian wheezy (stable) and previous versions are not supported; you willneed to build and install LTTng packagesfrom source for those.
Debian jessie (testing) and sid (unstable) have everything you need:
sudo apt-get install lttng-tools sudo apt-get install lttng-modules-dkms sudo apt-get install liblttng-ust-dev
openSUSE/RPM
openSUSE has had LTTng packages since version 12.3. To install LTTng, youfirst need to add an entry to your repository configuration. All LTTng repositoriesare availablehere.For example, the following commands will add the LTTng repository foropenSUSE 13.1:
sudo zypper addrepo http://download.opensuse.org/repositories/devel:/tools:/lttng/openSUSE_13.1/devel:tools:lttng.repo
Then, refresh the package database:
sudo zypper refresh
and install lttng-tools, lttng-modules and lttng-ust-devel:
sudo zypper install lttng-tools sudo zypper install lttng-modules sudo zypper install lttng-ust-devel
Arch Linux
LTTng packages are available in theAUR under the following names:lttng-tools,lttng-modulesandlttng-ust.
The three LTTng packages can be installed using the followingYaourt commands:
yaourt -S lttng-tools yaourt -S lttng-modules yaourt -S lttng-ust
If you're living on the edge, the AUR also contains the latest Git master branchfor each of those packages:lttng-tools-git,lttng-modules-gitandlttng-ust-git.
Embedded distributions
Some developers may be interested in tracing the Linux kernel and user spaceapplications running on embedded systems. LTTng is packaged by two popularembedded Linux distributions:Buildroot andOpenEmbedded/Yocto.
Buildroot
LTTng packages in Buildroot are lttng-tools, lttng-modules andlttng-libust.
To enable them, start the Buildroot configuration menu as usual:
make menuconfig
In:
- Kernel: make sure Linux kernel is enabled
- Toolchain: make sure the following options are enabled:
- Enable large file (files > 2GB) support
- Enable WCHAR support
In Target packages/Debugging, profiling and benchmark, enablelttng-modules andlttng-tools. InTarget packages/Libraries/Other, enablelttng-libust.
OpenEmbedded/Yocto
LTTng recipes are available in the openembedded-core layer ofOpenEmbedded:
lttng-toolslttng-moduleslttng-ust
Using BitBake, the simplest way to include LTTng recipes in yourtarget image is to add them toIMAGE_INSTALL_append inconf/local.conf:
IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
If you're using Hob, click Edit image recipe once you have selecteda machine and an image recipe. Then, in theAll recipes tab, searchforlttng and you should find and be able to include the three LTTngrecipes.
Enterprise distributions (RHEL,SLES)
To install LTTng on enterprise Linux distributions(such as RHEL andSLES), please seeEfficiOSEnterprise Packages.
Building from source
As previously stated, LTTng is shipped asthree packages: LTTng-tools, LTTng-modules and LTTng-UST. LTTng-toolscontains everything needed to control tracing sessions, whileLTTng-modules is only needed for Linux kernel tracing and LTTng-UST isonly needed for user space tracing.
The tarballs are available in theDownloadsection of the LTTng website.
Please refer to the README.md files provided by each package toproperly build and install them.
Tip:The aforementioned README.md filesare rendered as rich text whenviewed on GitHub.
If you're using Ubuntu, executing the following Bash scriptwill install the appropriate dependencies, clone the LTTngGit repositories, build the projects, and install them. The sources willbe cloned into~/src. Your user needs to be a sudoer for the installsteps to be completed.
#!/bin/bash
mkdir ~/src
cd ~/src
sudo apt-get update
sudo apt-get -y install build-essential libtool flex bison \
libpopt-dev uuid-dev libglib2.0-dev autoconf \
git libxml2-dev
git clone git://git.lttng.org/lttng-ust.git
git clone git://git.lttng.org/lttng-modules.git
git clone git://git.lttng.org/lttng-tools.git
git clone git://git.lttng.org/userspace-rcu.git
git clone git://git.efficios.com/babeltrace.git
cd userspace-rcu
./bootstrap && ./configure && make -j 4 && sudo make install
sudo ldconfig
cd ../lttng-ust
./bootstrap && ./configure && make -j 4 && sudo make install
sudo ldconfig
cd ../lttng-modules
make && sudo make modules_install
sudo depmod -a
cd ../lttng-tools
./bootstrap && ./configure && make -j 4 && sudo make install
sudo ldconfig
sudo cp extras/lttng-bash_completion /etc/bash_completion.d/lttng
cd ../babeltrace
./bootstrap && ./configure && make -j 4 && sudo make install
sudo ldconfig
Getting started with LTTng
This is a small guide to get started quickly with LTTng kernel and userspace tracing. For intermediate to advanced use cases and a morethorough understanding of LTTng, seeUsing LTTng andUnderstanding LTTng.
Before reading this guide, make sure LTTngis installed. You will at least needLTTng-tools. Also install LTTng-modules fortracing the Linux kernel and LTTng-USTfortracing your own user space applications.When your traces are finally written and complete, theViewing and analyzing your tracessection of this chapter will help you analyze your tracepoint events to investigate.
Tracing the Linux kernel
Make sure LTTng-tools and LTTng-modules packagesare installed.
Since you're about to trace the Linux kernel itself, let's look at theavailable kernel events using thelttng tool, which has aGit-like command line structure:
lttng list --kernel
Before tracing, you need to create a session:
sudo lttng create my-session
Tip:You can avoid using sudo in the previous and following commands if your user is a member of thetracing group.
my-session is the tracing session name and could be anything youlike.auto will be used if omitted.
Let's now enable some events for this session:
sudo lttng enable-event --kernel sched_switch,sched_process_fork
or you might want to simply enable all available kernel events (bewarethat trace files will grow rapidly when doing this):
sudo lttng enable-event --kernel --all
Start tracing:
sudo lttng start
By default, traces are saved in~/lttng-traces/name-date-time,wherename is the session name.
When you're done tracing:
sudo lttng stop sudo lttng destroy
Although destroy looks scary here, it doesn't actually destroy theoutputted trace files: it only destroys the tracing session.
What's next? Have a look atViewing and analyzing your tracesto view and analyze the trace you just recorded.
Tracing your own user application
The previous section helped you create a trace out of Linux kernel events.This section steps you through a simple example showing you how to traceaHello world program written in C.
Make sure LTTng-tools and LTTng-UST packagesare installed.
Tracing is just like having printf() calls at specific locations ofyour source code, albeit LTTng is much more faster and flexible thanprintf(). In the LTTng realm,tracepoint() is analogous toprintf().
Unlike printf(), though, tracepoint() does not use a format string toknow the types of its arguments: the formats of all tracepoints must bedefined before using them. So before even writing ourHello world program,we need to define the format of our tracepoint. This is done by writing atemplate file, with a name usually ending with the.tp extension (fortracepoint),which thelttng-gen-tp tool (shipped with LTTng-UST) will use to generatean object file (along with a.c file) and a header to be included in our application source code.
Here's the whole flow:
The template file format is a list of tracepoint definitionsand other optional definition entries which we will skip forthis quickstart. Each tracepoint is defined using theTRACEPOINT_EVENT() macro. For each tracepoint, you must provide:
- a provider name, which is the "scope" of this tracepoint (this usuallyincludes the company and project names)
- a tracepoint name
- a list of arguments for the eventual
tracepoint()call, each item being:- the argument C type
- the argument name
- a list of fields, which will be the actual fields of the recorded eventsfor this tracepoint
Here's a simple tracepoint definition example with two arguments: an integerand a string:
TRACEPOINT_EVENT(
hello_world,
my_first_tracepoint,
TP_ARGS(
int, my_integer_arg,
char*, my_string_arg
),
TP_FIELDS(
ctf_string(my_string_field, my_string_arg)
ctf_integer(int, my_integer_field, my_integer_arg)
)
)
The exact syntax is well explained in theC application instrumenting guide of theUsing LTTng chapter, as well as in theLTTng-UST manpage.
Save the above snippet as hello-tp.tp and run:
lttng-gen-tp hello-tp.tp
The following files will be created next to hello-tp.tp:
hello-tp.chello-tp.ohello-tp.h
hello-tp.o is the compiled object file of hello-tp.c.
Now, by including hello-tp.h in your own application, you may use thetracepoint defined above by properly refering to it when callingtracepoint():
#include <stdio.h> #include "hello-tp.h" int main(int argc, char* argv[]) { int x; puts("Hello, World!\nPress Enter to continue..."); /* The following getchar() call is only placed here for the purpose * of this demonstration, for pausing the application in order for * you to have time to list its events. It's not needed otherwise. */ getchar(); /* A tracepoint() call. Arguments, as defined in hello-tp.tp: * * 1st: provider name (always) * 2nd: tracepoint name (always) * 3rd: my_integer_arg (first user-defined argument) * 4th: my_string_arg (second user-defined argument) * * Notice the provider and tracepoint names are NOT strings; * they are in fact parts of variables created by macros in * hello-tp.h. */ tracepoint(hello_world, my_first_tracepoint, 23, "hi there!"); for (x = 0; x < argc; ++x) { tracepoint(hello_world, my_first_tracepoint, x, argv[x]); } puts("Quitting now!"); tracepoint(hello_world, my_first_tracepoint, x * x, "x^2"); return 0; }
Save this as hello.c, next to hello-tp.tp.
Notice hello-tp.h, the header file generated by lttng-gen-tp fromour template filehello-tp.tp, is included byhello.c.
You are now ready to compile the application with LTTng-UST support:
gcc -o hello hello.c hello-tp.o -llttng-ust -ldl
If you followed theTracing the Linux kernel section, thefollowing steps will look familiar.
First, run the application with a few arguments:
./hello world and beyond
You should see
Hello, World! Press Enter to continue...
Use the lttng tool to list all available user space events:
lttng list --userspace
You should see the hello_world:my_first_tracepoint tracepoint listedunder the./hello process.
Create a tracing session:
lttng create my-userspace-session
Enable the hello_world:my_first_tracepoint tracepoint:
lttng enable-event --userspace hello_world:my_first_tracepoint
Start tracing:
lttng start
Go back to the running hello application and press Enter. All tracepoint()calls will be executed and the program will finally exit.
Stop tracing:
lttng stop
Done! You may use lttng view to list the recorded events. This commandstartsbabeltracein the background, if it is installed:
lttng view
should output something like:
[18:10:27.684304496] (+?.?????????) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "hi there!", my_integer_field = 23 }
[18:10:27.684338440] (+0.000033944) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "./hello", my_integer_field = 0 }
[18:10:27.684340692] (+0.000002252) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "world", my_integer_field = 1 }
[18:10:27.684342616] (+0.000001924) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "and", my_integer_field = 2 }
[18:10:27.684343518] (+0.000000902) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "beyond", my_integer_field = 3 }
[18:10:27.684357978] (+0.000014460) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "x^2", my_integer_field = 16 }
When you're done, you may destroy the tracing session, which does notdestroy the generated trace files, leaving them available for furtheranalysis:
lttng destroy my-userspace-session
The next section presents other alternatives to view and analyze yourLTTng traces.
Viewing and analyzing your traces
This section describes how to visualize the data gathered after tracingthe Linux kernel or a user space application.
Many ways exist to read your LTTng traces:
babeltraceis a command line utility which converts trace formats;it supports the format used by LTTng,CTF, as well as a basictext output which may begreped. Thebabeltracecommand ispart of theBabeltrace project.- Babeltrace also includes a Python binding so that you mayeasily open and read an LTTng trace with your own script, benefitingfrom the power of Python.
- Trace Compassis an Eclipse plugin used to visualize and analyze various types oftraces, including LTTng's. It also comes as a standalone applicationand can be downloaded fromhere.
LTTng trace files are usually recorded in the ~/lttng-traces directory.Let's now view the trace and perform a basic analysis usingbabeltrace.
The simplest way to list all the recorded events of a trace is to pass itspath tobabeltrace with no options:
babeltrace ~/lttng-traces/my-session
babeltrace will find all traces within the given path recursively andoutput all their events, merging them intelligently.
Listing all the system calls of a Linux kernel trace with their arguments iseasy withbabeltrace andgrep:
babeltrace ~/lttng-traces/my-kernel-session | grep sys_
Counting events is also straightforward:
babeltrace ~/lttng-traces/my-kernel-session | grep sys_read | wc --lines
The text output of babeltrace is useful for isolating events by simplematching usinggrep and similar utilities. However, more elaborate filterssuch as keeping only events with a field value falling within a specific rangeare not trivial to write using a shell. Moreover, reductions and even themost basic computations involving multiple events are virtually impossibleto implement.
Fortunately, Babeltrace ships with a Python 3 binding which makes itreally easy to read the events of an LTTng trace sequentially and computethe desired information.
Here's a simple example using the Babeltrace Python binding. The followingscript accepts an LTTng Linux kernel trace path as its first argument andoutputs the short names of the top 5 running processes on CPU 0 during thewhole trace:
import sys from collections import Counter import babeltrace def top5proc(): if len(sys.argv) != 2: msg = 'Usage: python {} TRACEPATH'.format(sys.argv[0]) raise ValueError(msg) # a trace collection holds one to many traces col = babeltrace.TraceCollection() # add the trace provided by the user # (LTTng traces always have the 'ctf' format) if col.add_trace(sys.argv[1], 'ctf') is None: raise RuntimeError('Cannot add trace') # this counter dict will hold execution times: # # task command name -> total execution time (ns) exec_times = Counter() # this holds the last `sched_switch` timestamp last_ts = None # iterate events for event in col.events: # keep only `sched_switch` events if event.name != 'sched_switch': continue # keep only events which happened on CPU 0 if event['cpu_id'] != 0: continue # event timestamp cur_ts = event.timestamp if last_ts is None: # we start here last_ts = cur_ts # previous task command (short) name prev_comm = event['prev_comm'] # initialize entry in our dict if not yet done if prev_comm not in exec_times: exec_times[prev_comm] = 0 # compute previous command execution time diff = cur_ts - last_ts # update execution time of this command exec_times[prev_comm] += diff # update last timestamp last_ts = cur_ts # display top 10 for name, ns in exec_times.most_common(5): s = ns / 1000000000 print('{:20}{} s'.format(name, s)) if __name__ == '__main__': top5proc()
Save this script as top5proc.py and run it with Python 3, providing thepath to an LTTng Linux kernel trace as the first argument:
python3 top5proc.py ~/lttng-sessions/my-session-.../kernel
Make sure the path you provide is the directory containing actual tracefiles (channel0_0,metadata, etc.): thebabeltrace utility recursesdirectories, but the Python binding does not.
Here's an example of output:
swapper/0 48.607245889 s chromium 7.192738188 s pavucontrol 0.709894415 s Compositor 0.660867933 s Xorg.bin 0.616753786 s
Note that swapper/0 is the "idle" process of CPU 0 on Linux; since weweren't using the CPU that much when tracing, its first position in the listmakes sense.
Understanding LTTng
If you're going to use LTTng in any serious way, it is fundamental thatyou become familiar with its core concepts. Technical terms liketracing sessions,domains,channels and events are used overand over in theUsing LTTng chapter,and it is assumed that you understand what they mean when reading it.
LTTng, as you already know, is a toolkit. It would be wrongto call it a simpletool since it is composed of multiple interactingcomponents. This chapter also describes the latter, providing detailsabout their respective roles and how they connect together to formthe current LTTng ecosystem.
Core concepts
This section explains the various elementary concepts a user has to dealwith when using LTTng. They are:
Tracing session
A tracing session is—like any session—a container ofstate. Anything that is done when tracing using LTTng happens in thescope of a tracing session. In this regard, it is analogous to a bankwebsite's session: you can't interact online with your bank accountunless you are logged in a session, except for reading a few staticwebpages (LTTng, too, can report some static information that does notneed a created tracing session).
A tracing session holds the following attributes and objects (some ofwhich are described in the following sections):
- a name
- the tracing state (tracing started or stopped)
- the trace data output path/URL (local path or sent over the network)
- a mode (normal, snapshot or live)
- the snapshot output paths/URLs (if applicable)
- for each domain, a list ofchannels
- for each channel:
- a name
- the channel state (enabled or disabled)
- its parameters (event loss mode, sub-buffers size and count,timer periods, output type, trace files size and count, etc.)
- a list of added context information
- a list of events
- for each event:
- its state (enabled or disabled)
- a list of instrumentation points (tracepoints, system calls,dynamic probes, etc.)
- associated log levels
- a filter expression
All this information is completely isolated between tracing sessions.
Conceptually, a tracing session is a per-user object; thePlumbing section shows how this is actuallyimplemented. Any user may create as many concurrent tracing sessionsas desired. As you can see in the list above, even the tracing stateis a per-tracing session attribute, so that you may trace your targetsystem/application in a given tracing session with a specificconfiguration while another one stays inactive.
The trace data generated in a tracing session may be either savedto disk, sent over the network or not saved at all (in which casesnapshots may still be saved to disk or sent to a remote machine).
Domain
A tracing domain is the official term the LTTng project uses todesignate a tracer category.
There are currently four known domains:
- Linux kernel
- user space
java.util.logging(JUL)- log4j
Different tracers expose common features in their own interfaces, but,from a user's perspective, you still need to target a specific type oftracer to perform some actions. For example, since both kernel and userspace tracers support named tracepoints (probes manually inserted insource code), you need to specify which one is concerned when enablingan event because both domains could have existing events with the samename.
Some features are not available in all domains. Filtering enabledevents using custom expressions, for example, is currently notsupported in the kernel domain, but support could be added in thefuture.
Channel
A channel is a set of events with specific parameters and potentialadded context information. Channels have unique names per domain withina tracing session. A given event is always registered to at least onechannel; having an enabled event in two channels will produce a tracewith this event recorded twice everytime it occurs.
Channels may be individually enabled or disabled. Occurring events ofa disabled channel will never make it to recorded events.
The fundamental role of a channel is to keep a shared ring buffer, whereevents are eventually recorded by the tracer and consumed by a consumerdaemon. This internal ring buffer is divided into many sub-buffers ofequal size.
Channels, when created, may be fine-tuned thanks to a few parameters,many of them related to sub-buffers. The following subsections explainwhat those parameters are and in which situations you should manuallyadjust them.
Overwrite and discard event loss modes
As previously mentioned, a channel's ring buffer is divided into manyequally sized sub-buffers.
As events occur, they are serialized as trace data into a specificsub-buffer (yellow arc in the following animation) until it is full:when this happens, the sub-buffer is marked as consumable (red) andanother,empty (white) sub-buffer starts receiving the followingevents. The marked sub-buffer will be consumed eventually by a consumerdaemon (returns to white).
In an ideal world, sub-buffers are consumed faster than filled, like itis the case above. In the real world, however, all sub-buffers could befull at some point, leaving no space to record the following events. Bydesign, LTTng is anon-blocking tracer: when no empty sub-bufferexists, losing events is acceptable when the alternative would be tocause substantial delays in the instrumented application's execution.LTTng privileges performance over integrity, aiming at perturbing thetraced system as little as possible in order to make tracing of subtlerace conditions and rare interrupt cascades possible.
When it comes to losing events because no empty sub-buffer is available,the channel'sevent loss mode determines what to do amongst:
- Discard: drop the newest events until a sub-buffer is released.
- Overwrite: clear the sub-buffer containing the oldest recordedevents and start recording the newest events there. This mode issometimes calledflight recorder mode because it behaves like aflight recorder: always keep a fixed amount of the latest data.
Which mechanism you should choose depends on your context: prioritizethe newest or the oldest events in the ring buffer?
Beware that, in overwrite mode, a whole sub-buffer is abandoned as soonas a new event doesn't find an empty sub-buffer, whereas in discardmode, only the event that doesn't fit is discarded.
Also note that a count of lost events will be incremented and saved inthe trace itself when an event is lost in discard mode, whereas noinformation is kept when a sub-buffer gets overwritten before beingcommitted.
There are known ways to decrease your probability of losing events. Thenext section shows how tuning the sub-buffers count and size can beused to virtually stop losing events.
Sub-buffers count and size
For each channel, an LTTng user may set its number of sub-buffers andtheir size.
Note that there is a noticeable tracer's CPU overhead introduced whenswitching sub-buffers (marking a full one as consumable and switchingto an empty one for the following events to be recorded). Knowing this,the following list presents a few practical situations along with howto configure sub-buffers for them:
- High event throughput: in general, prefer bigger sub-buffers tolower the risk of losing events. Having bigger sub-buffers willalso ensure a lower sub-buffer switching frequency. The number ofsub-buffers is only meaningful if the channel is in overwrite mode:in this case, if a sub-buffer overwrite happens, you will still havethe other sub-buffers left unaltered.
- Low event throughput: in general, prefer smaller sub-bufferssince the risk of losing events is already low. Since eventshappen less frequently, the sub-buffer switching frequency shouldremain low and thus the tracer's overhead should not be a problem.
- Low memory system: if your target system has a low memorylimit, prefer fewer first, then smaller sub-buffers. Even if thesystem is limited in memory, you want to keep the sub-buffers asbig as possible to avoid a high sub-buffer switching frequency.
You should know that LTTng uses CTF as its trace format, which meansevent data is very compact. For example, the average LTTng Linux kernelevent weights about 32 bytes. A sub-buffer size of 1 MiB isthus considered big.
The previous situations highlight the major trade-off between a few bigsub-buffers and more, smaller sub-buffers: sub-buffer switchingfrequency vs. how much data is lost in overwrite mode. Assuming aconstant event throughput and using the overwrite mode, the twofollowing configurations have the same ring buffer total size:
- 2 sub-buffers of 4 MiB each lead to a very low sub-bufferswitching frequency, but if a sub-buffer overwrite happens, half ofthe recorded events so far (4 MiB) are definitely lost.
- 8 sub-buffers of 1 MiB each lead to 4 times the tracer'soverhead as the previous configuration, but if a sub-bufferoverwrite happens, only the eighth of events recorded so far aredefinitely lost.
In discard mode, the sub-buffers count parameter is pointless: use twosub-buffers and set their size according to the requirements of yoursituation.
Switch timer
The switch timer period is another important configurable feature ofchannels to ensure periodic sub-buffer flushing.
When the switch timer fires, a sub-buffer switch happens. This timermay be used to ensure that event data is consumed and committed totrace files periodically in case of a low event throughput:
It's also convenient when big sub-buffers are used to cope withsporadic high event throughput, even if the throughput is normallylower.
Buffering schemes
In the user space tracing domain, two buffering schemes areavailable when creating a channel:
- Per-PID buffering: keep one ring buffer per process.
- Per-UID buffering: keep one ring buffer for all processes ofa single user.
The per-PID buffering scheme will consume more memory than the per-UIDoption if more than one process is instrumented for LTTng-UST. However,per-PID buffering ensures that one process having a high eventthroughput won't fill all the shared sub-buffers, only its own.
The Linux kernel tracing domain only has one available buffering schemewhich is to use a single ring buffer for the whole system.
Event
An event, in LTTng's realm, is a term often used metonymically,having multiple definitions depending on the context:
- When tracing, an event is a point in space-time. Space, in atracing context, is the set of all executable positions of acompiled application by a logical processor. When a program isexecuted by a processor and some instrumentation point, orprobe, is encountered, an event occurs. This event is accompaniedby some contextual payload (values of specific variables at thispoint of execution) which may or may not be recorded.
- In the context of a recorded trace file, the term event impliesa recorded event.
- When configuring a tracing session, enabled events refer tospecific rules which could lead to the transfer of actualoccurring events (1) to recorded events (2).
The whole Core concepts section focuses on thethird definition. An event is always registered toone or morechannels and may be enabled or disabled at will per channel. A disabledevent will never lead to a recorded event, even if its channelis enabled.
An event (3) is enabled with a few conditions that must all be metwhen an event (1) happens in order to generate a recorded event (2):
- A probe or group of probes in the traced application must beexecuted.
- Optionally, the probe must have a log level matching alog level range specified when enabling the event.
- Optionally, the occurring event must satisfy a customexpression, orfilter, specified when enabling the event.
The following illustration summarizes how tracing sessions, domains,channels and events are related:
This diagram also shows how events may be individually enabled/disabled(green/grey) and how a given event may be registered to more than onechannel.
Plumbing
The previous section described the concepts at the heart of LTTng.This section summarizes LTTng's implementation: how those objects aremanaged by different applications and libraries working together toform the toolkit.
Overview
As mentioned previously, the whole LTTng suiteis made of the following packages: LTTng-tools, LTTng-UST, andLTTng-modules. Together, they provide different daemons, libraries,kernel modules and command line interfaces. The following tree showswhich usable component belongs to which package:
- LTTng-tools:
- session daemon (
lttng-sessiond) - consumer daemon (
lttng-consumerd) - relay daemon (
lttng-relayd) - tracing control library (
liblttng-ctl) - tracing control command line tool (
lttng)
- session daemon (
- LTTng-UST:
- user space tracing library (
liblttng-ust) and its headers - preloadable user space tracing helpers(
liblttng-ust-libc-wrapper,liblttng-ust-pthread-wrapper,liblttng-ust-cyg-profile,liblttng-ust-cyg-profile-fastandliblttng-ust-dl) - user space tracepoint code generator command line tool(
lttng-gen-tp) java.util.logging/log4j tracepoint providers(liblttng-ust-jul-jniandliblttng-ust-log4j-jni) and JARfile (liblttng-ust-agent.jar)
- user space tracing library (
- LTTng-modules:
- LTTng Linux kernel tracer module
- tracing ring buffer kernel modules
- many LTTng probe kernel modules
The following diagram shows how the most important LTTng componentsinteract. Plain black arrows represent trace data paths while dashedred arrows indicate control communications. The LTTng relay daemon isshown running on a remote system, although it could as well run on thetarget (monitored) system.
Each component is described in the following subsections.
Session daemon
At the heart of LTTng's plumbing is the session daemon, often calledby its command name,lttng-sessiond.
The session daemon is responsible for managing tracing sessions andwhat they logically contain (channel properties, enabled/disabledevents, etc.). By communicating locally with instrumented applications(using LTTng-UST) and with the LTTng Linux kernel modules(LTTng-modules), it oversees all tracing activities.
One of the many things that lttng-sessiond does is to keeptrack of the available event types. User space applications andlibraries actively connect and register to the session daemon when theystart. By contrast,lttng-sessiond seeks out and loads the appropriateLTTng kernel modules as part of its own initialization. Kernel eventtypes arepulled bylttng-sessiond, whereas user space event typesarepushed to it by the various user space tracepoint providers.
Using a specific inter-process communication protocol with Linux kerneland user space tracers, the session daemon can send channel informationso that they are initialized, enable/disable specific probes based onenabled/disabled events by the user, send event filters information toLTTng tracers so that filtering actually happens at the tracer site,start/stop tracing a specific application or the Linux kernel, etc.
The session daemon is not useful without some user controlling it,because it's only a sophisticated control interchange and thusdoesn't make any decision on its own.lttng-sessiond opens a localsocket for controlling it, albeit the preferred way to control it isusingliblttng-ctl, an installed C library hiding the communicationprotocol behind an easy-to-use API. Thelttng tool makes use ofliblttng-ctl to implement a user-friendly command line interface.
lttng-sessiond does not receive any trace data from instrumentedapplications; theconsumer daemons are the programs responsible forcollecting trace data using shared ring buffers. However, the sessiondaemon is the one that must spawn a consumer daemon and establisha control communication with it.
Session daemons run on a per-user basis. Knowing this, multipleinstances of lttng-sessiond may run simultaneously, each belongingto a different user and each operating independently of the others.Onlyroot's session daemon, however, may control LTTng kernel modules(i.e. the kernel tracer). With that in mind, if a user has no rootaccess on the target system, he cannot trace the system's kernel, butshould still be able to trace its own instrumented applications.
It has to be noted that, although only root's session daemon maycontrol the kernel tracer, thelttng-sessiond command has a--groupoption which may be used to specify the name of a special user groupallowed to communicate withroot's session daemon and thus recordkernel traces. By default, this group is namedtracing.
If not done yet, the lttng tool, by default, automatically starts asession daemon.lttng-sessiond may also be started manually:
lttng-sessiond
This will start the session daemon in foreground. Use
lttng-sessiond --daemonize
to start it as a true daemon.
To kill the current user's session daemon, pkill may be used:
pkill lttng-sessiond
The default SIGTERM signal will terminate it cleanly.
Several other options are available and described inlttng-sessiond's manpageor by runninglttng-sessiond --help.
Consumer daemon
The consumer daemon, or lttng-consumerd, is a program sharing somering buffers with user applications or the LTTng kernel modules tocollect trace data and output it at some place (on disk or sent overthe network to an LTTng relay daemon).
Consumer daemons are created by a session daemon as soon as events areenabled within a tracing session, well before tracing is activatedfor the latter. Entirely managed by session daemons,consumer daemons survive session destruction to be reused later,should a new tracing session be created. Consumer daemons are alwaysowned by the same user as their session daemon. When its owner sessiondaemon is killed, the consumer daemon also exits. This is becausethe consumer daemon is always the child process of a session daemon.Consumer daemons should never be started manually. For this reason,they are not installed in one of the usual locations listed in thePATH environment variable.lttng-sessiond has, however, abunch of options tospecify custom consumer daemon paths if, for some reason, a consumerdaemon other than the default installed one is needed.
There are up to two running consumer daemons per user, whereas only onesession daemon may run per user. This is because each process hasindependent bitness: if the target system runs a mixture of 32-bit and64-bit processes, it is more efficient to have separate corresponding32-bit and 64-bit consumer daemons. The root user is an exception: itmay have up tothree running consumer daemons: 32-bit and 64-bitinstances for its user space applications and one more reserved forcollecting kernel trace data.
As new tracing domains are added to LTTng, the development community'sintent is to minimize the need for additionnal consumer daemon instancesdedicated to them. For instance, thejava.util.logging (JUL) domainevents are in fact mapped to the user space domain, thus tracing thisparticular domain is handled by existing user space domain consumerdaemons.
Relay daemon
When a tracing session is configured to send its trace data over thenetwork, an LTTngrelay daemon must be used at the other end toreceive trace packets and serialize them to trace files. This setupmakes it possible to trace a target system without ever committing tracedata to its local storage, a feature which is useful for embeddedsystems, amongst others. The command implementing the relay daemonislttng-relayd.
The basic use case of lttng-relayd is to transfer trace data receivedover the network to trace files on the local file system. The relaydaemon must listen on two TCP ports to achieve this: one control port,used by the target session daemon, and one data port, used by thetarget consumer daemon. The relay and session daemons agree on commondefault ports when custom ones are not specified.
Since the communication transport protocol for both ports is standardTCP, the relay daemon may be started either remotely or locally (on thetarget system).
While two instances of consumer daemons (32-bit and 64-bit) may runconcurrently for a given user,lttng-relayd needs only be of itshost operating system's bitness.
The other important feature of LTTng's relay daemon is the support ofLTTng live. LTTng live is an application protocol to view events asthey arrive. The relay daemon will still record events in trace files,but atee may be created to inspect incoming events. Using LTTng livelocally thus requires to run a local relay daemon.
Control library and command line interface
The LTTng control library, liblttng-ctl, can be used to communicatewith the session daemon using a C API that hides the underlyingprotocol's details.liblttng-ctl is part of LTTng-tools.
liblttng-ctl may be used by including its "master" header:
#include <lttng/lttng.h>
Some objects are referred by name (C string), such as tracing sessions,but most of them require creating a handle first usinglttng_create_handle(). The best available developer documentation forliblttng-ctl is, for the moment, its installed header files as such.Every function/structure is thoroughly documented.
The lttng program is the de facto standard user interface tocontrol LTTng tracing sessions.lttng usesliblttng-ctl tocommunicate with session daemons behind the scenes.Its manpage is exhaustive, aswell as its command line help (lttngcmd --help,wherecmd is the command name).
The Controlling tracing section is a featuretour of thelttng tool.
User space tracing library
The user space tracing part of LTTng is possible thanks to the userspace tracing library,liblttng-ust, which is part of the LTTng-USTpackage.
liblttng-ust provides header files containing macros used to definetracepoints and create tracepoint providers, as well as a shared objectthat must be linked to individual applications to connect to andcommunicate with a session daemon and a consumer daemon as soon as theapplication starts.
The exact mechanism by which an application is registered to thesession daemon is beyond the scope of this documentation. The only thingyou need to know is that, since the library constructor does this jobautomatically, tracepoints may be safely inserted anywhere in the sourcecode without prior manual initialization of liblttng-ust.
The liblttng-ust-session daemon collaboration also provides aninteresting feature: user space events may be enabledbeforeapplications actually start. By doing this and starting tracing beforelaunching the instrumented application, you make sure that even theearliest occurring events can be recorded.
The C application instrumenting guide of theUsing LTTng chapter focuses on usingliblttng-ust:instrumenting, building/linking and running a user application.
LTTng kernel modules
The LTTng Linux kernel modules provide everything needed to trace theLinux kernel: various probes, a ring buffer implementation for aconsumer daemon to read trace data and the tracer itself.
Only in exceptional circumstances should you ever need to load theLTTng kernel modules manually: it is normally the responsability ofroot's session daemon to do so. Even if you were to develop yourown LTTng probe module—for tracing a custom kernel or some kernelmodule (this topic is covered in theLinux kernel instrumenting guide oftheUsing LTTng chapter)—youshould use the--extra-kmod-probes option of the session daemon toappend your probe to the default list. The session and consumer daemonsof regular users do not interact with the LTTng kernel modules at all.
LTTng kernel modules are installed, by default, in/usr/lib/modules/release/extra, whererelease is the kernel release(seeuname --kernel-release).
Using LTTng
Using LTTng involves two main activities: instrumenting andcontrolling tracing.
Instrumenting is the process of inserting probesinto some source code. It can be done manually, by writing tracepointcalls at specific locations in the source code of the program to trace,or more automatically using dynamic probes (address in assembled code,symbol name, function entry/return, etc.).
It has to be noted that, as an LTTng user, you may not have to worryabout the instrumentation process. Indeed, you may want to trace aprogram already instrumented. As an example, the Linux kernel isthoroughly instrumented, which is why you can trace it without caringabout adding probes.
Controlling tracing is everythingthat can be done by the LTTng session daemon, which is controlled usingliblttng-ctl or its command line utility,lttng: creating tracingsessions, listing tracing sessions and events, enabling/disablingevents, starting/stopping the tracers, taking snapshots, etc.
This chapter is a complete user guide of both activities,with common use cases of LTTng exposed throughout the text. It isassumed that you are familiar with LTTng's concepts (events, channels,domains, tracing sessions) and that you understand the roles of itscomponents (daemons, libraries, command line tools); if not, we inviteyou to read theUnderstanding LTTng chapterbefore you begin reading this one.
If you're new to LTTng, we suggest that you rather start with theGetting started small guide first, then comeback here to broaden your knowledge.
If you're only interested in tracing the Linux kernel with its currentinstrumentation, you may skip theInstrumenting section.
Instrumenting
There are many examples of tracing and monitoring in our everyday life.You have access to real-time and historical weather reports and forecaststhanks to weather stations installed around the country. You know yourpossibly hospitalized friends' and family's hearts are safe thanks toelectrocardiography. You make sure not to drive your car too fastand have enough fuel to reach your destination thanks to gauges visibleon your dashboard.
All the previous examples have something in common: they rely onprobes. Without electrodes attached to the surface of a body'sskin, cardiac monitoring would be futile.
LTTng, as a tracer, is no different from the real life examples above.If you're about to trace a software system, i.e. record its history ofexecution, you better have probes in the subject you'retracing: the actual software. Various ways were developed to do this.The most straightforward one is to manually place probes, calledtracepoints, in the software's source code. The Linux kernel tracingdomain also allows probes added dynamically.
If you're only interested in tracing the Linux kernel, it may very wellbe that your tracing needs are already appropriately covered by LTTng'sbuilt-in Linux kernel tracepoints and other probes. Or you may be inpossession of a user space application which has already beeninstrumented. In such cases, the work will reside entirely in the designand execution of tracing sessions, allowing you to jump toControlling tracing right now.
This section focuses on the following use cases of instrumentation:
- C andC++ applications
- prebuilt user space tracing helpers
- Java application
- Linux kernel module or thekernel itself
- the
/proc/lttng-loggerABI
Some advanced techniques arealso presented at the very end.
C application
Instrumenting a C (or C++) application, be it an executable program ora library, implies using LTTng-UST, theuser space tracing component of LTTng. For C/C++ applications, theLTTng-UST package includes a dynamically loaded library(liblttng-ust), C headers and the lttng-gen-tp command line utility.
Since C and C++ are the base languages of virtually all otherprogramming languages(Java virtual machine, Python, Perl, PHP and Node.js interpreters, etc.),implementing user space tracing for an unsupported language is just amatter of using the LTTng-UST C API at the right places.
The usual work flow to instrument a user space C application withLTTng-UST is:
- Define tracepoints (actual probes)
- Write tracepoint providers
- Insert tracepoints into target source code
- Package (build) tracepoint providers
- Build user application and link it with tracepoint providers
The steps above are discussed in greater detail in the followingsubsections.
Tracepoint provider
Before jumping into defining tracepoints and insertingthem into the application source code, you must understand what atracepoint provider is.
For the sake of this guide, consider the following two files:
tp.h:
#undef TRACEPOINT_PROVIDER #define TRACEPOINT_PROVIDER my_provider #undef TRACEPOINT_INCLUDE #define TRACEPOINT_INCLUDE "./tp.h" #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ) #define _TP_H #include <lttng/tracepoint.h> TRACEPOINT_EVENT( my_provider, my_first_tracepoint, TP_ARGS( int, my_integer_arg, char*, my_string_arg ), TP_FIELDS( ctf_string(my_string_field, my_string_arg) ctf_integer(int, my_integer_field, my_integer_arg) ) ) TRACEPOINT_EVENT( my_provider, my_other_tracepoint, TP_ARGS( int, my_int ), TP_FIELDS( ctf_integer(int, some_field, my_int) ) ) #endif /* _TP_H */ #include <lttng/tracepoint-event.h>
tp.c:
#define TRACEPOINT_CREATE_PROBES #include "tp.h"
The two files above are defining a tracepoint provider. A tracepointprovider is some sort of namespace fortracepoint definitions. Tracepointdefinitions are written above with theTRACEPOINT_EVENT() macro, and alloweventualtracepoint() calls respecting their definitions to be insertedinto the user application's C source code (we explore this in alater section).
Many tracepoint definitions may be part of the same tracepoint providerand many tracepoint providers may coexist in a user space application. Atracepoint provider is packaged either:
- directly into an existing user application's C source file
- as an object file
- as a static library
- as a shared library
The two files above, tp.h and tp.c, show a typical template forwriting a tracepoint provider. LTTng-UST was designed so that twotracepoint providers should not be defined in the same header file.
We will now go through the various parts of the above files andgive them a meaning. As you may have noticed, the LTTng-UST API forC/C++ applications is some preprocessor sorcery. The LTTng-UST macrosused in your application and those in the LTTng-UST headers arecombined to produce actual source code needed to make tracing possibleusing LTTng.
Let's start with the header file, tp.h. It begins with
#undef TRACEPOINT_PROVIDER #define TRACEPOINT_PROVIDER my_provider
TRACEPOINT_PROVIDER defines the name of the provider to which thefollowing tracepoint definitions will belong. It is used internally byLTTng-UST headers andmust be defined. SinceTRACEPOINT_PROVIDERcould have been defined by another header file also included by the sameC source file, the best practice is to undefine it first.
Note:Names in LTTng-UST follow the Cidentifier syntax (starting with a letter and containing eitherletters, numbers or underscores); they arenot C strings(not surrounded by double quotes). This is because LTTng-UST macrosuse those identifier-like strings to create symbols (named types andvariables).
The tracepoint provider is a group of tracepoint definitions; its chosenname should reflect this. A hierarchy like Java packages is recommended,using underscores instead of dots, e.g.,org_company_project_component.
Next is TRACEPOINT_INCLUDE:
#undef TRACEPOINT_INCLUDE #define TRACEPOINT_INCLUDE "./tp.h"
This little bit of instrospection is needed by LTTng-UST to includeyour header at various predefined places.
Include guard follows:
#if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ) #define _TP_H
Add these precompiler conditionals to ensure the tracepoint eventgeneration can include this file more than once.
The TRACEPOINT_EVENT() macro is defined in a LTTng-UST header file whichmust be included:
#include <lttng/tracepoint.h>
This will also allow the application to use the tracepoint() macro.
Next is a list of TRACEPOINT_EVENT() macro calls which create theactual tracepoint definitions. We will skip this for the moment andcome back to how to useTRACEPOINT_EVENT()in a later section. Just pay attention tothe first argument: it's always the name of the tracepoint providerbeing defined in this header file.
End of include guard:
#endif /* _TP_H */
Finally, include <lttng/tracepoint-event.h> to expand the macros:
#include <lttng/tracepoint-event.h>
That's it for tp.h. Of course, this is only a header file; it must beincluded in some C source file to actually use it. This is the job oftp.c:
#define TRACEPOINT_CREATE_PROBES #include "tp.h"
When TRACEPOINT_CREATE_PROBES is defined, the macros used in tp.h,which is included just after, will actually create the source code forLTTng-UST probes (global data structures and functions) out of yourtracepoint definitions. How exactly this is done is out of this text's scope.TRACEPOINT_CREATE_PROBES is discussed furtherin Building/linking tracepoint providers and the user application.
You could include other header files like tp.h here to create the probesof different tracepoint providers, e.g.:
#define TRACEPOINT_CREATE_PROBES #include "tp1.h" #include "tp2.h"
The rule is: probes of a given tracepoint providermust be created in exactly one source file. This source file could be oneof your project's; it doesn't have to be on its own liketp.c, althougha later sectionshows that doing so allows packaging the tracepoint providersindependently and keep them out of your application, also making itpossible to reuse them between projects.
The following sections explain how to define tracepoints, how to use thetracepoint() macro to instrument your user space C application and howto build/link tracepoint providers and your application with LTTng-USTsupport.
Using lttng-gen-tp
LTTng-UST ships with lttng-gen-tp, a handy command line utility forgenerating most of the stuff discussed above. It takes atemplate file,with a name usually ending with the.tp extension, containing onlytracepoint definitions, and outputs a tracepoint provider (either a Csource file or a precompiled object file) with its header file.
lttng-gen-tp should suffice in static linkingsituations. When using it, write a template file containing a list ofTRACEPOINT_EVENT() macro calls. The tool will find the provider namesused and generate the appropriate files which are going to look a lotliketp.h and tp.c above.
Just call lttng-gen-tp like this:
lttng-gen-tp my-template.tp
my-template.c, my-template.o and my-template.h will be createdin the same directory.
You may specify custom C flags passed to the compiler invoked bylttng-gen-tp using theCFLAGS environment variable:
CFLAGS=-I/custom/include/path lttng-gen-tp my-template.tp
For more information on lttng-gen-tp, seeits manpage.
Defining tracepoints
As written in Tracepoint provider,tracepoints are defined using theTRACEPOINT_EVENT() macro. Each tracepoint, when called using thetracepoint() macro in the actual application's source code, generatesa specific event type with its own fields.
Let's have another look at the example above, with a few added comments:
TRACEPOINT_EVENT(
/* tracepoint provider name */
my_provider,
/* tracepoint/event name */
my_first_tracepoint,
/* list of tracepoint arguments */
TP_ARGS(
int, my_integer_arg,
char*, my_string_arg
),
/* list of fields of eventual event */
TP_FIELDS(
ctf_string(my_string_field, my_string_arg)
ctf_integer(int, my_integer_field, my_integer_arg)
)
)
The tracepoint provider name must match the name of the tracepointprovider in which this tracepoint is defined(seeTracepoint provider). In other words,always use the same string as the value ofTRACEPOINT_PROVIDER above.
The tracepoint name will become the event name once events are recordedby the LTTng-UST tracer. It must follow the tracepoint provider namesyntax: start with a letter and contain either letters, numbers orunderscores. Two tracepoints under the same provider cannot have thesame name, i.e. you cannot overload a tracepoint like you wouldoverload functions and methods in C++/Java.
Note:The concatenation of the tracepointprovider name and the tracepoint name cannot exceed 254 characters. Ifit does, the instrumented application will compile and run, but LTTngwill issue multiple warnings and you could experience serious problems.
The list of tracepoint arguments gives this tracepoint its signature:see it like the declaration of a C function. The format ofTP_ARGS()arguments is: C type, then argument name; repeat as needed, up to tentimes. For example, if we were to replicate the signature of C standardlibrary'sfseek(), the TP_ARGS() part would look like:
TP_ARGS(
FILE*, stream,
long int, offset,
int, origin
),
Of course, you will need to include appropriate header files beforethe TRACEPOINT_EVENT() macro calls if any argument has a complex type.
TP_ARGS() may not be omitted, but may be empty. TP_ARGS(void) isalso accepted.
The list of fields is where the fun really begins. The fields definedin this list will be the fields of the events generated by the executionof this tracepoint. Each tracepoint field definition has a Cargument expression which will be evaluated when the execution reachesthe tracepoint. Tracepoint arguments may be used freely in thoseargument expressions, but theydon't have to.
There are several types of tracepoint fields available. The macros todefine them are given and explained in theLTTng-UST library reference section.
Field names must follow the standard C identifier syntax: letter, thenoptional sequence of letters, numbers or underscores. Each field must havea different name.
Those ctf_*() macros are added to the TP_FIELDS() part ofTRACEPOINT_EVENT(). Note that they are not delimited by commas.TP_FIELDS() may be empty, but theTP_FIELDS(void) form isnotaccepted.
The following snippet shows how argument expressions may be used intracepoint fields and how they may refer freely to tracepoint arguments.
/* for struct stat */ #include <sys/types.h> #include <sys/stat.h> #include <unistd.h> TRACEPOINT_EVENT( my_provider, my_tracepoint, TP_ARGS( int, my_int_arg, char*, my_str_arg, struct stat*, st ), TP_FIELDS( /* simple integer field with constant value */ ctf_integer( int, /* field C type */ my_constant_field, /* field name */ 23 + 17 /* argument expression */ ) /* my_int_arg tracepoint argument */ ctf_integer( int, my_int_arg_field, my_int_arg ) /* my_int_arg squared */ ctf_integer( int, my_int_arg_field2, my_int_arg * my_int_arg ) /* sum of first 4 characters of my_str_arg */ ctf_integer( int, sum4, my_str_arg[0] + my_str_arg[1] + my_str_arg[2] + my_str_arg[3] ) /* my_str_arg as string field */ ctf_string( my_str_arg_field, /* field name */ my_str_arg /* argument expression */ ) /* st_size member of st tracepoint argument, hexadecimal */ ctf_integer_hex( off_t, /* field C type */ size_field, /* field name */ st->st_size /* argument expression */ ) /* st_size member of st tracepoint argument, as double */ ctf_float( double, /* field C type */ size_dbl_field, /* field name */ (double) st->st_size /* argument expression */ ) /* half of my_str_arg string as text sequence */ ctf_sequence_text( char, /* element C type */ half_my_str_arg_field, /* field name */ my_str_arg, /* argument expression */ size_t, /* length expression C type */ strlen(my_str_arg) / 2 /* length expression */ ) ) )
As you can see, having a custom argument expression for each fieldmakes tracepoints very flexible for tracing a user space C application.This tracepoint definition is reused later in this guide, whenactually using tracepoints in a user space application.
Using tracepoint classes
In LTTng-UST, a tracepoint class is a class of tracepoints sharing thesame field types and names. Atracepoint instance is one instance ofsuch a declared tracepoint class, with its own event name and tracepointprovider name.
What is documented in Defining tracepointsis actually how to declare a tracepoint class and define atracepoint instance at the same time. Without revealing the internalsof LTTng-UST too much, it has to be noted that one serializationfunction is created for each tracepoint class. A serializationfunction is responsible for serializing the fields of a tracepointinto a sub-buffer when tracing. For various performance reasons, whenyour situation requires multiple tracepoints with different names, butwith the same fields layout, the best practice is to manually createa tracepoint class and instantiate as many tracepoint instances asneeded. One positive effect of such a design, amongst other advantages,is that all tracepoint instances of the same tracepoint class willreuse the same serialization function, thus reducing cache pollution.
As an example, here are three tracepoint definitions as we know them:
TRACEPOINT_EVENT(
my_app,
get_account,
TP_ARGS(
int, userid,
size_t, len
),
TP_FIELDS(
ctf_integer(int, userid, userid)
ctf_integer(size_t, len, len)
)
)
TRACEPOINT_EVENT(
my_app,
get_settings,
TP_ARGS(
int, userid,
size_t, len
),
TP_FIELDS(
ctf_integer(int, userid, userid)
ctf_integer(size_t, len, len)
)
)
TRACEPOINT_EVENT(
my_app,
get_transaction,
TP_ARGS(
int, userid,
size_t, len
),
TP_FIELDS(
ctf_integer(int, userid, userid)
ctf_integer(size_t, len, len)
)
)
In this case, three tracepoint classes are created, with one tracepointinstance for each of them:get_account,get_settings andget_transaction. However, they all share the same field names andtypes. Declaring one tracepoint class and three tracepoint instances ofthe latter is a better design choice:
/* the tracepoint class */ TRACEPOINT_EVENT_CLASS( /* tracepoint provider name */ my_app, /* tracepoint class name */ my_class, /* arguments */ TP_ARGS( int, userid, size_t, len ), /* fields */ TP_FIELDS( ctf_integer(int, userid, userid) ctf_integer(size_t, len, len) ) ) /* the tracepoint instances */ TRACEPOINT_EVENT_INSTANCE( /* tracepoint provider name */ my_app, /* tracepoint class name */ my_class, /* tracepoint/event name */ get_account, /* arguments */ TP_ARGS( int, userid, size_t, len ) ) TRACEPOINT_EVENT_INSTANCE( my_app, my_class, get_settings, TP_ARGS( int, userid, size_t, len ) ) TRACEPOINT_EVENT_INSTANCE( my_app, my_class, get_transaction, TP_ARGS( int, userid, size_t, len ) )
Of course, all those names and TP_ARGS() invocations are redundant,but some C preprocessor magic can solve this:
#define MY_TRACEPOINT_ARGS \ TP_ARGS( \ int, userid, \ size_t, len \ ) TRACEPOINT_EVENT_CLASS( my_app, my_class, MY_TRACEPOINT_ARGS, TP_FIELDS( ctf_integer(int, userid, userid) ctf_integer(size_t, len, len) ) ) #define MY_APP_TRACEPOINT_INSTANCE(name) \ TRACEPOINT_EVENT_INSTANCE( \ my_app, \ my_class, \ name, \ MY_TRACEPOINT_ARGS \ ) MY_APP_TRACEPOINT_INSTANCE(get_account) MY_APP_TRACEPOINT_INSTANCE(get_settings) MY_APP_TRACEPOINT_INSTANCE(get_transaction)
Assigning log levels to tracepoints
Optionally, a log level can be assigned to a defined tracepoint.Assigning different levels of importance to tracepoints can be useful;when controlling tracing sessions,you can choose to only enable tracepointsfalling into a specific log level range.
Log levels are assigned to defined tracepoints using theTRACEPOINT_LOGLEVEL() macro. The latter must be usedafter havingusedTRACEPOINT_EVENT() for a given tracepoint. TheTRACEPOINT_LOGLEVEL() macro has the following construct:
TRACEPOINT_LOGLEVEL(<provider name>, <tracepoint name>, <log level>)
where the first two arguments are the same as the first two argumentsof TRACEPOINT_EVENT() and <log level> is oneof the values given in theLTTng-UST library referencesection.
As an example, let's assign a TRACE_DEBUG_UNIT log level to ourprevious tracepoint definition:
TRACEPOINT_LOGLEVEL(my_provider, my_tracepoint, TRACE_DEBUG_UNIT)
Probing the application's source code
Once tracepoints are properly defined within a tracepoint provider,they may be inserted into the user application to be instrumentedusing thetracepoint() macro. Its first argument is the tracepointprovider name and its second is the tracepoint name. The next, optionalarguments are defined by theTP_ARGS() part of the definition ofthe tracepoint to use.
As an example, let us again take the following tracepoint definition:
TRACEPOINT_EVENT(
/* tracepoint provider name */
my_provider,
/* tracepoint/event name */
my_first_tracepoint,
/* list of tracepoint arguments */
TP_ARGS(
int, my_integer_arg,
char*, my_string_arg
),
/* list of fields of eventual event */
TP_FIELDS(
ctf_string(my_string_field, my_string_arg)
ctf_integer(int, my_integer_field, my_integer_arg)
)
)
Assuming this is part of a file named tp.h which defines the tracepointprovider and which is included bytp.c, here's a complete C applicationcalling this tracepoint (multiple times):
#define TRACEPOINT_DEFINE #include "tp.h" int main(int argc, char* argv[]) { int i; tracepoint(my_provider, my_first_tracepoint, 23, "Hello, World!"); for (i = 0; i < argc; ++i) { tracepoint(my_provider, my_first_tracepoint, i, argv[i]); } return 0; }
For each tracepoint provider, TRACEPOINT_DEFINE must be defined intoexactly one translation unit (C source file) of the user application,before including the tracepoint provider header file. In other words,for a given tracepoint provider, you cannot define TRACEPOINT_DEFINE,and then include its header file in two separate C source files ofthe same application.TRACEPOINT_DEFINE is discussed further inBuilding/linking tracepoint providers and the user application.
As another example, remember this definition we wrote in a previoussection (comments are stripped):
/* for struct stat */ #include <sys/types.h> #include <sys/stat.h> #include <unistd.h> TRACEPOINT_EVENT( my_provider, my_tracepoint, TP_ARGS( int, my_int_arg, char*, my_str_arg, struct stat*, st ), TP_FIELDS( ctf_integer(int, my_constant_field, 23 + 17) ctf_integer(int, my_int_arg_field, my_int_arg) ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg) ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] + my_str_arg[2] + my_str_arg[3]) ctf_string(my_str_arg_field, my_str_arg) ctf_integer_hex(off_t, size_field, st->st_size) ctf_float(double, size_dbl_field, (double) st->st_size) ctf_sequence_text(char, half_my_str_arg_field, my_str_arg, size_t, strlen(my_str_arg) / 2) ) )
Here's an example of calling it:
#define TRACEPOINT_DEFINE #include "tp.h" int main(void) { struct stat s; stat("/etc/fstab", &s); tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s); return 0; }
When viewing the trace, assuming the file size of /etc/fstab is301 bytes, the event generated by the execution of this tracepointshould have the following fields, in this order:
my_constant_field 40 my_int_arg_field 23 my_int_arg_field2 529 sum4_field 389 my_str_arg_field "Hello, World!" size_field 0x12d size_dbl_field 301.0 half_my_str_arg_field "Hello,"
Building/linking tracepoint providers and the user application
This section explains the final step of using LTTng-UST for tracinga user space C application (beside running the application): building andlinking tracepoint providers and the application itself.
As discussed above, the macros used by the user-written tracepoint providerheader file are useless until actually used to create probes code(global data structures and functions) in a translation unit (C source file).This is accomplished by definingTRACEPOINT_CREATE_PROBES in a translationunit and then including the tracepoint provider header file.WhenTRACEPOINT_CREATE_PROBES is defined, macros used and included bythe tracepoint provider header will output actual source code needed by anyapplication using the defined tracepoints. DefiningTRACEPOINT_CREATE_PROBES produces code used when registeringtracepoint providers when the tracepoint provider package loads.
The other important definition is TRACEPOINT_DEFINE. This one createsglobal, per-tracepoint structures referencing the tracepoint providersdata. Those structures are required by the actual functions insertedwheretracepoint() macros are placed and need to be defined by theinstrumented application.
Both TRACEPOINT_CREATE_PROBES and TRACEPOINT_DEFINE need to be definedat some places in order to trace a user space C application using LTTng.Although explaining their exact mechanism is beyond the scope of thisdocument, the reason they both exist separately is to allow the traceproviders to be packaged as a shared object (dynamically loaded library).
There are two ways to compile and link the tracepoint providerswith the application:statically ordynamically. Both methods are covered in thefollowing subsections.
Static linking
With the static linking method, compiled tracepoint providers are copiedinto the target application. There are three ways to do this:
- Use one of your existing C source files to create probes.
- Create probes in a separate C source file and build it as anobject file to be linked with the application (more decoupled).
- Create probes in a separate C source file, build it as anobject file and archive it to create astatic library(more decoupled, more portable).
The first approach is to define TRACEPOINT_CREATE_PROBES and includeyour tracepoint provider(s) header file(s) directly into an existing Csource file. Here's an example:
#include <stdlib.h> #include <stdio.h> /* ... */ #define TRACEPOINT_CREATE_PROBES #define TRACEPOINT_DEFINE #include "tp.h" /* ... */ int my_func(int a, const char* b) { /* ... */ tracepoint(my_provider, my_tracepoint, buf, sz, limit, &tt) /* ... */ } /* ... */
Again, before including a given tracepoint provider header file,TRACEPOINT_CREATE_PROBES andTRACEPOINT_DEFINE must be defined inone,and only one, translation unit. Other C source files of thesame application may includetp.h to use tracepoints withthetracepoint() macro, but must not defineTRACEPOINT_CREATE_PROBES/TRACEPOINT_DEFINE again.
This translation unit may be built as an object file by making sure toadd . to the include path:
gcc -c -I. file.c
The second approach is to isolate the tracepoint provider code into aseparate object file by using a dedicated C source file to create probes:
#define TRACEPOINT_CREATE_PROBES #include "tp.h"
TRACEPOINT_DEFINE must be defined by a translation unit of theapplication. Since we're talking about static linking here, it could aswell be defined directly in the file above, before#include "tp.h":
#define TRACEPOINT_CREATE_PROBES #define TRACEPOINT_DEFINE #include "tp.h"
This is actually what lttng-gen-tp does, and isthe recommended practice.
Build the tracepoint provider:
gcc -c -I. tp.c
Finally, the resulting object file may be archived to create amore portable tracepoint provider static library:
ar rc tp.a tp.o
Using a static library does have the advantage of centralising thetracepoint providers objects so they can be shared between multipleapplications. This way, when the tracepoint provider is modified, thesource code changes don't have to be patched into each application's sourcecode tree. The applications need to be relinked after each change, but neednot to be otherwise recompiled (unless the tracepoint provider's APIchanges).
Regardless of which method you choose, you end up with an object file(potentially archived) containing the trace providers assembled code.To link this code with the rest of your application, you must also linkwithliblttng-ust andlibdl:
gcc -o app tp.o other.o files.o of.o your.o app.o -llttng-ust -ldl
or
gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -ldl
If you're using a BSDsystem, replace-ldl with-lc:
gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -lc
The application can be started as usual, e.g.:
./app
The lttng command line tool can be used tocontrol tracing.
Dynamic linking
The second approach to package the tracepoint providers is to usedynamic linking: the library and its member functions are explicitlysought, loaded and unloaded at runtime usinglibdl.
It has to be noted that, for a variety of reasons, the created sharedlibrary will be dynamicallyloaded, as opposed to dynamicallylinked. The tracepoint provider shared object is, however, linkedwithliblttng-ust, so thatliblttng-ust is guaranteed to be loadedas soon as the tracepoint provider is. If the tracepoint provider isnot loaded, since the application itself is not linked withliblttng-ust, the latter is not loaded at all and the tracepoint callsbecome inert.
The process to create the tracepoint provider shared object is prettymuch the same as the static library method, except that:
- since the tracepoint provider is not part of the applicationanymore,
TRACEPOINT_DEFINEmust be defined, for each tracepointprovider, in exactly one translation unit (C source file) of theapplication; TRACEPOINT_PROBE_DYNAMIC_LINKAGEmust be defined next toTRACEPOINT_DEFINE.
Regarding TRACEPOINT_DEFINE and TRACEPOINT_PROBE_DYNAMIC_LINKAGE,the recommended practice is to use a separate C source file in yourapplication to define them, and then include the tracepoint providerheader files afterwards, e.g.:
#define TRACEPOINT_DEFINE #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE /* include the header files of one or more tracepoint providers below */ #include "tp1.h" #include "tp2.h" #include "tp3.h"
TRACEPOINT_PROBE_DYNAMIC_LINKAGE makes the macros included afterwards(by including the tracepoint provider header, which itself includesLTTng-UST headers) aware that the tracepoint provider is to be loadeddynamically and not part of the application's executable.
The tracepoint provider object file used to create the shared libraryis built like it is using the static library method, only with the-fpic option added:
gcc -c -fpic -I. tp.c
It is then linked as a shared library like this:
gcc -shared -Wl,--no-as-needed -o tp.so -llttng-ust tp.o
As previously stated, this tracepoint provider shared object isn'tlinked with the user application: it will be loaded manually. This iswhy the application is built with no mention of this tracepointprovider, but still needslibdl:
gcc -o app other.o files.o of.o your.o app.o -ldl
Now, to make LTTng-UST tracing available to the application, theLD_PRELOAD environment variable is used to preload the tracepointprovider shared librarybefore the application actually starts:
LD_PRELOAD=/path/to/tp.so ./app
Note:It is not safe to use dlclose() on a tracepoint provider shared object that is being actively used for tracing, due to a lack of reference counting from LTTng-UST to the shared object.
For example, statically linking a tracepoint provider to a shared object which is to be dynamically loaded by an application (e.g., a plugin) is not safe: the shared object, which contains the tracepoint provider, could be dynamically closed (dlclose()) at any time by the application.
To instrument a shared object, either:
- Statically link the tracepoint provider to the application, or
- Build the tracepoint provider as a shared object (following the procedure shown in this section), and preload it when tracing is needed using the
LD_PRELOADenvironment variable.
Your application will still work without this preloading, albeit withoutLTTng-UST tracing support:
./app
Using LTTng-UST with daemons
Some extra care is needed when using liblttng-ust with daemonapplications that callfork(),clone() or BSD's rfork() withouta followingexec() family system call. Theliblttng-ust-forklibrary must be preloaded for the application.
Example:
LD_PRELOAD=liblttng-ust-fork.so ./app
Or, if you're using a tracepoint provider shared library:
LD_PRELOAD="liblttng-ust-fork.so /path/to/tp.so" ./app
pkg-config
On some distributions, LTTng-UST is shipped with a pkg-config metadatafile, so that you may use thepkg-config tool:
pkg-config --libs lttng-ust
This will return -llttng-ust -ldl on Linux systems.
You may also check the LTTng-UST version using pkg-config:
pkg-config --modversion lttng-ust
For more information about pkg-config, seeits manpage.
Using tracef()
tracef() is a small LTTng-UST API to avoid defining your owntracepoints and tracepoint providers. The signature oftracef() isthe same asprintf()'s.
The tracef() utility function was developed to make user space tracingsuper simple, albeit with notable disadvantages compared to custom,full-fledged tracepoint providers:
- All generated events have the same provider/event names, respectively
lttng-ust-tracefandevent. - There's no static type checking.
- The only event field you actually get, named
msg, is a stringpotentially containing the values you passed to the functionusing your own format. This also means that you cannot use filteringusing a custom expression at runtime because there are no isolatedfields. - Since
tracef()uses C standard library'svasprintf()functionin the background to format the strings at runtime, itsexpected performance is lower than using custom tracepoint providerswith typed fields, which do not require a conversion to a string.
Thus, tracef() is useful for quick prototyping and debugging, butshould not be considered for any permanent/serious applicationinstrumentation.
To use tracef(), first include <lttng/tracef.h> in the C source filewhere you need to insert probes:
#include <lttng/tracef.h>
Use tracef() like you would use printf() in your source code, e.g.:
/* ... */ tracef("my message, my integer: %d", my_integer); /* ... */
Link your application with liblttng-ust:
gcc -o app app.c -llttng-ust
Execute the application as usual:
./app
Voilà! Use the lttng command line tool tocontrol tracing.
LTTng-UST environment variables and special compilation flags
A few special environment variables and compile flags may affect thebehavior of LTTng-UST.
LTTng-UST's debugging can be activated by setting the environmentvariable LTTNG_UST_DEBUG to 1 when launching the application. Itcan also be enabled at compile time by definingLTTNG_UST_DEBUG whencompiling LTTng-UST (using the-DLTTNG_UST_DEBUG compiler option).
The environment variable LTTNG_UST_REGISTER_TIMEOUT can be used tospecify how long the application should wait for thesession daemon'sregistration done commandbefore proceeding to execute the main program. The timeout value isspecified in milliseconds. 0 meansdon't wait. -1 meanswait forever. Setting this environment variable to 0 is recommendedfor applications with time contraints on the process startup time.
The default value of LTTNG_UST_REGISTER_TIMEOUT (when not defined)is3000 ms.
The compilation definition LTTNG_UST_DEBUG_VALGRIND should be enabledat build time (-DLTTNG_UST_DEBUG_VALGRIND) to allowliblttng-ustto be used withValgrind.The side effect of definingLTTNG_UST_DEBUG_VALGRIND is that per-CPUbuffering is disabled.
C++ application
Because of C++'s cross-compatibility with the C language, C++applications can be readily instrumented with the LTTng-UST C API.
Follow the C application user guide above. Itshould be noted that, in this case, tracepoint providers should havethe typical.cpp,.cxx or .cc extension and be built withg++instead ofgcc. This is the easiest way of avoiding linking errorsdue to symbol name mangling incompatibilities between both languages.
Prebuilt user space tracing helpers
The LTTng-UST package provides a few helpers that one may finduseful in some situations. They all work the same way: you mustpreload the appropriate shared object before running the userapplication (using theLD_PRELOAD environment variable).
The shared objects are normally found in /usr/lib.
The current installed helpers are:
liblttng-ust-libc-wrapper.soandliblttng-ust-pthread-wrapper.so:C standard library and POSIX threads tracingliblttng-ust-cyg-profile.soandliblttng-ust-cyg-profile-fast.so:function tracingliblttng-ust-dl.so:dynamic linker tracing
The following subsections document what helpers instrument exactlyand how to use them.
C standard library and POSIX threads tracing
liblttng-ust-libc-wrapper.so and liblttng-ust-pthread-wrapper.socan add instrumentation to respectively some C standard library andPOSIX threads functions.
The following functions are traceable by liblttng-ust-libc-wrapper.so:
| TP provider name | TP name | Instrumented function |
|---|---|---|
ust_libc | malloc | malloc() |
calloc | calloc() | |
realloc | realloc() | |
free | free() | |
memalign | memalign() | |
posix_memalign | posix_memalign() |
The following functions are traceable byliblttng-ust-pthread-wrapper.so:
| TP provider name | TP name | Instrumented function |
|---|---|---|
ust_pthread | pthread_mutex_lock_req | pthread_mutex_lock() (request time) |
pthread_mutex_lock_acq | pthread_mutex_lock() (acquire time) | |
pthread_mutex_trylock | pthread_mutex_trylock() | |
pthread_mutex_unlock | pthread_mutex_unlock() |
All tracepoints have fields corresponding to the arguments of thefunction they instrument.
To use one or the other with any user application, independently ofhow the latter is built, do:
LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
or
LD_PRELOAD=liblttng-ust-pthread-wrapper.so my-app
To use both, do:
LD_PRELOAD="liblttng-ust-libc-wrapper.so liblttng-ust-pthread-wrapper.so" my-app
When the shared object is preloaded, it effectively replaces thefunctions listed in the above tables by wrappers which add tracepointsand call the replaced functions.
Of course, like any other tracepoint, the ones above need to be enabledin order for LTTng-UST to generate events. This is done using thelttng command line tool(seeControlling tracing).
Function tracing
Function tracing is the recording of which functions are entered andleft during the execution of an application. Like with any LTTng event,the precise time at which this happens is also kept.
GCC and clang have an option named-finstrument-functionswhich generates instrumentation calls for entry and exit to functions.The LTTng-UST function tracing helpers, liblttng-ust-cyg-profile.soandliblttng-ust-cyg-profile-fast.so, take advantage of this featureto add instrumentation to the two generated functions (which containcyg_profile in their names, hence the shared object's name).
In order to use LTTng-UST function tracing, the translation units toinstrument must be built using the-finstrument-functions compilerflag.
LTTng-UST function tracing comes in two flavors, each providingdifferent trade-offs:liblttng-ust-cyg-profile-fast.so andliblttng-ust-cyg-profile.so.
liblttng-ust-cyg-profile-fast.so is a lightweight variant thatshould only be used where it can beguaranteed that the complete eventstream is recorded without any missing events. Any kind of duplicateinformation is left out. This version registers the followingtracepoints:
| TP provider name | TP name | Description/fields |
|---|---|---|
lttng_ust_cyg_profile_fast | func_entry | Function entry
|
func_exit | Function exit |
Assuming no event is lost, having only the function addresses on entryis enough for creating a call graph (remember that a recorded eventalways contains the ID of the CPU that generated it). A tool likeaddr2linemay be used to convert function addresses back to source files namesand line numbers.
The other helper,liblttng-ust-cyg-profile.so,is a more robust variant which also works for use cases whereevents might get discarded or not recorded from application startup.In these cases, the trace analyzer needs extra information to beable to reconstruct the program flow. This version registers thefollowing tracepoints:
| TP provider name | TP name | Description/fields |
|---|---|---|
lttng_ust_cyg_profile | func_entry | Function entry
|
func_exit | Function exit
|
To use one or the other variant with any user application, assuming atleast one translation unit of the latter is compiled with the-finstrument-functions option, do:
LD_PRELOAD=liblttng-ust-cyg-profile-fast.so my-app
or
LD_PRELOAD=liblttng-ust-cyg-profile.so my-app
It might be necessary to limit the number of source files where-finstrument-functions is used to prevent excessive amount of tracedata to be generated at runtime.
Tip: When using GCC, at least, you may use the -finstrument-functions-exclude-function-list option to avoid instrumenting entries and exits of specific symbol names.
All events generated from LTTng-UST function tracing are provided onlog level TRACE_DEBUG_FUNCTION, which is useful to easily enablefunction tracing events in your tracing session using the--loglevel-only option oflttng enable-event(seeControlling tracing).
Dynamic linker tracing
This LTTng-UST helper causes all calls to dlopen() and dlclose()in the target application to be traced with LTTng.
The helper's shared object, liblttng-ust-dl.so, registers thefollowing tracepoints when preloaded:
| TP provider name | TP name | Description/fields |
|---|---|---|
ust_baddr | push |
|
pop |
|
To use this LTTng-UST helper with any user application, independently ofhow the latter is built, do:
LD_PRELOAD=liblttng-ust-dl.so my-app
Of course, like any other tracepoint, the ones above need to be enabledin order for LTTng-UST to generate events. This is done using thelttng command line tool(seeControlling tracing).
Java application
LTTng-UST provides a logging back-end for Java applications usingeitherjava.util.logging(JUL), orApache log4j 1.2.This back-end is called the LTTng-UST Java agent, and is responsiblefor communications with an LTTng session daemon.
Note:The latest stable version of LTTng does not support Log4j 2.
From the user's point of view, once the LTTng-UST Java agent has beeninitialized, JUL and log4j loggers may be created and used as usual.The agent adds its own handler to theroot logger, so that allloggers may generate LTTng events with no effort.
Common JUL/log4j features are supported using the lttng tool(see Controlling tracing):
- listing all logger names
- enabling/disabling events per logger name
- JUL/log4j log levels
Here's an example using java.util.logging:
import java.util.logging.Logger; import org.lttng.ust.agent.LTTngAgent; public class Test { private static final int answer = 42; public static void main(String[] argv) throws Exception { // create a logger Logger logger = Logger.getLogger("jello"); // call this as soon as possible (before logging) LTTngAgent lttngAgent = LTTngAgent.getLTTngAgent(); // log at will! logger.info("some info"); logger.warning("some warning"); Thread.sleep(500); logger.finer("finer information; the answer is " + answer); Thread.sleep(123); logger.severe("error!"); // not mandatory, but cleaner lttngAgent.dispose(); } }
Here's the same example, this time using log4j:
import org.apache.log4j.Logger; import org.apache.log4j.BasicConfigurator; import org.lttng.ust.agent.LTTngAgent; public class Test { private static final int answer = 42; public static void main(String[] argv) throws Exception { // create and configure a logger Logger logger = Logger.getLogger(Test.class); BasicConfigurator.configure(); // call this as soon as possible (before logging) LTTngAgent lttngAgent = LTTngAgent.getLTTngAgent(); // log at will! logger.info("some info"); logger.warn("some warning"); Thread.sleep(500); logger.debug("debug information; the answer is " + answer); Thread.sleep(123); logger.error("error!"); logger.fatal("fatal error!"); // not mandatory, but cleaner lttngAgent.dispose(); } }
The LTTng-UST Java agent classes are packaged in a JAR file namedliblttng-ust-agent.jar. It is typically located in/usr/lib/lttng/java. To compile the snippets above(saved asTest.java), do:
javac -cp /usr/lib/lttng/java/liblttng-ust-agent.jar:$LOG4JCP Test.java
where $LOG4JCP is the log4j 1.2 JAR file path, if you're using log4j.
You can run the resulting compiled class like this:
java -cp /usr/lib/lttng/java/liblttng-ust-agent.jar:$LOG4JCP:. Test
Note:OpenJDK 7 is used for development and continuous integration, thus this version is directly supported. However, the LTTng-UST Java agent has also been tested with OpenJDK 6.
Linux kernel
The Linux kernel can be instrumented for LTTng tracing, either its coresource code or a kernel module. It has to be noted that Linux isreadily traceable using LTTng since many parts of its source code arealready instrumented: this is the job of the upstreamLTTng-modulespackage. This section presents how to add LTTng instrumentation where itdoes not currently exist and how to instrument custom kernel modules.
All LTTng instrumentation in the Linux kernel is based on an existinginfrastructure which bears the name of its main macro,TRACE_EVENT().This macro is used to define tracepoints,each tracepoint having a name, usually with thesubsys_name format,subsys being the subsystem name andname the specific event name.
Tracepoints defined with TRACE_EVENT() may be inserted anywhere inthe Linux kernel source code, after what callbacks, calledprobes,may be registered to execute some action when a tracepoint isexecuted. This mechanism is directly used by ftrace and perf,but cannot be used as is by LTTng: an adaptation layer is added tosatisfy LTTng's specific needs.
With that in mind, this documentation does not cover the TRACE_EVENT()format and how to use it, but it is mandatory to understand it and useit to instrument Linux for LTTng. A series ofLWN articles explainTRACE_EVENT() in details:part 1,part 2, andpart 3.Once you master TRACE_EVENT() enough for your use case, continuereading this section so that you can add the LTTng adaptation layer ofinstrumentation.
This section first discusses the general method of instrumenting theLinux kernel for LTTng. This method is then reused for the specificcase of instrumenting a kernel module.
Instrumenting the Linux kernel for LTTng
This section explains strictly how to add custom LTTnginstrumentation to the Linux kernel. It does not explain how themacros actually work and the internal mechanics of the tracer.
You should have a Linux kernel source code tree to work with.Throughout this section, all file paths are relative to the root ofthis tree unless otherwise stated.
You will need a copy of the LTTng-modules Git repository:
git clone git://git.lttng.org/lttng-modules.git
The steps to add custom LTTng instrumentation to a Linux kernelinvolves defining and using the mainlineTRACE_EVENT() tracepointsfirst, then writing and using the LTTng adaptation layer.
Defining/using tracepoints with mainline TRACE_EVENT() infrastructure
The first step is to define tracepoints using the mainline LinuxTRACE_EVENT() macro and insert tracepoints where you want them.Your tracepoint definitions reside in a header file ininclude/trace/events. If you're adding tracepoints to an existingsubsystem, edit its appropriate header file.
As an example, the following header file (let's call itinclude/trace/events/hello.h) defines one tracepoint usingTRACE_EVENT():
/* subsystem name is "hello" */ #undef TRACE_SYSTEM #define TRACE_SYSTEM hello #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_HELLO_H #include <linux/tracepoint.h> TRACE_EVENT( /* "hello" is the subsystem name, "world" is the event name */ hello_world, /* tracepoint function prototype */ TP_PROTO(int foo, const char* bar), /* arguments for this tracepoint */ TP_ARGS(foo, bar), /* LTTng doesn't need those */ TP_STRUCT__entry(), TP_fast_assign(), TP_printk("", 0) ); #endif /* this part must be outside protection */ #include <trace/define_trace.h>
Notice that we don't use any of the last three arguments: theyare left empty here because LTTng doesn't need them. You would only fillTP_STRUCT__entry(),TP_fast_assign() andTP_printk() if you wereto also use this tracepoint for ftrace/perf.
Once this is done, you may place calls to trace_hello_world()wherever you want in the Linux source code. As an example, let us placesuch a tracepoint in theusb_probe_device() static function(drivers/usb/core/driver.c):
/* called from driver core with dev locked */ static int usb_probe_device(struct device *dev) { struct usb_device_driver *udriver = to_usb_device_driver(dev->driver); struct usb_device *udev = to_usb_device(dev); int error = 0; trace_hello_world(udev->devnum, udev->product); /* ... */ }
This tracepoint should fire every time a USB device is plugged in.
At the top of driver.c, we need to include our actual tracepointdefinition and, in this case (one place per subsystem), defineCREATE_TRACE_POINTS, which will create our tracepoint:
/* ... */ #include "usb.h" #define CREATE_TRACE_POINTS #include <trace/events/hello.h> /* ... */
Build your custom Linux kernel. In order to use LTTng, make sure thefollowing kernel configuration options are enabled:
CONFIG_MODULES(loadable module support)CONFIG_KALLSYMS(load all symbols for debugging/kksymoops)CONFIG_HIGH_RES_TIMERS(high resolution timer support)CONFIG_TRACEPOINTS(kernel tracepoint instrumentation)
Boot the custom kernel. The directory/sys/kernel/debug/tracing/events/hello should exist if everythingwent right, with ahello_world subdirectory.
Adding the LTTng adaptation layer
The steps to write the LTTng adaptation layer are, in yourLTTng-modules copy's source code tree:
- In
instrumentation/events/lttng-module,add a headersubsys.hfor your customsubsystemsubsysand write yourtracepoint definitions using LTTng-modules macros in it.Those macros look like the mainline kernel equivalents,but they present subtle, yet important differences. - In
probes, create the C source file of the LTTng probe kernelmodule for your subsystem. It should be namedlttng-probe-subsys.c. - Edit
probes/Makefileso that the LTTng-modules projectbuilds your custom LTTng probe kernel module. - Build and install LTTng kernel modules.
Following our hello_world event example, here's the content ofinstrumentation/events/lttng-module/hello.h:
#undef TRACE_SYSTEM #define TRACE_SYSTEM hello #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_HELLO_H #include <linux/tracepoint.h> LTTNG_TRACEPOINT_EVENT( /* format identical to mainline version for those */ hello_world, TP_PROTO(int foo, const char* bar), TP_ARGS(foo, bar), /* possible differences */ TP_STRUCT__entry( __field(int, my_int) __field(char, char0) __field(char, char1) __string(product, bar) ), /* notice the use of tp_assign()/tp_strcpy() and no semicolons */ TP_fast_assign( tp_assign(my_int, foo) tp_assign(char0, bar[0]) tp_assign(char1, bar[1]) tp_strcpy(product, bar) ), /* This one is actually not used by LTTng either, but must be * present for the moment. */ TP_printk("", 0) /* no semicolon after this either */ ) #endif /* other difference: do NOT include <trace/define_trace.h> */ #include "../../../probes/define_trace.h"
Some possible entries for TP_STRUCT__entry() and TP_fast_assign(),in the case of LTTng-modules, are shown in theLTTng-modules reference section.
The best way to learn how to use the above macros is to inspectexisting LTTng tracepoint definitions ininstrumentation/events/lttng-moduleheader files. Compare them with the Linux kernel mainline versionsininclude/trace/events.
The next step is writing the LTTng probe kernel module C source file.This one is namedlttng-probe-subsys.cinprobes. You may always use the following template:
#include <linux/module.h> #include "../lttng-tracer.h" /* Build time verification of mismatch between mainline TRACE_EVENT() * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments. */ #include <trace/events/hello.h> /* create LTTng tracepoint probes */ #define LTTNG_PACKAGE_BUILD #define CREATE_TRACE_POINTS #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module #include "../instrumentation/events/lttng-module/hello.h" MODULE_LICENSE("GPL and additional rights"); MODULE_AUTHOR("Your name <your-email>"); MODULE_DESCRIPTION("LTTng hello probes"); MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "." __stringify(LTTNG_MODULES_MINOR_VERSION) "." __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION) LTTNG_MODULES_EXTRAVERSION);
Just replace hello with your subsystem name. In this example,<trace/events/hello.h>, which is the original mainline tracepointdefinition header, is included for verification purposes: theLTTng-modules build system is able to emit an error at build time whenthe arguments of the mainline TRACE_EVENT() definitions do not matchthe ones of the LTTng-modules adaptation layer(LTTNG_TRACEPOINT_EVENT()).
Edit probes/Makefile and add your new kernel module objectnext to existing ones:
# ... obj-m += lttng-probe-module.o obj-m += lttng-probe-power.o obj-m += lttng-probe-hello.o # ...
Time to build! Point to your custom Linux kernel source tree usingthe KERNELDIR variable:
make KERNELDIR=/path/to/custom/linux
Finally, install modules:
sudo make modules_install
Tracing
The Controlling tracing section explainshow to use thelttng tool to create and control tracing sessions.Although thelttng tool will load the appropriate known LTTng kernelmodules when needed (by launchingroot's session daemon), it won'tload your customlttng-probe-hello module by default. You need tomanually start an LTTng session daemon asroot and use the--extra-kmod-probes option to append your custom probe module to thedefault list:
sudo pkill -u root lttng-sessiond sudo lttng-sessiond --extra-kmod-probes=hello
The first command makes sure any existing instance is killed. Ifyou're not interested in using the default probes, or if you onlywant to use a few of them, you could use--kmod-probes instead,which specifies an absolute list:
sudo lttng-sessiond --kmod-probes=hello,ext4,net,block,signal,sched
Confirm the custom probe module is loaded:
lsmod | grep lttng_probe_hello
The hello_world event should appear in the list when doing
lttng list --kernel | grep hello
You may now create an LTTng tracing session, enable the hello_worldkernel event (and others if you wish) and start tracing:
sudo lttng create my-session sudo lttng enable-event --kernel hello_world sudo lttng start
Plug a few USB devices, then stop tracing and inspect the trace (ifBabeltraceis installed):
sudo lttng stop sudo lttng view
Here's a sample output:
[15:30:34.835895035] (+?.?????????) hostname hello_world: { cpu_id = 1 }, { my_int = 8, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
[15:30:42.262781421] (+7.426886386) hostname hello_world: { cpu_id = 1 }, { my_int = 9, char0 = 80, char1 = 97, product = "Patriot Memory" }
[15:30:48.175621778] (+5.912840357) hostname hello_world: { cpu_id = 1 }, { my_int = 10, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
Two USB flash drives were used for this test.
You may change your LTTng custom probe, rebuild it and reload it atany time when not tracing. Make sure you remove the old module(either by killing the root LTTng session daemon which loaded themodule in the first place, or by usingmodprobe --remove directly)before loading the updated one.
Instrumenting an out-of-tree Linux kernel module for LTTng
Instrumenting a custom Linux kernel module for LTTng follows the exactsame steps asadding instrumentation to the Linux kernel itself,the only difference being that your mainline tracepoint definitionheader doesn't reside in the mainline source tree, but in yourkernel module source tree.
The only reference to this mainline header is in the LTTng customprobe's source code (probes/lttng-probe-hello.c in our example), forbuild time verification:
/* ... */ /* Build time verification of mismatch between mainline TRACE_EVENT() * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments. */ #include <trace/events/hello.h> /* ... */
The preferred, flexible way to include your module's mainlinetracepoint definition header is to put it in a specific directoryrelative to your module's root, e.g.,tracepoints, and include itrelative to your module's root directory in the LTTng custom probe'ssource:
#include <tracepoints/hello.h>
You may then build LTTng-modules by adding your module's rootdirectory as an include path to the extra C flags:
make ccflags-y=-I/path/to/kernel/module KERNELDIR=/path/to/custom/linux
Using ccflags-y allows you to move your kernel module to anotherdirectory and rebuild the LTTng-modules project with no change tosource files.
LTTng logger ABI
The lttng-tracer Linux kernel module, installed by the LTTng-modulespackage, creates a special LTTng logger ABI file/proc/lttng-loggerwhen loaded. Writing text data to this file generates an LTTng kerneldomain event namedlttng_logger.
Unlike other kernel domain events, lttng_logger may be enabled byany user, not only root users or members of the tracing group.
To use the LTTng logger ABI, simply write a string to/proc/lttng-logger:
echo -n 'Hello, World!' > /proc/lttng-logger
The msg field of the lttng_logger event contains the recordedmessage.
Note:Messages are split in chunks of 1024 bytes.
The LTTng logger ABI is a quick and easy way to trace some events fromuser space through the kernel tracer. However, it is much more basicthan LTTng-UST: it's slower (involves system call round-trip to thekernel and only supports logging strings). The LTTng logger ABI isparticularly useful for recording logs as LTTng traces from shellscripts, potentially combining them with other Linux kernel/user spaceevents.
Advanced techniques
This section presents some advanced techniques related toLTTng instrumenting.
Instrumenting a 32-bit application on a 64-bit system
In order to trace a 32-bit application running on a 64-bit system,LTTng must use a dedicated 32-bitconsumer daemon. This section discusses how tobuild that daemon (which isnot part of the default 64-bit LTTngbuild) and the LTTng 32-bit tracing libraries, and how to instrumenta 32-bit application in that context.
Make sure you install all 32-bit versions of LTTng dependencies.Their names can be found in theREADME.md files of each LTTng packagesource. How to find and install them will vary depending on your targetLinux distribution.gcc-multilib is a common package name for themultilib version of GCC, which you will also need.
The following packages will be built for 32-bit support on a 64-bitsystem: Userspace RCU,LTTng-UST and LTTng-tools.
Building 32-bit Userspace RCU
Follow this:
git clone git://git.urcu.so/urcu.git cd urcu ./bootstrap ./configure --libdir=/usr/lib32 CFLAGS=-m32 make sudo make install sudo ldconfig
The -m32 C compiler flag creates 32-bit object files and --libdirindicates where to install the resulting libraries.
Building 32-bit LTTng-UST
Follow this:
git clone http://git.lttng.org/lttng-ust.git
cd lttng-ust
./bootstrap
./configure --prefix=/usr \
--libdir=/usr/lib32 \
CFLAGS=-m32 CXXFLAGS=-m32 \
LDFLAGS=-L/usr/lib32
make
sudo make install
sudo ldconfig
-L/usr/lib32 is required for the build to find the 32-bit versionsof Userspace RCU and other dependencies.
Note:Depending on your Linux distribution, 32-bit libraries could be installed at a different location than/usr/lib32. For example, Debian is known to install some 32-bit libraries in/usr/lib/i386-linux-gnu.
In this case, make sure to set LDFLAGS to all the relevant 32-bit library paths, e.g.,LDFLAGS="-L/usr/lib32 -L/usr/lib/i386-linux-gnu".
Note:You may add options to ./configure if you need them, e.g., for Java and SystemTap support. Look at./configure --help for more information.
Building 32-bit LTTng-tools
Since the host is a 64-bit system, most 32-bit binaries and libraries ofLTTng-tools are not needed; the host will use their 64-bit counterparts.The required step here is building and installing a 32-bit consumerdaemon.
Follow this:
git clone http://git.lttng.org/lttng-tools.git
cd lttng-ust
./bootstrap
./configure --prefix=/usr \
--libdir=/usr/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
LDFLAGS=-L/usr/lib32
make
cd src/bin/lttng-consumerd
sudo make install
sudo ldconfig
The above commands build all the LTTng-tools project as 32-bitapplications, but only installs the 32-bit consumer daemon.
Building 64-bit LTTng-tools
Finally, you need to build a 64-bit version of LTTng-tools which isaware of the 32-bit consumer daemon previously built and installed:
make clean
./bootstrap
./configure --prefix=/usr \
--with-consumerd32-libdir=/usr/lib32 \
--with-consumerd32-bin=/usr/lib32/lttng/libexec/lttng-consumerd
make
sudo make install
sudo ldconfig
Henceforth, the 64-bit session daemon will automatically find the32-bit consumer daemon if required.
Building an instrumented 32-bit C application
Let us reuse the Hello world example ofTracing your own user application(Getting started chapter).
The instrumentation process is unaltered.
First, a typical 64-bit build (assuming you're running a 64-bit system):
gcc -o hello64 -I. hello.c hello-tp.c -ldl -llttng-ust
Now, a 32-bit build:
gcc -o hello32 -I. -m32 hello.c hello-tp.c -L/usr/lib32 \
-ldl -llttng-ust -Wl,-rpath,/usr/lib32
The -rpath option, passed to the linker, will make the dynamic loadercheck for libraries in/usr/lib32 before looking in its default paths,where it should find the 32-bit version ofliblttng-ust.
Running 32-bit and 64-bit versions of an instrumented C application
Now, both 32-bit and 64-bit versions of the Hello world example abovecan be traced in the same tracing session. Use thelttng tool as usualto create a tracing session and start tracing:
lttng create session-3264 lttng enable-event -u -a ./hello32 ./hello64 lttng stop
Use lttng view to verify both processes weresuccessfully traced.
Controlling tracing
Once you're in possession of a software that is properlyinstrumented for LTTng tracing, be it thanks tothe built-in LTTng probes for the Linux kernel, a custom userapplication or a custom Linux kernel, all that is left is actuallytracing it. As a user, you control LTTng tracing using a single commandline interface: thelttng tool. This tool usesliblttng-ctl behindthe scene to connect to and communicate with session daemons. LTTngsession daemons may either be started manually (lttng-sessiond) orautomatically by thelttng command when needed. Trace data maybe forwarded to the network and used elsewhere using an LTTng relaydaemon (lttng-relayd).
The manpages of lttng, lttng-sessiond and lttng-relayd are prettycomplete, thus this section is not an online copy of the latter (weleave this contents for theOnline LTTng manpages section).This section is rather a tour of LTTngfeatures through practical examples and tips.
If not already done, make sure you understand the core conceptsand how LTTng components connect together by reading theUnderstanding LTTng chapter; this sectionassumes you are familiar with them.
Creating and destroying tracing sessions
Whatever you want to do with lttng, it has to happen inside atracing session, created beforehand. A session, in general, is aper-user container of state. A tracing session is no different; itkeeps a specific state of stuff like:
- session name
- enabled/disabled channels with associated parameters
- enabled/disabled events with associated log levels and filters
- context information added to channels
- tracing activity (started or stopped)
and more.
A single user may have many active tracing sessions. LTTng sessiondaemons are the ultimate owners and managers of tracing sessions. Foruser space tracing, each user has its own session daemon. Since Linuxkernel tracing requires root privileges, onlyroot's session daemonmay enable and trace kernel events. However, lttng has a--groupoption (which is passed to lttng-sessiond when starting it) tospecify the name of atracing group which selected users may be partof to be allowed to communicate withroot's session daemon. Bydefault, the tracing group name is tracing.
To create a tracing session, do:
lttng create my-session
This will create a new tracing session named my-session and make itthe current one. If you don't specify any name (calling onlylttng create), your tracing session will be namedauto. Tracesare written in~/lttng-traces/session- followedby the tracing session's creation date/time by default, wheresession is the tracing session name. To save themat a different location, use the--output option:
lttng create --output /tmp/some-directory my-session
You may create as many tracing sessions as you wish:
lttng create other-session lttng create yet-another-session
You may view all existing tracing sessions using the list command:
lttng list
The state of a current tracing session is kept in ~/.lttngrc. Eachinvocation oflttng reads this file to set its current tracingsession name so that you don't have to specify a session name for eachcommand. You could edit this file manually, but the preferred way toset the current tracing session is to use theset-session command:
lttng set-session other-session
Most lttng commands accept a --session option to specify the nameof the target tracing session.
Any existing tracing session may be destroyed using the destroycommand:
lttng destroy my-session
Providing no argument to lttng destroy will destroy the currenttracing session. Destroying a tracing session will stop any tracingrunning within the latter. Destroying a tracing session frees resourcesacquired by the session daemon and tracer side, making sure to flushall trace data.
You can't do much with LTTng using only the create, set-sessionanddestroy commands oflttng, but it is essential to know them inorder to control LTTng tracing, which always happen within the scope ofa tracing session.
Enabling and disabling events
Inside a tracing session, individual events may be enabled or disabledso that tracing them may or may not generate trace data.
We sometimes use the term event metonymically throughout this text torefer to a specific condition, orrule, that could lead, whensatisfied, to an actual occurring event (a point at a specific positionin source code/binary program, logical processor and time capturingsome payload) being recorded as trace data. This specific condition iscomposed of:
- A domain (kernel, user space,
java.util.logging, or log4j)(required). - One or many instrumentation points in source code or binaryprogram (tracepoint name, address, symbol name, function name,logger name, etc.) to be executed (required).
- A log level (each instrumentation point declares its own loglevel) or log level range to match (optional; only valid for userspace domain).
- A custom user expression, or filter, that must evaluate totrue when a tracepoint is executed (optional; only valid for userspace domain).
All conditions are specified using arguments passed to theenable-event command of thelttng tool.
Condition 1 is specified using either --kernel/-k (kernel),--userspace/-u (user space),--jul/-j(JUL), or--log4j/-l (log4j).Exactly one of those four arguments must be specified.
Condition 2 is specified using one of:
--tracepoint: tracepoint--probe: dynamic probe (address, symbol name or combinationof both in binary program; only valid for kernel domain)--function: function entry/exit (address, symbol name orcombination of both in binary program; only valid for kernel domain)--syscall: system call entry/exit (only valid for kerneldomain)
When none of the above is specified, enable-event defaults tousing--tracepoint.
Condition 3 is specified using one of:
--loglevel: log level range from 0 to a specific log level--loglevel-only: specific log level
See lttng enable-event --help for the complete list of log levelnames.
Condition 4 is specified using the --filter option. This filter isa C-like expression, potentially reading real-time values of eventfields, that has to evaluate totrue for the condition to be satisfied.Event fields are read using plain identifiers while context fieldsmust be prefixed with$ctx.. See lttng enable-event --help forall usage details.
The aforementioned arguments are combined to create and enable events.Each unique combination of arguments leads to a differentenabled event. The log level and filter arguments are optional, theirdefault values being respectively all log levels and a filter whichalways returns true.
Here are a few examples (you mustcreate a tracing sessionfirst):
lttng enable-event -u --tracepoint my_app:hello_world
lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_WARNING
lttng enable-event -u --tracepoint 'my_other_app:*'
lttng enable-event -u --tracepoint my_app:foo_bar \
--filter 'some_field <= 23 && !other_field'
lttng enable-event -k --tracepoint sched_switch
lttng enable-event -k --tracepoint gpio_value
lttng enable-event -k --function usb_probe_device usb_probe_device
lttng enable-event -k --syscall --all
The wildcard symbol, *, matches anything and may only be used atthe end of the string when specifying atracepoint. Make sure touse it between single quotes in your favorite shell to avoidundesired shell expansion.
System call events can be enabled individually, too:
lttng enable-event -k --syscall open lttng enable-event -k --syscall read lttng enable-event -k --syscall fork,chdir,pipe
The complete list of available system call events can beobtained using
lttng list --kernel --syscall
You can see a list of events (enabled or disabled) using
lttng list some-session
where some-session is the name of the desired tracing session.
What you're actually doing when enabling events with specific conditionsis creating awhitelist of traceable events for a given channel.Thus, the following case presents redundancy:
lttng enable-event -u --tracepoint my_app:hello_you lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_DEBUG
The second command, matching a log level range, is useless since the firstcommand enables all tracepoints matching the same name,my_app:hello_you.
Disabling an event is simpler: you only need to provide the eventname to the disable-event command:
lttng disable-event --userspace my_app:hello_you
This name has to match a name previously given to enable-event (ithas to be listed in the output oflttng list some-session).The* wildcard is supported, as long as you also used it in apreviousenable-event invocation.
Disabling an event does not add it to some blacklist: it simply removesit from its channel's whitelist. This is why you cannot disable an eventwhich wasn't previously enabled.
A disabled event will not generate any trace data, even if all itsspecified conditions are met.
Events may be enabled and disabled at will, either when LTTng tracersare active or not. Events may be enabled before a user space applicationis even started.
Basic tracing session control
Once you havecreated a tracing sessionandenabled one or more events,you may activate the LTTng tracers for the current tracing session atany time:
lttng start
Subsequently, you may stop the tracers:
lttng stop
LTTng is very flexible: user space applications may be launched beforeor after the tracers are started. Events will only be recorded if theyare properly enabled and if they occur while tracers are started.
A tracing session name may be passed to both the start and stopcommands to start/stop tracing a session other than the current one.
Enabling and disabling channels
As mentioned in theUnderstanding LTTng chapter, enabledevents are contained in a specific channel, itself contained in aspecific tracing session. A channel is a group of events withtunable parameters (event loss mode, sub-buffer size, number ofsub-buffers, trace file sizes and count, etc.). A given channel mayonly be responsible for enabled events belonging to one domain: eitherkernel or user space.
If you only used the create, enable-event and start/stopcommands of the lttng tool so far, one or two channels wereautomatically created for you (one for the kernel domain and/or onefor the user space domain). The default channels are both namedchannel0; channels from different domains may have the same name.
The current channels of a given tracing session can be viewed with
lttng list some-session
where some-session is the name of the desired tracing session.
To create and enable a channel, use the enable-channel command:
lttng enable-channel --kernel my-channel
This will create a kernel domain channel named my-channel withdefault parameters in the current tracing session.
Note:Because of a current limitation, all channels must becreated prior to beginning tracing in a given tracing session, i.e. before the first time you dolttng start.
Since a channel is automatically created by enable-event only for the specified domain, you cannot, for example, enable a kernel domain event, start tracing and then enable a user space domain event because no user space channel exists yet and it's too late to create one.
For this reason, make sure to configure your channels properly before starting the tracers for the first time!
Here's another example:
lttng enable-channel --userspace --session other-session --overwrite \
--tracefile-size 1048576 1mib-channel
This will create a user space domain channel named 1mib-channel inthe tracing session namedother-session that loses new events byoverwriting previously recorded events (instead of the default mode ofdiscarding newer ones) and saves trace files with a maximum size of1 MiB each.
Note that channels may also be created using the --channel option oftheenable-event command when the provided channel name doesn't existfor the specified domain:
lttng enable-event --kernel --channel some-channel sched_switch
If no kernel domain channel named some-channel existed before callingthe above command, it would be created with default parameters.
You may enable the same event in two different channels:
lttng enable-event --userspace --channel my-channel app:tp lttng enable-event --userspace --channel other-channel app:tp
If both channels are enabled, the occurring app:tp event willgenerate two recorded events, one for each channel.
Disabling a channel is done with the disable-event command:
lttng disable-event --kernel some-channel
The state of a channel precedes the individual states of events withinit: events belonging to a disabled channel, even if they areenabled, won't be recorded.
Fine-tuning channels
There are various parameters that may be fine-tuned with theenable-channel command. The latter are well documented inthe manpage oflttngand in the Channel section of theUnderstanding LTTng chapter. For basictracing needs, their default values should be just fine, but here are afew examples to break the ice.
As the frequency of recorded events increases—either because theevent throughput is actually higher or because you enabled more eventsthan usual—event loss might be experienced. Since LTTng neverwaits, by design, for sub-buffer space availability (non-blockingtracer), when a sub-buffer is full and no empty sub-buffers are left,there are two possible outcomes: either the new events that do not fitare rejected, or they start replacing the oldest recorded events.The choice of which algorithm to use is a per-channel parameter, thedefault being discarding the newest events until there is some spaceleft. If your situation always needs the latest events at the expenseof writing over the oldest ones, create a channel with the--overwriteoption:
lttng enable-channel --kernel --overwrite my-channel
When an event is lost, it means no space was available in anysub-buffer to accommodate it. Thus, if you want to cope with sporadichigh event throughput situations and avoid losing events, you need toallocate more room for storing them in memory. This can be done byeither increasing the size of sub-buffers or by adding sub-buffers.The following example creates a user space domain channel with16 sub-buffers of 512 kiB each:
lttng enable-channel --userspace --num-subbuf 16 --subbuf-size 512k big-channel
Both values need to be powers of two, otherwise they are rounded upto the next one.
Two other interesting available parameters of enable-channel are--tracefile-size and--tracefile-count, which respectively limitthe size of each trace file and the their count for a given channel.When the number of written trace files reaches its limit for a givenchannel-CPU pair, the next trace file will overwrite the very firstone. The following example creates a kernel domain channel with amaximum of three trace files of 1 MiB each:
lttng enable-channel --kernel --tracefile-size 1M --tracefile-count 3 my-channel
An efficient way to make sure lots of events are generated is enablingall kernel events in this channel and starting the tracer:
lttng enable-event --kernel --all --channel my-channel lttng start
After a few seconds, look at trace files in your tracing sessionoutput directory. For two CPUs, it should look like:
my-channel_0_0 my-channel_1_0 my-channel_0_1 my-channel_1_1 my-channel_0_2 my-channel_1_2
Amongst the files above, you might see one in each group with a sizelower than 1 MiB: they are the files currently being written.
Since all those small files are valid LTTng trace files, LTTng traceviewers may read them. It is the viewer's responsibility to properlymerge the streams so as to present an ordered list to the user.Babeltracemerges LTTng trace files correctly and is fast at doing it.
Adding some context to channels
If you read all the sections ofControlling tracing so far, you should beable to create tracing sessions, create and enable channels and eventswithin them and start/stop the LTTng tracers. Event fields recorded intrace files provide important information about occurring events, butsometimes external context may help you solve a problem faster. Thissection discusses how to add context information to events of aspecific channel using the lttng tool.
There are various available context values which can accompany eventsrecorded by LTTng, for example:
- process information:
- identifier (PID)
- name
- priority
- scheduling priority (niceness)
- thread identifier (TID)
- the hostname of the system on which the event occurred
- plenty of performance counters using perf:
- CPU cycles, stalled cycles, idle cycles, etc.
- cache misses
- branch instructions, misses, loads, etc.
- CPU faults
- etc.
The full list is available in the output of lttng add-context --help.Some of them are reserved for a specific domain (kernel oruser space) while others are available for both.
To add context information to one or all channels of a given tracingsession, use theadd-context command:
lttng add-context --userspace --type vpid --type perf:thread:cpu-cycles
The above example adds the virtual process identifier and per-threadCPU cycles count values to all recorded user space domain events of thecurrent tracing session. Use the--channel option to select a specificchannel:
lttng add-context --kernel --channel my-channel --type tid
adds the thread identifier value to all recorded kernel domain eventsin the channelmy-channel of the current tracing session.
Beware that context information cannot be removed from channels onceit's added for a given tracing session.
Saving and loading tracing session configurations
Configuring a tracing session may be long: creating and enablingchannels with specific parameters, enabling kernel and user spacedomain events with specific log levels and filters, adding contextto some channels, etc. If you're going to use LTTng to solve realworld problems, chances are you're going to have to record events usingthe same tracing session setup over and over, modifying a few variableseach time in your instrumented program or environment. To avoidconstant tracing session reconfiguration, thelttng tool is able tosave and load tracing session configurations to/from XML files.
To save a given tracing session configuration, do:
lttng save my-session
where my-session is the name of the tracing session to save. Tracingsession configurations are saved to~/.lttng/sessions by default;use the--output-path option to change this destination directory.
All configuration parameters are saved:
- tracing session name
- trace data output path
- channels with their state and all their parameters
- context information added to channels
- events with their state, log level and filter
- tracing activity (started or stopped)
To load a tracing session, simply do:
lttng load my-session
or, if you used a custom path:
lttng load --input-path /path/to/my-session.lttng
Your saved tracing session will be restored as if you just configuredit manually.
Sending trace data over the network
The possibility of sending trace data over the network comes as abuilt-in feature of LTTng-tools. For this to be possible, an LTTngrelay daemon must be executed and listening on the machine wheretrace data is to be received, and the user must create a tracingsession using appropriate options to forward trace data to the remoterelay daemon.
The relay daemon listens on two different TCP ports: one for controlinformation and the other for actual trace data.
Starting the relay daemon on the remote machine is as easy as:
lttng-relayd
This will make it listen to its default ports: 5342 for control and5343 for trace data. The--control-port and--data-port options maybe used to specify different ports.
Traces written by lttng-relayd are written to~/lttng-traces/hostname/session bydefault, wherehostname is the host name of thetraced (monitored) system andsession is thetracing session name. Use the --output option to write trace dataoutside~/lttng-traces.
On the sending side, a tracing session must be created using thelttng tool with the--set-url option to connect to the distantrelay daemon:
lttng create my-session --set-url net://distant-host
The URL format is described in the output of lttng create --help.The above example will use the default ports; the--ctrl-url and--data-url options may be used to set the control and data URLsindividually.
Once this basic setup is completed and the connection is established,you may use thelttng tool on the target machine as usual; everythingyou do will be transparently forwarded to the remote machine if needed.For example, a parameter changing the maximum size of trace files willhave an effect on the distant relay daemon actually writing the trace.
Viewing events as they arrive
We have seen how trace files may be produced by LTTng out of generatedapplication and Linux kernel events. We have seen that those trace filesmay be either recorded locally by consumer daemons or remotely usinga relay daemon. And we have seen that the maximum size and count oftrace files is configurable for each channel. With all those features,it's still not possible to read a trace file as it is being writtenbecause it could be incomplete and appear corrupted to the viewer.There is a way to view events as they arrive, however: usingLTTng live.
LTTng live is implemented, in LTTng, solely on the relay daemon side.As trace data is sent over the network to a relay daemon by a (possiblyremote) consumer daemon, atee may be created: trace data will berecorded to trace filesas well as being transmitted to aconnected live viewer:
In order to use this feature, a tracing session must created in livemode on the target system:
lttng create --live
An optional parameter may be passed to --live to set the intervalof time (in microseconds) between flushes to the network(1 second is the default):
lttng create --live 100000
will flush every 100 ms.
If no network output is specified to the create command, a localrelay daemon will be spawned. In this very common case, viewing a livetrace is easy: enable events and start tracing as usual, then uselttng view to start the default live viewer:
lttng view
The correct arguments will be passed to the live viewer so that itmay connect to the local relay daemon and start reading live events.
You may also wish to use a live viewer not running on the targetsystem. In this case, you should specify a network output when usingthecreate command (--set-url or--ctrl-url/--data-url options).A distant LTTng relay daemon should also be started to receive controland trace data. By default,lttng-relayd listens on 127.0.0.1:5344for an LTTng live connection. Otherwise, the desired URL may bespecified using its--live-port option.
Thebabeltraceviewer supports LTTng live as one of its input formats.babeltrace isthe default viewer when usinglttng view. To use it manually, firstlist active tracing sessions by doing the following (assuming the relaydaemon to connect to runs on the same host):
babeltrace --input-format lttng-live net://localhost
Then, choose a tracing session and start viewing events as they arriveusing LTTng live, e.g.:
babeltrace --input-format lttng-live net://localhost/host/hostname/my-session
Taking a snapshot
The normal behavior of LTTng is to record trace data as trace files.This is ideal for keeping a long history of events that occurred onthe target system and applications, but may be too much data in somesituations. For example, you may wish to trace your applicationcontinuously until some critical situation happens, in which case youwould only need the latest few recorded events to perform the desiredanalysis, not multi-gigabyte trace files.
LTTng has an interesting feature called snapshots. When creatinga tracing session in snapshot mode, no trace files are written; thetracers' sub-buffers are constantly overwriting the oldest recordedevents with the newest. At any time, either when the tracers are startedor stopped, you may take a snapshot of those sub-buffers.
There is no difference between the format of a normal trace file and theformat of a snapshot: viewers of LTTng traces will also support LTTngsnapshots. By default, snapshots are written to disk, but they may alsobe sent over the network.
To create a tracing session in snapshot mode, do:
lttng create --snapshot my-snapshot-session
Next, enable channels, events and add context to channels as usual.Once a tracing session is created in snapshot mode, channels will beforced to use theoverwrite mode(--overwrite option of the enable-channel command; also calledflight recorder mode) and have anmmap() channel type(--output mmap).
Start tracing. When you're ready to take a snapshot, do:
lttng snapshot record --name my-snapshot
This will record a snapshot named my-snapshot of all channels ofall domains of the current tracing session. By default, snapshots filesare recorded in the path returned bylttng snapshot list-output. Youmay change this path or decide to send snapshots over the networkusing either:
- an output path/URL specified when creating the tracing session(
lttng create) - an added snapshot output path/URL using
lttng snapshot add-output - an output path/URL provided directly to the
lttng snapshot recordcommand
Method 3 overrides method 2 which overrides method 1. When specifyinga URL, a relay daemon must be listening on some machine (seeSending trace data over the network).
If you need to make absolutely sure that the output file won't belarger than a certain limit, you can set a maximum snapshot size whentaking it with the--max-size option:
lttng snapshot record --name my-snapshot --max-size 2M
Older recorded events will be discarded in order to respect thismaximum size.
Machine interface
The lttng tool aims at providing a command output as human-readable aspossible. While this output is easy to parse by a human being, machineswill have a hard time.
This is why the lttng tool provides the general --mi option, whichmust specify a machine interface output format. As of the latestLTTng stable release, only thexml format is supported. A schemadefinition (XSD) is madeavailableto ease the integration with external tools as much as possible.
The --mi option can be used in conjunction with all lttng commands.Here are some examples:
lttng --mi xml create some-session lttng --mi xml list some-session lttng --mi xml list --kernel lttng --mi xml enable-event --kernel --syscall open lttng --mi xml start
Reference
This chapter presents various references for LTTng packages such as linksto online manpages, tables needed by the rest of the text, descriptionsof library functions, etc.
Online LTTng manpages
LTTng packages currently install the following manpages, availableonline using the links below:
- LTTng-tools
- LTTng-UST
LTTng-UST
This section presents references of the LTTng-UST package.
LTTng-UST library (liblttng‑ust)
The LTTng-UST library, or liblttng-ust, is the main shared objectagainst which user applications are linked to make LTTng user spacetracing possible.
The C application guide shows the completeprocess to instrument, build and run a C/C++ application usingLTTng-UST, while this section contains a few important tables.
Tracepoint fields macros (for TP_FIELDS())
The available macros to define tracepoint fields, which should be listedwithinTP_FIELDS() inTRACEPOINT_EVENT(), are:
| Macro | Description/arguments |
|---|---|
| Standard integer, displayed in base 10
|
ctf_integer_hex(t, n, e) | Standard integer, displayed in base 16
|
ctf_integer_network(t, n, e) | Integer in network byte order (big endian), displayed in base 10
|
ctf_integer_network_hex(t, n, e) | Integer in network byte order, displayed in base 16
|
| Floating point number
|
| Null-terminated string; undefined behavior if
|
| Statically-sized array of integers
|
| Statically-sized array, printed as text; no need to be null-terminated
|
| Dynamically-sized array of integers; type of
|
| Dynamically-sized array, displayed as text; no need to be null-terminated; undefined behavior if
|
The _nowrite versions omit themselves from the session trace, but areotherwise identical. This means the_nowrite fields won't be writtenin the recorded trace. Their primary purpose is to make someof the event context available to theevent filters without having tocommit the data to sub-buffers.
Tracepoint log levels (for TRACEPOINT_LOGLEVEL())
The following table shows the available log level values for theTRACEPOINT_LOGLEVEL() macro:
| Enum label | Enum value | Description |
|---|---|---|
TRACE_EMERG | 0 | System is unusable |
TRACE_ALERT | 1 | Action must be taken immediately |
TRACE_CRIT | 2 | Critical conditions |
TRACE_ERR | 3 | Error conditions |
TRACE_WARNING | 4 | Warning conditions |
TRACE_NOTICE | 5 | Normal, but significant, condition |
TRACE_INFO | 6 | Informational message |
TRACE_DEBUG_SYSTEM | 7 | Debug information with system-level scope (set of programs) |
TRACE_DEBUG_PROGRAM | 8 | Debug information with program-level scope (set of processes) |
TRACE_DEBUG_PROCESS | 9 | Debug information with process-level scope (set of modules) |
TRACE_DEBUG_MODULE | 10 | Debug information with module (executable/library) scope (set of units) |
TRACE_DEBUG_UNIT | 11 | Debug information with compilation unit scope (set of functions) |
TRACE_DEBUG_FUNCTION | 12 | Debug information with function-level scope |
TRACE_DEBUG_LINE | 13 | Debug information with line-level scope (TRACEPOINT_EVENT default) |
TRACE_DEBUG | 14 | Debug-level message |
Higher log level numbers imply the most verbosity (expect higher tracingthroughput). Log levels 0 through 6 and log level 14 matchsysloglevel semantics. Log levels 7 through 13 offer more fine-grainedselection of debug information.
LTTng-modules
This section presents references of the LTTng-modules package.
Tracepoint fields macros (for TP_STRUCT__entry())
This table describes possible entries for the TP_STRUCT__entry() partofLTTNG_TRACEPOINT_EVENT():
| Macro | Description/arguments |
|---|---|
| Standard integer, displayed in base 10
|
| Standard integer, displayed in base 16
|
| Standard integer, displayed in base 8
|
| Integer in network byte order (big endian), displayed in base 10
|
| Integer in network byte order (big endian), displayed in base 16
|
| Statically-sized array, elements displayed in base 10
|
| Statically-sized array, elements displayed in base 16
|
| Statically-sized array, displayed as text
|
| Dynamically-sized array, displayed in base 10
|
| Dynamically-sized array, displayed in base 16
|
| Dynamically-sized array, displayed as text
|
| Null-terminated string; undefined behavior if
|
The above macros should cover the majority of cases. For advanced items,see probes/lttng-events.h.
Tracepoint assignment macros (for TP_fast_assign())
This table describes possible entries for the TP_fast_assign() partofLTTNG_TRACEPOINT_EVENT():
| Macro | Description/arguments |
|---|---|
| Assignment of C expression
|
| Memory copy of
|
| Memory copy of
|
| Memory copy of dynamically-sized array from
|
| String copy of
|
扫一扫
1303
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
