[LTTng学习之旅]------开始之前

最近读了不少跟操作系统相关的资料。在deadline等多重压力下。终于要开始动手操作一下了。

考虑到Linux下各种性能分析工具 种类实在是太多。又多又杂。而且大多数是命令行形式，可能不好实现可视化的展示，没法跟老板汇报。暂时选择LTTng作为trace工具应用的切入点。

看了一下LTTng的网站做的还挺好看，虽然在知乎上有人对他颇有微词。但是我作为一个初心者还远远没有到指点江山的程度。

欸，不知道这个系列会不会写下去啊。很多博客写着写着就变成了枯燥的文档翻译。很没意思。

Welcome to the LTTng Documentation!

The Linux Trace Toolkit: next generation is an open source software toolkit which you can use to trace the Linux kernel, user applications, and user libraries at the same time.

OK 可以 trace 内核 用户程序 库。很牛。

LTTng consists of:

• Kernel modules to trace the Linux kernel.

• Shared libraries to trace C/C++ user applications.

• Java packages to trace Java applications which use java.util.logging or Apache log4j 1.2.

• A Python package to trace Python applications which use the standard logging package.

• A kernel module to trace shell scripts and other user applications without a dedicated instrumentation mechanism.

• Daemons and a command-line tool, lttng, to control the LTTng tracers。

还可以干Java（虽然我不会现在还不会Java，但是有几本书）和 Python（这个我会，但是写大点的程序还要查查手册）。

背景知识

What is tracing?

As the history of software engineering progressed and led to what we now take for granted—complex, numerous and interdependent software applications running in parallel on sophisticated operating systems like Linux—the authors of such components, software developers, began feeling a natural urge to have tools that would ensure the robustness and good performance of their masterpieces.

One major achievement in this field is, inarguably, the GNU debugger (GDB), an essential tool for developers to find and fix bugs. But even the best debugger won’t help make your software run faster, and nowadays, faster software means either more work done by the same hardware, or cheaper hardware for the same work.

A profiler is often the tool of choice to identify performance bottlenecks. Profiling is suitable to identify where performance is lost in a given piece of software. The profiler outputs a profile, a statistical summary of observed events, which you may use to discover which functions took the most time to execute. However, a profiler won’t report why some identified functions are the bottleneck. Bottlenecks might only occur when specific conditions are met, conditions that are sometimes impossible to capture by a statistical profiler, or impossible to reproduce with an application altered by the overhead of an event-based profiler. For a thorough investigation of software performance issues, a history of execution is essential, with the recorded values of variables and context fields you choose, and with as little influence as possible on the instrumented application. This is where tracing comes in handy.

探查性能瓶颈的首选工具 就是剖析。剖析基于events。GDB也是基于events.

剖析可以告诉你瓶颈在哪，但是不会告诉为什么产生瓶颈。这个时候上Trace手段。

Tracing is a technique used to understand what goes on in a running software system. The piece of software used for tracing is called a tracer, which is conceptually similar to a tape recorder. When recording, specific instrumentation points placed in the software source code generate events that are saved on a giant tape: a trace file. You can record user application and operating system events at the same time, opening the possibility of resolving a wide range of problems that would otherwise be extremely challenging.

特殊的Trace代码插入程序中，进行记录。就像录音机一样都录下来。

Tracing is often compared to logging. However, tracers and loggers are two different tools, serving two different purposes. Tracers are designed to record much lower-level events that occur much more frequently than log messages, often in the range of thousands per second, with very little execution overhead. Logging is more appropriate for a very high-level analysis of less frequent events: user accesses, exceptional conditions (errors and warnings, for example), database transactions, instant messaging communications, and such. Simply put, logging is one of the many use cases that can be satisfied with tracing.

The list of recorded events inside a trace file can be read manually like a log file for the maximum level of detail, but it’s generally much more interesting to perform application-specific analyses to produce reduced statistics and graphs that are useful to resolve a given problem. Trace viewers and analyzers are specialized tools designed to do this.

In the end, this is what LTTng is: a powerful, open source set of tools to trace the Linux kernel and user applications at the same time. LTTng is composed of several components actively maintained and developed by its community.

LTTng的竞品

Excluding proprietary solutions, a few competing software tracers exist for Linux:

dtrace4linux

A port of Sun Microsystems' DTrace to Linux.

The tool interprets user scripts and is responsible for loading code into the Linux kernel for further execution and collecting the outputted data.dtrace

偶哟 这个是那个 Brendan大佬搞的吗？

eBPF

A subsystem in the Linux kernel in which a virtual machine can execute programs passed from the user space to the kernel.

You can attach such programs to tracepoints and kprobes thanks to a system call, and they can output data to the user space when executed thanks to different mechanisms (pipe, VM register values, and eBPF maps, to name a few).

目前最火的Trace技术。内核注入技术？理论上可以干任何事情。玩转LTTng之后就来玩这个。

ftrace

The de facto function tracer of the Linux kernel.

Its user interface is a set of special files in sysfs.

这个是函数基本的追踪吗？感觉说不定是LTTng的底层机制之一。

perf

A performance analysis tool for Linux which supports hardware performance counters, tracepoints, as well as other counters and types of probes.

The controlling utility of perf is the command line/text UI tool.perf

这个我用来画过火焰图。。

strace

A command-line utility which records system calls made by a user process, as well as signal deliveries and changes of process state.

strace makes use of ptrace to fulfill its function.

sysdig

Like SystemTap, uses scripts to analyze Linux kernel events.

You write scripts, or chisels in the jargon of sysdig, in Lua and sysdig executes them while it traces the system or afterwards. The interface of sysdig is the command-line tool as well as the text UI-based tool.sysdigcsysdig

SystemTap

A Linux kernel and user space tracer which uses custom user scripts to produce plain text traces.

SystemTap converts the scripts to the C language, and then compiles them as Linux kernel modules which are loaded to produce trace data. The primary user interface of SystemTap is the command-line tool.stap

以前的顶流吧。现在已经排在最后了？多亏了eBPF。大家都去玩内核虚拟机了。

The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, a file format optimized for the production and analyses of multi-gigabyte data.

LTTng is the result of more than 10 years of active open source development by a community of passionate developers. LTTng is currently available on major desktop and server Linux distributions.

The main interface for tracing control is a single command-line tool named . The latter can create several recording sessions, enable and disable recording event rules on the fly, filter events efficiently with custom user expressions, start and stop tracing, and much more. LTTng can write the traces on the file system or send them over the network, and keep them totally or partially. You can make LTTng execute user-defined actions when LTTng emits an event. You can view the traces once tracing becomes inactive or as LTTng records events.lttng

Install LTTng now and start tracing!