The Performance of Open Source Software Ninja

< 原文链接>

The Performance of Open Source Software Ninja - Evan Martin

• 开源软件Ninja的性能

• Software Design by Example in Python is now in beta.
  All the material is free to read and re-use under open licenses, and we would be very grateful for feedback and corrections.

• python中的软件设计测试示例还在测试。所有的资料在开源许可下都是可以免费阅读和重用的，我们会非常感激所有的反馈和修正。

• If you enjoy these books, you may also enjoy Software Design by Example in JavaScript, Research Software Engineering with Python, JavaScript for Data Science, Teaching Tech Together, and It Will Never Work in Theory.

• 如果你喜欢这些书，你也可能会喜欢JavaScript的软件设计示例，基于python的软件工程研究，用于数据科学的JavaScript，一起教技术，和理论上永不可行

• Ninja is a build system similar to Make. As input you describe the commands necessary to process source files into target files. Ninja uses these commands to bring targets up to date. Unlike many other build systems, Ninja’s main design goal was speed.

• Ninja是一个类似于Make的构建系统。作为输入，你描述了将源文件处理为目标文件的必要指令。 Ninja 使用这些指令来更新目标。不像其他的构建系统，Ninja的主要设计目标是速度。

• I wrote Ninja while working on Google Chrome. I started Ninja as an experiment to find out if Chrome’s build could be made faster. To successfully build Chrome, Ninja’s other main design goal followed: Ninja needed to be easily embedded within a larger build system.

• 我在谷歌浏览器工作的时候写的Ninja。我将开始Ninja作为一个实验去找出能否使浏览器的构建更快些。为了更成功的构建浏览器，Ninja的另一个设计目标如下：Ninja必须很容易嵌入到另一个大型构建系统。

• Ninja has been quietly successful, gradually replacing the other build systems used by Chrome. After Ninja was made public others contributed code to make the popular CMake build system generate Ninja files–now Ninja is also used to develop CMake-based projects like LLVM and ReactOS. Other projects, like TextMate, target Ninja directly from their custom build.

• Ninja已经悄悄地成功了，渐渐取代了浏览器上的其他构建系统。在Ninja开源之后，其他人也贡献了代码使得流行的CMake构建系统生成了Ninja文件-先在Ninja也被用于开发基于CMake的项目像LLVM和ReactOS。其他的项目，像TextMate，Ninja目标文件会直接从他们的定制构件中产生。

• I worked on Chrome from 2007 to 2012, and started Ninja in 2010. There are many factors contributing to the build performance of a project as large as Chrome (today around 40,000 files of C++ code generating an output binary around 90 MB in size). During my time I touched many of them, from distributing compilation across multiple machines to tricks in linking. Ninja primarily targets only one piece–the front of a build. This is the wait between starting the build and the time the first compile starts to run. To understand why that is important it is necessary to understand how we thought about performance in Chrome itself.

• 我从2007到2012年在谷歌浏览器工作，并且在2010年开始Ninja项目。像谷歌浏览器一样大的项目的构建性能受很多因素的影响(如今大约有40000个c++代码文件，生成了差不多90MB大小的二进制文件)。在这期间我接触了他们中的许多，从在多台机器中分布式编译到链接的技巧。Ninja的主要目标只有一个-构建的前面。这是在开始构建和第一次编译启动运行的时间之间的等待。为了理解这个为什么重要，有必要知道我们如何看待Chrome自身的性能。

---

A Small History of Chrome

• 关于chrome的一小段历史

• Discussion of all of Chrome’s goals is out of scope here, but one of the defining goals of the project was speed. Performance is a broad goal that spans all of computer science, and Chrome uses nearly every trick available, from caching to parallelization to just-in-time compilation. Then there was startup speed–how long it the program took to show up on the screen after clicking the icon–which seems a bit frivolous in comparison.

• chrome的所有目标的讨论超出了此处的范围，但这个项目定义的目标之一就是速度。性能是一项横跨所有计算机科学的广阔目标，并且chrome使用了几乎所有技巧去达到它，从缓存到并行化再到即时编译。然后是启动速度，程序从点击图标后到显示在屏幕上的时间，相比之下这似乎有些轻浮。

• Why care about startup speed? For a browser, a quick startup conveys a feeling of lightness, that doing something on the web is as trival an action as opening a text file. Further, the impact of latency on your happiness and on losing your train of thought is well-studied in human-computer interaction. Latency is especially a focus of web companies like Google or Amazon, who are in a good position to measure and experiment on the effect of latency–and who have done experiments that show that delays of even milliseconds have measurable effects on how frequently people use the site or make purchases. It’s a small frustration that adds up subconsciously.

• 为什么这么关注启动速度？对于一个浏览器，一个快速启动传达了一种轻量感，在网页上处理问题像是打开一个文本文件一样琐碎的行为。更长远来说，人机交互中很好的研究了延迟对于幸福感和失去思路的影响。延迟是像谷歌和亚马逊一样的互联网公司尤其关注的点，这是一种很好的定位，去衡量和实验延迟的影响，他们做了实验显示哪怕是只有毫秒级的延迟对于人们使用这个网站进行购买的频率都有可衡量的影响。这种小小的挫折感会在潜意识里进行积攒。

• Chrome’s approach to starting quickly was a clever trick by one of the first engineers on Chrome. As soon as they got their skeleton application to the point where it showed a window on the screen, they created a benchmark measuring that speed along with a continuous build that tracked it. Then, in Brett Wilson’s words, "a very simple rule: this test can never get any slower."1 As code was added to Chrome, maintenance of this benchmark demanded extra engineering effort2–in some cases work was delayed until it was truly needed, or data used during startup was precomputed–but the primary “trick” to performance, and the one that made the greatest impression on me, was simply to do less work.

• 谷歌的快速启动方法是谷歌最早的一批工程师想出的聪明办法。当他们从应用框架中得到显示在屏幕上的窗口上的一个点，他们创建了一个基准用来测量并跟踪持续构建的速度。然后，用Brett Wilson的话来说，“一个非常简单的规则，这个测试可以从不会变得更慢”。一个是因为加入到chrome中的代码，维护此基准需要额外的工程努力；二是在某些情况下，工作被推迟直至它真的被需要，或者在启动期间使用的数据已经被预编译。但性能的主要技巧和给我最大影响的只是少去做些工作。

• I joined the Chrome team without any intention of working on build tools. My background and platform of choice was Linux, and I wanted to be the Linux guy. To limit scope the project was initially Windows-only; I took it as my role to help finish the Windows implementation so that I could then make it run on Linux.

• 我加入谷歌团队没有任何工作于构建工具的意图。我的背景和平台选择是Linux，而且我像成为Linux的伙伴。为了限制范围，这个项目一开始进食基于Windows的；我接过来只是因为我的职责是帮助完成Windows的实现如此我可以让它运行在Linux上。

• When starting work on other platforms, the first hurdle was sorting out the build system. By that point Chrome was already large (complete, in fact–Chrome for Windows was released in 2008 before any ports had started), so efforts to switch even the Visual Studio-based Windows build to a different build system wholesale were conflicting with ongoing development. It felt like replacing the foundation of a building while it was in use.

• 当开始在其他平台上工作时，第一个障碍是挑选构建系统。那时chrome已经很大型了（事实上，Windows版本的chrome在开始任何移植之前已经于2008年发布），所以将基于visual studio的Windows构建转换到其他构建系统的努力与现有的开发产生了大规模的冲突。这感觉就像是要替换掉正在使用的建筑物的地基。

• Members of the Chrome team came up with an incremental solution called GYP3 which could be used to generate, one subcomponent at a time, the Visual Studio build files already used by Chrome in addition to the build files that would be used on other platforms.

• chrome团队的成员提出一种增量式的解决方案叫做GYP3，它可以被用作生成，除了已经被其他平台使用的构建文件外，Visual Studio的构建文件已经被chrome一次一个子部分使用。

• The input to GYP is simple: the desired name of the output accompanied by plain text lists of source files, the occasional custom rule like “process each IDL file to generate an additional source file”, and some conditional behaviors (e.g., only use certain files on certain platforms). GYP then takes this high-level description and generates platform-native build files.

• GYP的输入十分简单：需要的输出伴同源文件的纯文本列表，偶然的自定义规则像”处理每一个IDL文件并生成附加的源文件“，和一些条件行为（例如，只在确定的平台使用确定的文件）。GYP带着这个高级别描述并生成平台相关的构建文件。

• On the Mac “native build files” meant Xcode project files. On Linux, however, there was no obvious single choice. The initial attempt used the Scons build system, but I was dismayed to discover that a GYP-generated Scons build could take 30 seconds to start while Scons computed which files had changed. I figured that Chrome was roughly the size of the Linux kernel so the approach taken there ought to work. I rolled up my sleeves and wrote the code to make GYP generate plain Makefiles using tricks from the kernel’s Makefiles.

• 在Mac”原生构建文件“意味着xcode工程文件。然而在Linux，这里没有显而易见的单独选择。最初试图使用Scons构建系统，但是我沮丧的发现GYP生成的Scons构建文件需要30秒去启动，当Scons计算哪些文件已经被修改时。我认为Chrome的大小和Linux内核大约相似，所以相似的处理方法应当有效。我卷起我的袖子并写下了使用GYP生成文本Makefiles的代码，使用的是内核Makefiles的方法。

• Thus I unintentionally began my descent into build system madness. There are many factors that make building software take time, from slow linkers to poor parallelization, and I dug into all of them. The Makefile approach was initially quite fast but as we ported more of Chrome to Linux, increasing the number of files used in the build, it grew slower.

• 因此我无意间开始陷入到构建系统的痴狂当中。构建系统耗时有很多的因素，从缓慢的链接到不良的并行化，我深入挖掘了他们中的所有原因。这个Makefile途径最初非常快，但当我们将chrome移植到Linux上时，增加了构建中使用文件的数目，它的生成变慢了。

• As I worked on the port I found one part of the build process especially frustrating: I would make a change to a single file, run make, realize I’d left out a semicolon, run make again, and each time the wait would be long enough that I would forget what I was working on. I thought back to how hard we fought against latency for end users. “How can this be taking so long,” I’d wonder, “there can’t be that much work to do.” As an experiment I started Ninja, to see how simple I could make it.

• 当我工作于移植时，我发现构建处理中尤其使人沮丧的一部分是：我会对单个文件进行改变，运行make，意识到我忽略了一个分号，再次运行make，每一次的等待时间长到我都忘了我在着手做什么。我回想到我们与终端客户的延迟抗争所做的努力。“这怎么会花这么长时间”，我想，“不可能有这么多工作需要做”。作为实验，我开始了Ninja，来看我能做到多简单。

---

The Design of Ninja

• Ninja的设计

• At a high level any build system performs three main tasks. It will (1) load and analyze build goals, (2) figure out which steps need to run in order to achieve those goals, and (3) execute those steps.

• 在高级别上任何构建系统执行三个主要的任务。1. 导入并分析构建目标；2. 计算出哪一步需要运行能够达成这些目标；3. 执行这些步骤

• To make startup in step (1) fast, Ninja needed to do a minimal amount of work while loading the build files. Build systems are typically used by humans, which means they provide a convenient, high-level syntax for expressing build goals. It also means that when it comes time to actually build the project the build system must process the instructions further: for example, at some point Visual Studio must concretely decide based on the build configuration where the output files must go, or which files must be compiled with a C++ or C compiler.

• 为了使第一步的启动更快，当导入构建文件时Ninja需要一个极小数量的工作。构建系统普遍被人使用，这意味着他们提供了一个便捷、高级别的语法用来表达构建目标。这同样意味着当实际构建项目中的构建系统使必须进一步的处理指令：例如，在某些方面，Visual Studio必须基于构建配置具体地决定输出文件需要放到的地方，或者哪些文件必须使用c++或c编译器进行编译。

• Because of this, GYP’s work in generating Visual Studio files was effectively limited to translating lists of source files into the Visual Studio syntax and leaving Visual Studio to do the bulk of the work. With Ninja I saw the opportunity to do as much work as possible in GYP. In a sense, when GYP generates Ninja build files, it does all of the above computation once. GYP then saves a snapshot of that intermediate state into a format that Ninja can quickly load for each subsequent build.

• 由于这个原因，GYP的工作在生成Visual Studio 文件时实际上受限于源文件列表转换为Visual Studio 语法和让Visual Studio 去完成主体工作。使用Ninja我看到了可以和GYP做一样多的工作的可能的机会。从某种意义上说，当GYP生成Ninja构建文件时，它一次性完成了所有的计算过程。这时GYP保存了作为中间状态的快照，变成一种Ninja可以在任一子部分构建并快速导入的形式。

• Ninja’s build file language is therefore simple to the point of being inconvenient for humans to write. There are no conditionals or rules based on file extensions. Instead, the format is just a list of which exact paths produce which exact outputs. These files can be loaded quickly, requiring almost no interpretation.

• 因此Ninja的构建文件语言非常简单，以至于不方便人们书写。这里没有基于文件扩展的条件或规则。相反的，它的形式仅仅是确切路径生成确切的输出的列表。这些文件可以很快的被导入，几乎不需要解释。

• This minimalist design counterintuitively leads to greater flexibility. Because Ninja lacks higher-level knowledge of common build concepts like an output directory or current configuration, Ninja is simple to plug into larger systems (e.g., CMake, as we later found) that have different opinions about how builds should be organized. For example, Ninja is agnostic as to whether build outputs (e.g., object files) are placed alongside the source files (considered poor hygiene by some) or in a separate build output directory (considered hard to understand by others). Long after releasing Ninja I finally thought of the right metaphor: whereas other build systems are compilers, Ninja is an assembler.

• 这个极简主义的设计违反直觉带来了更好的灵活性。因为Ninja缺乏普通构建概念的高级知识，像是输出目录或者当前配置，Ninja很容易插入到关于如何组织构建有不同观点的大型系统（例如cmake，正如我们后来发现的那样）。例如，Ninja不知道构建输出（目标文件）是同源文件放在一起（一些人认为不整洁）还是单独放到构建输出目录（另外一些人认为很难理解）。在发布Ninja后很久我才最终意识到合适的比喻：尽管其他构建系统是编译器，Ninja是汇编器。

---

What Ninja Does

• Ninja做些什么

• If Ninja pushes most of the work to the build file generator, what is there left to do? The above ideology is nice in principle but real world needs are always more complicated. Ninja grew (and lost) features over the course of its development. At every point, the important question was always “can we do less?” Here is a brief overview of how it works.

• 如果Ninja将所有的工作推进为构建文件生成器，那这里还需要做什么呢？它所有的思想体系在原则上都很赞但在现实世界的需求都总是更加复杂。Ninja在它的发展过程中增加（或失去）特征。在每一个节点，最重要的问题总是“我们可以做得更少吗”。这是它工作概览的一个信仰。

• A human needs to debug the files when the build rules are wrong, so .ninja build files are plain text, similar to Makefiles, and they support a few abstractions to make them more readable.

• 一个人当运行规则是错误时需要调试文件，所以.ninja文件都是纯文本文件，与Makefiles很相似，同时他们也支持一些抽象规则来使得他们更加具有可读性。

• The first abstraction is the “rule”, which represents a single tool’s command-line invocation. A rule is then shared between different build steps. Here is an example of the Ninja syntax for declaring a rule named “compile” that runs the gcc compiler along with two build statements that make use of it for specific files.

• 第一个抽象是是“规则”，它代表了一个单个工具的命令行调用。然后再不同的构建步骤中共享规则。这里是Ninja语法的关于定义一个名号叫“编译”的规则的示例，运行gcc编译器以及两个用于特定文件的编译语句。

rule compile
  command = gcc -Wall -c $in -o $out
build out/foo.o: compile src/foo.c
build out/bar.o: compile src/bar.c

• The second abstraction is the variable. In the example above, these are the dollar-sign-prefixed identifiers ($in and $out). Variables can represent both the inputs and outputs of a command and can be used to make short names for long strings. Here is an extended compile definition that makes use of a variable for compiler flags:
• 第二个抽象是变量。在以上的示例中，他们都是美元标志的前置标识符（$in和$out）。变量可以同时代表一个命令的输入和输出而且可以用来给长字符取一个短名字。这里是一个扩展的编译定义使用了变量作为编译标志：

cflags = -Wall
rule compile
  command = gcc $cflags -c $in -o $out

• Variable values used in a rule can be shadowed in the scope of a single build block by indenting their new definition. Continuing the above example, the value of cflags can be adjusted for a single file:
• 在规则中使用可变的变量可以通过在一个单独的构建模块范围内中缩进他们的新定义实现隐藏。持续上述示例，cflags中的变量可以在单个文件中进行调整：

build out/file_with_extra_flags.o: compile src/baz.c
  cflags = -Wall -Wextra

• Rules behave almost like functions and variables behave like arguments. These two simple features are dangerously close to a programming language–the opposite of the “do no work” goal. But they have the important benefit of reducing repeated strings which is not only useful for humans but also for computers, reducing the quantity of text to be parsed.

• 规则的行为近似函数，变量的规则像是参数。这两个简单的特征都非常近似程序语言，与“不做任何工作”的目标相反。但是他们有对于减少于人类和计算机都很有用的重复字符串有着重要意义，可以减少要解析的文本数量。

• The build file, once parsed, describes a graph of dependencies: the final output binary depends on linking a number of objects, each of which is the result of compiling sources. Specifically it is a bipartite graph, where “nodes” (input files) point to “edges” (build commands) which point to nodes (output files). The build process then traverses this graph.

• 构建文件的一次解析，描述了依赖的图表：最终输出的二进制输出取决于连接的许多目标，这些目标都是编译源文件的结果。尤其它是一个二分表，（输入文件）节点指向边缘（构建指令），边缘则是指向（输出文件）节点。然后构建过程横跨整个图表。

• Given a target output to build, Ninja first walks up the graph to identify the state of each edge’s input files: that is, whether or not the input files exist and what their modification times are. Ninja then computes a plan. The plan is the set of edges that need to be executed in order to bring the final target up to date, according to the modification times of the intermediate files. Finally, the plan is executed, walking down the graph and checking off edges as they are executed and successfully completed.

• 给出需要构建的目标输出，Ninja首先遍历图表来识别每一个边缘的输入文件状态：即，输入文件是否存在和他们的修改时间是什么。然后Ninja就会计算一个计划。这个计划是需要执行以使得最终的目标更新的边缘的列表。最终，这个计划被执行，横跨整个图标并且确认所有的边缘都被执行并成功被完成。

• Once these pieces were in place I could establish a baseline benchmark for Chrome: the time to run Ninja again after successfully completing a build. That is the time to load the build files, examine the built state, and determine there was no work to do. The time it took for this benchmark to run was just under a second. This was my new startup benchmark metric. However, as Chrome grew, Ninja had to keep getting faster to keep that metric from regressing.

• 当这些部分都到位后我可以创建一个Chrome的极限基准：成功完成一个构建后再次运行Ninja的时间。这就是导入构建文件的时间，检查构建状态，并确定没有工作要做了。运行这个基准的时间仅需要一秒以下。这是我新的启动基准的标准。然而随着chrome的发展，Ninja不得不保持变得更快来防止这个指标倒退。

---

Optimizing Ninja

• 优化Ninja

• The initial implementation of Ninja was careful to arrange the data structures in order to allow a fast build, but wasn’t particularly clever in terms of optimizations. Once the program worked, I reasoned, a profiler could reveal which pieces mattered.

• Ninja的最初实现是很小心的去安排数据结构来使得构建快速，但是在优化方面并不特别灵巧。我推断，当程序工作时，解析器就能揭示哪部分最重要。

• Over the years, profiling pointed at different pieces of the program. Sometimes the worst offender was a single hot function that could be micro-optimized. At other times, it suggested something more broad like being careful not to allocate or copy memory except when necessary. There were also cases where a better representation or data structure had the most impact. What follows is a walk through the Ninja implementation and some of the more interesting stories about its performance.

• 数年以后，分析指向程序的不同部分。有时最坏的错误是一个单独的被频繁调用的程序可以进行微观优化。同时，它表明了一些更为长远的事情例如如非必要不要去分配或复制内存。在某些情况下更好的表示或者数据结构会产生最大的影响。接下来就是一些Ninja实现步骤和一些关于它的性能的更有趣的故事

---

Parsing

• 解析

• Initially Ninja used a hand-written lexer and a recursive descent parser. The syntax was simple enough, I thought. It turns out that for a large enough project like Chrome, simply parsing the build files (named with the extension .ninja) can take a surprising amount of time.

• 最初Ninja使用一个手写的词法分析器和一个递归下降解释器。我想这个的语法足够简单。它证明一个像chrome一样足够大的项目，简单地解析构建文件（使用名为.ninja的扩展）可以带来令人惊喜的时间。

• The original function to analyze a single character soon appeared in profiles:

• 分析单个字符的原始功能很快就显现了雏形

static bool IsIdentifierCharacter(char c) {
  return
    ('a' <= c && c <= 'z') ||
    ('A' <= c && c <= 'Z') ||
    // and so on...
}

• A simple fix–at the time saving 200 ms–was to replace the function with a 256-entry lookup table that could be indexed by the input character. Such a thing is trivial to generate using Python code like:
• 一个简单的固定时间保存200毫秒来替换一个256条目查找表的函数，可以被输入字符进行索引。使用python来生成这样的代码时很容易的

cs = set()
for c in string.ascii_letters + string.digits + r'+,-./\_$':
    cs.add(ord(c))
for i in range(256):
    print '%d,' % (i in cs),

• This trick kept Ninja fast for quite a while. Eventually we moved to something more principled: re2c, the lexer generator used by PHP. It can generate more complex lookup tables and trees of unintelligible code. For example:
• 这个方法使得Ninja在相当长的时间里保持很快。最终我们转向了更有规则地东西：re2c，使用php写成地词法生成器。它可以生成更复杂的难以理解的查找表和树。例如：

if (yych <= 'b') {
    if (yych == '`') goto yy24;
    if (yych <= 'a') goto yy21;
    // and so on...

• It remains an open question as to whether treating the input as text in the first place is a good idea. Perhaps we will eventually require Ninja’s input to be generated in some machine-friendly format that would let us avoid parsing for the most part.
• 首先将输入作为文本是否是一个好主意是它仍然保留的一个开发问题。也许我们最终需要Ninja的输入以一些机器友好的形式进行生成，这会使我们在很大部分上避免解析。

---

Canonicalization

• 规范化

• Ninja avoids using strings to identify paths. Instead, Ninja maps each path it encounters to a unique Node object and the Node object is used in the rest of the code. Reusing this object ensures that a given path is only ever checked on disk once, and the result of that check (i.e., the modification time) can be reused in other code.

• Ninja避免使用标识路径的字符串。相反的，Ninja将每个路径映射为接触到的独一无二的目标节点并且目标节点在其他的代码中也被使用。重复使用这个目标确保每一个给出的路径仅被磁盘确认一次，并且检查（例，修改时间）的结果可以被其他代码再次使用。

• The pointer to the Node object serves as a unique identity for that path. To test whether two Nodes refer to the same path it is sufficient to compare pointers rather than perform a more costly string comparison. For example, as Ninja walks up the graph of inputs to a build, it keeps a stack of dependendent Nodes to check for dependency loops: if A depends on B depends on C depends on A, the build can’t proceed. This stack, representing files, can be implemented as a simple array of pointers, and pointer equality can be used to check for duplicates.

• 目标节点的指针服务就像路径的一个独一无二的标识。对于测试两个节点是否指代相同的路径，比较指针比完成一个更加费时的字符串比较更加高效。例如，当Ninja遍历一个构建的输入图表时，它维护了一堆依赖节点来检查依赖循环：如果a依赖于b依赖于c依赖于a，这个构建不能进行。这个堆栈，代表文件，可以被实现为一个简单的指针数组，指针相等可以被用于检查重复项。

• To always use the same Node for a single file, Ninja must reliably map every possible name for a file into the same Node object. This requires a canonicalization pass on all paths mentioned in input files, which transforms a path like foo/…/bar.h into just bar.h. Initially Ninja simply required all paths to be provided in canonical form but that ends up not working for a few reasons. One is that user-specified paths (e.g., the command-line ninja ./bar.h) are reasonably expected to work correctly. Another is that variables may combine to make non-canonical paths. Finally, the dependency information emitted by gcc may be non-canonical.

• 对于单个文件总是使用相同的节点，Ninja必须可靠地将每一个文件可能的名字映射为相同的目标节点。这需要对所有输入文件中提到的路径进行规范化处理，就像将foo/../bar.h路径转换为bar.h。最初Ninja仅仅需要提供规范形式的路径但最后总会有些原因导致它不工作。一是用户定义的特殊路径(像命令行的ninja ./bar.h)相当希望可以正确工作。另一个是变量可能组合成非规范化路径。最后，gcc发出的依赖信息可能是非规范的。

• Thus most of what Ninja ends up doing is path processing, so canonicalizing paths is another hot point in profiles. The original implementation was written for clarity, not performance, so standard optimization techniques–like removing a double loop or avoiding memory allocation–helped considerably.

• 因此Ninja做的大部分工作都是路径处理，所以规范的路径是它的另一个特性。原始实现是为了明确性而不是性能，所以优化的标准像去除双重循环或者避免内存在很大程度上有所帮助。

---

The Build Log

• 构建日志

• Often micro-optimizations like the above are less impactful than structural optimizations where you change the algorithm or approach. This was the case with Ninja’s build log.

• 通常像以上这些微优化的影响比改变算法或方法的结构化优化要少。Ninja的构建日志就是这种情况。

• One part of the Linux kernel build system tracks the commands used to generate outputs. Consider a motivating example: you compile an input foo.c into an output foo.o, and then change the build file such that it should be rebuilt with different compilation flags. For the build system to know that it needs to rebuild the output, it must either note that foo.o depends on the build files themselves (which, depending on the organization of the project, might mean that a change to the build files would cause the entire project to rebuild), or record the commands used to generate each output and compare them for each build.

• Linux内核构建系统的一部分追踪命令用来生成输出。考虑一个激励性例子：你编译了一个输入文件foo.c得到一个foo.o的输出，然后修改构建文件这样它可以使用不同的编译标志进行重建。对于编译系统需要知道它需要重新构建输出，它要么记录foo.o依赖于构建文件本身（依赖于项目的组织可能意味着构建文件的改变会导致整个项目需要重新构建），要么记录用来生成每一个输出的命令并且在每次编译时进行比较。

• The kernel (and consequently the Chrome Makefiles and Ninja) takes the latter approach. While building, Ninja writes out a build log that records the full commands used to generate each output. Then for each subsequent build, Ninja loads the previous build log and compares the new build’s commands to the build log’s commands to detect changes. This, like loading build files or path canonicalization, was another hot point in profiles.

• 内核（因此包括chrome的makefile和ninja）使用的是后者的方法。当构建的时候，Ninja书写记录所有命令的构建日志用来生成每一个输出。然后对于每一个子构建，Ninja导入之前的构建日志并且比较构建日志命令的新的构建命令行来检测改变。像这样导入构建文件或者路径规范，是它配置中的另一个热点。

• After making a few smaller optimizations Nico Weber, a prolific contributor to Ninja, implemented a new format for the build log. Rather than recording commands, which are frequently very long and take a lot of time to parse, Ninja instead records a hash of the command. In subsequent builds, Ninja compares the hash of the command that is about to be run to the logged hash. If the two hashes differ, the output is out of date. This approach was very successful. Using hashes reduced the size of the build log dramatically–from 200 MB to less than 2 MB on Mac OS X–and made it over 20 times faster to load.

• 在Nico Weber（Ninja的一个高产贡献者）进行了一些小的优化后，实现了构建日志的一种新的形式。相较于记录命令（这时常来说会非常长并且需要很多时间去解析），Ninja使用哈希表来记录命令。在子构建中，Ninja比较哈希表中的命令（关于将要与逆行的哈希日志）。如果两个哈希表不同，输出就是过时的。这个方法非常成功。在Mac OS X上使用哈希表戏剧性的将构建日志的大小从200MB减少到小于2MB，并且使他超过20次导入得更快。

---

Dependency Files

• 依赖文件

• There is an additional store of metadata that must be recorded and used across builds. To correctly build C/C++ code a build system must accomodate dependencies between header files. Suppose foo.c contains the line #include “bar.h” and bar.h itself includes the line #include “baz.h”. All three of those files (foo.c, bar.h, baz.h) then affect the result of compilation. For example, changes to baz.h should still trigger a rebuild of foo.o.

• 这里有一个额外的元数据存储，需要在编译过程中被记录和使用。为了正确地构建c/c++代码，构建系统需要要在头文件之间容纳依赖项。假设foo.c包含#include "bar.h"同时bar.h本身包含 #include "baz.h"。然后这三个文件(foo.c, bar.h, baz.h)会影响编译的结果。例如，改变baz.h依旧会触发foo.o的重新构建。

• Some build systems use a “header scanner” to extract these dependencies at build time, but this approach can be slow and is difficult to make exactly correct in the presence of #ifdef directives. Another alternative is to require the build files to correctly report all dependencies, including headers, but this is cumbersome for developers: every time you add or remove an #include statement you need to modify or regenerate the build.

• 一些构建系统使用一个“头文件扫描器”在构建时间提取这些依赖，但是这种方法很慢并且在很难使得在#ifdef指令中完全正确。另一个选择是需要构建文件正确的报告所有依赖，包括头文件，但这对于开发者来说是繁琐的：每一次你添加或者移除一个#include声明，你需要修改或者更新构建。

• A better approach relies on the fact that at compile time gcc (and Microsoft’s Visual Studio) can output which headers were used to build the output. This information, much like the command used to generate an output, can be recorded and reloaded by the build system so that the dependencies can be tracked exactly. For a first-time build, before there is any output, all files will be compiled so no header dependency is necessary. After the first compilation, modifications to any files used by an output (including modifications that add or remove additional dependencies) will cause a rebuild, keeping the dependency information up-to-date.

• 一个更好的方式依赖于实际编译时gcc（和Microsoft’s Visual Studio）可以输出哪个头文件被用于构建输出。这个信息，非常类似于使用命令去生成输出，可以被构建系统记录和重载，如此依赖可以被准确跟踪。对于第一次构建，在有任何输出之前，所有的文件会被编译，所以没有头文件依赖需要。在第一次编译后，被输出（包括添加或者删除额外依赖的修改）使用的任意文件的修改会导致重新构建，保持依赖信息的更新。

• When compiling, gcc writes header dependencies in the format of a Makefile. Ninja then includes a parser for the (simplified subset) Makefile syntax and loads all of this dependency information at the next build. Loading this data is a major bottleneck. On a recent Chrome build, the dependency information produced by gcc sums to 90 MB of Makefiles, all of which reference paths which must be canonicalized before use.

• 当编译的时候，gcc使用Makefile的形式书写头文件依赖。然后Ninja包含Makefile语法的解析（简化子集）并且在下一次构建时加载所有依赖信息。加载这些数据是一个主要的障碍。在最近的一次Chrome的构建中，gcc生成的依赖文件信息总计为90MB的Makefiles，所有的引用路径在使用前需要被规范化。

• Much like with the other parsing work, using re2c and avoiding copies where possible helped with performance. However, much like how work was shifted to GYP, this parsing work can be pushed to a time other than the critical path of startup. Our most recent work on Ninja (at the time of this writing, the feature is complete but not yet released) has been to make this processing happen eagerly during the build.

• 就像其他解析工作一样，使用re2c并且避免复制有可能对性能有帮助。然而，就像工作怎么转移到GYP一样，这个解析工作可以推到启动的关键路径之外的一定时间。我们最近在Ninja上的工作已经使得这种处理在构建期间急切地出现了。

• Once Ninja has started executing build commands, all of the performance-critical work has been completed and Ninja is mostly idle as it waits for the commands it executes to complete. In this new approach for header dependencies, Ninja uses this time to process the Makefiles emitted by gcc as they are written, canonicalizing paths and processing the dependencies into a quickly deserializable binary format. On the next build Ninja only needs to load this file. The impact is dramatic, particularly on Windows. (This is discussed further later in this chapter.)

• 一次Ninja已经开始执行构建命令，所有的性能关键工作已经编译完成并且Ninja基本都是空闲的因为它在等待它执行完成的命令。在这种新的头文件依赖，Ninja使用这种时间去处理被gcc激发的Makefiles因为他们正在被写入，规范化的路径和将依赖处理为一种快速的序列化二进制格式。在下一次构建中，Ninja仅仅需要加载这些文件。这种影响是戏剧性的，尤其是在Windows上。（这些会在这章的后面讨论）

• The “dependency log” needs to store thousands of paths and dependencies between those paths. Loading this log and adding to it needs to be fast. Appending to this log should be safe, even in the event of an interruption such as a cancelled build.

• “依赖日志”需要储存成千上万的路径和在这些路径之间的依赖。加载这些日志并且添加到它需要很快速。在日志中进行追加需要很安全，甚至在一些中断的场合例如一个取消了的构建。

• After considering many database-like approaches I finally came up with a trivial implementation: the file is a sequence of records and each record is either a path or a list of dependencies. Each path written to the file is assigned a sequential integer identifier. Dependencies are then lists of integers. To add dependencies to the file, Ninja first writes new records for each path that doesn’t yet have an identifier and then writes the dependency record using those identifiers. When loading the file on a subsequent run Ninja can then use a simple array to map identifiers to their Node pointers.

• 在考虑了很多类似数据库的方法之后，我最终想出的一个简单实现的方法：文件是一系列的记录并且每一个记录不是一个路径或者一个依赖的数组。写入文件的每一个路径都被指定了一个顺序整数标识符。然后依赖就是整数的数组。在文件中添加依赖，Ninja首先将没有任何标识符的路径写上新的记录，然后使用这些标识符写下依赖记录。当在一个序列中导入文件，运行Ninja可以使用一个简单的数组来将标识符映射到他们的节点指针。

---

Executing a Build

• 执行构建

• Performance-wise the process of executing the commands judged necessary according to the dependencies discussed above is relatively uninteresting because the bulk of the work that needs to be done is performed in those commands (i.e., in the compilers, linkers, etc.), not in Ninja itself.

• 执行命令处理的效率判断是必要的，根据以上讨论到的依赖是相对无关的，因为工作的主体需要在这些命令中执行完成（例如，在编译器、连接器等等），而不是在Ninja它自身中完成。

• Ninja runs build commands in parallel by default, based on the number of available CPUs on the system. Since commands running simultaneously could have their outputs interleave, Ninja buffers all output from a command until that command completes before printing its output. The resulting output appears as if the commands were run serially.

• Ninja基于系统可用cpu的数量，默认并行运行构建命令。因为命令同时运行会使他们的输出交织，Ninja缓存一个命令中所有的输出直到这个命令在打印它的输出前完成。最终结果输出方法就好像命令连续地运行。

• This control over command output allows Ninja to carefully control its total output. During the build Ninja displays a single line of status while running; if the build completes successfully the total printed output of Ninja is a single line. This doesn’t make Ninja run any quicker but it makes Ninja feel fast, which is almost as important to the original goal as real speed is.

• 这种对命令输出的控制允许Ninja仔细地控制他的总输出。在构建期间，Ninja在运行时显示单独的一行状态；如果构建成功地完成Ninja总的打印输出是单独地一行。这不会使Ninja运行得更快，但它使Ninja感觉更快，这对于最初的目标几乎和真实速度一样重要。

---

Supporting Windows

• 支持Windows

• I wrote Ninja for Linux. Nico (mentioned previously) did the work to make it function on Mac OS X. As Ninja became more widely used people started asking about Windows support.

• 我为Linux写的Ninja。Nico的工作使得它能够在MAC OS X上运转。因为Ninja被更广泛的使用，人们开始询问Windows的支持。

• At a superficial level supporting Windows wasn’t too hard. There were some straightforward changes like making the path separator a backslash or changing the Ninja syntax to allow colons in a path (like c:\foo.txt). Once those changes were in place the larger problems surfaced. Ninja was designed with behavioral assumptions from Linux; Windows is different in small but important ways.

• 在一个表面地层次支持Windows不是很难。这里有一些直接的改变像是将路径以反斜杠分隔或者改变Ninja的语法使它允许路径中的冒号(就像c:\foo.txt)。当这些改变都到位后更大的问题浮出水面。Ninja是假设基于Linux的行为模式设计的；Windows在一些细微但是重要的方法上有些不同。

• For example, Windows has a relatively low limit on the length of a command, an issue that comes up when constructing the command passed to a final link step that may mention most files in the project. The Windows solution for this is “response” files, and only Ninja (not the generator program in front of Ninja) is equipped to manage these.

• 例如，Windows对于命令的长度有一个相对较低的限制，这出现了一个问题，当将一个构建命令传递给一个最终的链接步骤(这回提及项目中的绝大多数文件)。这个的Windows解决方案是“响应文件”，而且只有Ninja(而不是Ninja之前的生成程序)需要配置管理这些。

• A more important performance problem is that file operations on Windows are slow and Ninja works with a lot of files. Visual Studio’s compiler emits header dependencies by simply printing them while compiling, so Ninja on Windows currently includes a tool that wraps the compiler to make it produce the gcc-style Makefile dependency list required by Ninja. This large number of files, already a bottleneck on Linux, is much worse on Windows where opening a file is much more costly. The aforementioned new approach to parsing dependencies at build time fits perfectly on Windows, allowing us to drop the intermediate tool entirely: Ninja is already buffering the command’s output, so it can parse the dependencies directly from that buffer, sidestepping the intermediate on-disk Makefile used with gcc.

• 一个更加重要的性能问题是，在Windows上文件操作很慢，而Ninja工作时需要处理很多的文件。Visual Studio的编译器通过在编译时简单地打印它们来激发头文件依赖，所以当前Windows上的Ninja包括一个工具，它可以将编译器转换为生成Ninja需要的gcc风格的makefile依赖列表。这些很大数量的文件，已经是Linux上的一个瓶颈，这个问题在Windows上更为恶劣，当处理文件时它更加费时。上面提到的在工件时间解析依赖的新方法在Windows上很适用，它允许我们彻底地丢下中间工具：Ninja已将缓存了命令的输出，所以它可以直接从缓冲中解析依赖，使用gcc避开磁盘上的中间Makefile。

• Getting the modification time of a file–GetFileAttributesEx() on Windows and stat() on non-Windows platforms–seems to be about 100 times slower on Windows than it is on Linux.

• 获取文件的修改时间在Windows上的GetFileAttributesEx()和非Windows平台的stat()，Windows上的比Linux上的慢差不多100倍。

• It is possible this is due to “unfair” factors like antivirus software but in practice those factors exist on end-user systems so Ninja performance suffers. The Git version control system, which similarly needs to get the state of many files, can use multiple threads on Windows to execute file checks in parallel. Ninja ought to adopt this feature.

• 这种可能是由于“不公平”因素，例如防病毒软件，但在操作中这些因素存在于终端用户系统，所以Ninja的性能会受到影响。Git版本控制系统，同样需要获取很多文件的状态，可以使用Windows上的多线程来并行执行文件检查。Ninja应当采纳这种特征。

---

Conclusions and Alternative Designs

• 结论和可供替换的设计

• Occasionally on the mailing list someone will suggest that Ninja ought to work instead as a memory-resident daemon or server, especially one coupled with file modification monitors (e.g., inotify on Linux). All these concerns about the time taken to load data and the time to write it back out would not be an issue if Ninja just stayed around between builds.

• 有时在邮件列表中，有人会建议Ninja应当以内存驻守进程或服务的方式工作，尤其是它与文件修改监视器耦合（例如Linux中的inotify）。如果Ninja仅仅只是在呆在构建时期，所有这些关于导入数据的时间和把它写回去的时间的担忧都不会是一个问题。

• In fact, that design was my original plan for Ninja. It was only after I saw the first build worked quickly that I realized it might be possible to make Ninja work without needing a server component. It may yet be necessary as Chrome continues to grow, but the simpler approach, where we gain speed by doing less work rather than more complex machinery, will always be the most appealing to me. It is my hope some other restructurings (like the changes we made to use a lexer generator or the new dependency format on Windows) will be enough.

• 事实上，这个设计是我对于Ninja的原始计划。直到我看到第一次构建工作很迅速后，我意识到让Ninja以不需要服务部分的方式工作是可行的。它仍然是必要的因为chrome在持续发展，但是更简单的方法，我们通过做更少的事情而不是更复杂的来获取速度，这会是最吸引我的。这是我的希望，一些其他的重组(就像是我们做出的使用词法分析生成器或者在Windows上的新的依赖格式的改变)就将会是足够的。

• Simplicity is a virtue in software; the question is always how far it can go. Ninja managed to cut much of the complexity from a build system by delegating certain expensive tasks to other tools (GYP or CMake), and because of this it is useful in projects other than the one it was made for. Ninja’s simple code hopefully encouraged contributions– the majority of work for supporting OS X, Windows, CMake, and other features was done by contributors. Ninja’s simple semantics have led to experiments by others to reimplement it in other languages (Scheme and Go, to my knowledge).

• 简朴在软件中是一种美德；问题是它能走多远呢。Ninja设法通过将确定耗费的任务委托给其他工具(GYP或者CMake)来减少一个构建系统中的复杂度，由于这点，他在除它以外的项目中也很有用。Ninja的简单代码很希望地鼓励贡献，关于支持OS X, Windows, CMake的主要工作，和一些其他的功能都是贡献者完成的。Ninja简单的语义已经导致其他人使用其他语言(据我所知，有Scheme和Go)来复现它的实验。

• Do milliseconds really matter? Among the greater concerns of software it might be silly to worry about. However, having worked on projects with slower builds, I find more than productivity is gained; a quick turnaround gives a project a feeling of lightness that makes me happy to play with it. And code that is fun to hack on is the reason I write software in the first place. In that sense speed is of primary importance.

• 几毫秒真的这么重要吗？在对于软件的更多的关注中，这种忧虑可能是愚蠢的。然而，通过在构建更慢的构建中工作，我发现获得的不仅仅是生产效率；一个快速的转变让一个项目感觉到一种轻量感，这让我使用它时很开心。我起初写软件的原因是写代码很有趣。从这个意义上讲，速度至关重要。

---

Acknowledgements

• 致谢

• Special thanks are due to the many contributors to Ninja, some of whom you can find listed on Ninja’s GitHub project page.

• 特别感谢Ninja的许多贡献者，你可以在Ninja的GitHub项目页陈列中看到他们中的一些人。