il2cpp_IL2CPP内部构件：测试框架

il2cpp

This is the eighth and final post in the IL2CPP Internals series. In this post I’ll deviate a bit from the content of previous posts, and not discuss some aspect of how IL2CPP works at compile time or run time. Instead, we’ll take a look at a brief overview of how we develop and test IL2CPP.

这是IL2CPP Internals系列的第八篇也是最后一篇文章。 在本文中，我将与以前的内容有所不同，而不是讨论IL2CPP在编译时或运行时如何工作的某些方面。 相反，我们将简要概述如何开发和测试IL2CPP。

测试优先开发 (Test-first development)

The IL2CPP team has a strong test-first development mentality. Much of the code for IL2CPP is written using the practice of Test Driven Development (TDD), and very few pull requests are merged to the IL2CPP code without significant test coverage.

IL2CPP团队具有强烈的测试优先开发心态。 IL2CPP的许多代码都是使用“ 测试驱动开发” (TDD)的做法编写的，并且很少有拉取请求被合并到IL2CPP代码中，而没有相当大的测试范围。

Since IL2CPP has a finite (although rather large) set of inputs – the ECMA 335 spec- the process of developing it fits nicely with TDD concepts. Most of tests are written before production code, and these tests always need to fail in an expected way before the code to make them pass is written.

由于IL2CPP具有有限的(虽然相当大)输入集-ECMA 335规范-开发它的过程非常适合TDD概念。 大多数测试是在生产代码之前编写的，而这些测试总是需要以预期的方式失败，然后才能编写使它们通过的代码。

This process helps to drive the design of IL2CPP, but it also provides the development team with a large bank of tests which run rather quickly and exercise nearly all of the existing behavior in IL2CPP. As a development team, this test suite provides two important benefits.

此过程有助于推动IL2CPP的设计，但它也为开发团队提供了大量测试，这些测试运行得相当快，几乎可以执行IL2CPP中的所有现有行为。 作为开发团队，此测试套件具有两个重要的好处。

测试统计 (Testing statistics)

The various types of tests that we run against the IL2CPP code base break down into a few different levels. Here are the number of tests we current have a each level (I’ll discuss what each type of test actually is below).

针对IL2CPP代码库运行的各种类型的测试分为几个不同的级别。 这是我们当前每个级别具有的测试数量(我将在下面讨论每种测试的实际含义)。

• Unit tests

  单元测试

• Integration tests

  整合测试

If all of these tests are green, then we feel confident that we can ship IL2CPP at that moment. We maintain one main development branch for IL2CPP, which always tracks the leading edge branch for development in Unity as a whole. The tests are always green on this main development branch. When they break (which does happen once in a while), someone usually fixes them within a few minutes.

如果所有这些测试都是绿色的，那么我们有信心在那时可以发货IL2CPP。 我们为IL2CPP维护了一个主要的开发分支，该分支始终跟踪整个Unity开发的前沿分支。 在这个主要的开发分支上，测试始终是绿色的。 当它们断裂时(偶尔会发生)，通常有人会在几分钟之内修复它们。

Since developers on our team are forking this main branch for personal development often, it needs to be green at all times. The build and test status for both the main development branch and personal branches are maintained on Katana, Unity’s internal build management system.

由于我们团队中的开发人员经常将这一主要分支用于个人开发，因此它需要始终保持绿色。 主开发部门和个人部门的构建和测试状态均由Unity的内部构建管理系统Katana维护。

We use NUnit to run all of these tests and the drive NUnit in one of three different ways

我们使用NUnit以三种不同方式之一运行所有这些测试并驱动NUnit

• Windows: ReSharper

  Windows： ReSharper

• OSX: Xamarin Studio

  OSX： Xamarin Studio

Types of tests

测试类型

I mentioned four different types of tests above without much explanation. Each of these types of tests serves a different purpose, and they all work together to help keep IL2CPP development moving forward.

我上面提到了四种不同类型的测试，但没有太多解释。 每种类型的测试都有不同的用途，并且它们都可以共同帮助保持IL2CPP的发展。

The unit tests verify the behavior of a small bit of code, typically a method. They set up a situation, execute the code under test, and finally assert some expected behavior.

单元测试可以验证少量代码的行为，通常是一种方法。 他们设置了一个情况，执行了测试中的代码，最后声明了一些预期的行为。

The integration tests for IL2CPP actually run the il2cpp.exe utility on an assembly, compile the generated C++ code to an executable, then run the executable. Since we have a nice reference for IL2CPP behavior (the existing version of Mono used in Unity), these integration tests also run the same assembly with Mono (and .Net, on Windows). Our test runner then compares the results of the two (or three) runs dumped to standard output and reports any differences. So the IL2CPP integration tests don’t have explicit expected values or assertions listed in the test code like the unit tests do.

IL2CPP的集成测试实际上是在程序集上运行il2cpp.exe实用程序，将生成的C ++代码编译为可执行文件，然后运行该可执行文件。 由于我们对IL2CPP行为(Unity中使用的Mono的现有版本)有很好的参考，因此这些集成测试还使用Mono(和Windows中的.Net)运行相同的程序集。 然后，我们的测试运行器比较转储到标准输出中的两个(或三个)运行的结果，并报告任何差异。 因此，IL2CPP集成测试没有像单元测试那样在测试代码中列出明确的期望值或断言。

C# unit tests

C＃单元测试

These tests are the fastest, and lowest level tests that we write. They are used to verify the behavior of many parts of il2cpp.exe, the AOT compiler utility for IL2CPP. Since il2cpp.exe is written entirely in C#, we can use fast C# unit tests to get good turn-around time for changes. All of the C# unit tests complete in a few seconds on a nice development machine.

这些测试是我们编写的最快，最低级别的测试。 它们用于验证il2cpp.exe(IL2CPP的AOT编译器实用程序)的许多部分的行为。 由于il2cpp.exe完全用C＃编写，因此我们可以使用快速的C＃单元测试来获得良好的更改周转时间。 所有C＃单元测试都在一台不错的开发机器上在几秒钟内完成。

C++ unit tests

C ++单元测试

The vast majority of the runtime code for IL2CPP (called libil2cpp) is written in C++. For parts of that code which are not easily accessible from a public API, we use C++ unit tests. We have relatively few of these tests, as most of the behavior of code in libil2cpp can be exercised via our larger integration test suite. These tests to require more time than you might expect for unit tests to run, as they need to run il2cpp.exe itself to set up their fixture data.

IL2CPP的绝大多数运行时代码(称为libil2cpp)都是用C ++编写的。 对于无法从公共API轻松访问的部分代码，我们使用C ++单元测试。 这些测试相对较少，因为libil2cpp中的大多数代码行为都可以通过更大的集成测试套件来实现。 这些测试所需的时间比运行单元测试所需的时间长，因为它们需要运行il2cpp.exe本身来设置其夹具数据。

C# integration tests

C＃集成测试

The largest and most comprehensive test suite for IL2CPP is the C# integration test suite. These tests a divided into smaller segments, focusing on tests that verify behavior of icalls, code generation, p/invoke, and general behavior. Most of the tests in this suite are rather short, only about 5 – 10 lines long. The entire suite runs in less than one minute on most machines, but we can run it with various IL2CPP options related to things like stripping and code generation.

C2集成测试套件是针对IL2CPP的最大，最全面的测试套件。 这些测试分为较小的部分，重点是验证icalls行为，代码生成，p / invoke和常规行为的测试。 该套件中的大多数测试都相当短，只有大约5 – 10行长。 整个套件在大多数计算机上的运行时间不到一分钟，但是我们可以使用与剥离和代码生成等相关的各种IL2CPP选项来运行它。

IL integration tests

IL整合测试

These tests are similar in toolchain to the C# integration tests. However, instead of writing the test code in C#, we use the ILGenerator class to directly create an assembly. Although these tests can take a bit more time to write than C# tests, they offer increased flexibility. Often we run into problems with IL code that is invalid or not generated by our current Mono C# compiler. In these cases, we can often write a good test case with IL code. The tests are also beneficial for comprehensive testing of opcodes like conv.i (and similar opcodes in its family) which have clear behavior with many slight variations. All of the IL tests complete end to end in less than one minute.

这些测试在工具链中与C＃集成测试相似。 但是，不是使用C＃编写测试代码，而是使用ILGenerator类直接创建一个程序集。 尽管这些测试可能比C＃测试花费更多时间，但是它们提供了更大的灵活性。 通常，我们当前的Mono C＃编译器会生成无效或未生成的IL代码问题。 在这些情况下，我们通常可以使用IL代码编写一个好的测试用例。 这些测试还有助于对conv.i (及其系列中的类似操作码)之类的操作码进行全面测试，这些操作码具有明显的行为，但有许多细微变化。 所有的IL测试都在一分钟之内完成首尾相接。

We run all of these tests through many variations and options on Katana. From a clean pull of the source code to completed test runs, we see about 20-30 minutes of runtime depending on the load on the build farm.

我们通过Katana上的许多变体和选项来运行所有这些测试。 从源代码的干净提取到完成的测试运行，根据构建场的负载，我们看到大约20-30分钟的运行时间。

为什么要进行这么多的集成测试？ (Why so many integration tests?)

Based on these descriptions, it might seem like our test pyramid for IL2CPP is upside down. And indeed, the end-to-end integration tests (near the top of the pyramid) make up most of our test coverage.

基于这些描述，似乎我们的IL2CPP 测试金字塔倒挂了。 实际上，端到端集成测试(在金字塔顶部附近)构成了我们测试的大部分内容。

Following TDD practice with test times more than a few seconds can be difficult as well. We work to mitigate this by allowing individual segments of the integration test suites to run, and by doing incremental building of the C++ code generated in the test suites (this is how we are proving out some incremental building possibilities for Unity projects with IL2CPP, so stay tuned). Then the turn-around time for an individual test is reasonable (although still not as fast as we would like).

按照TDD练习，测试时间超过几秒钟也很困难。 我们通过允许运行集成测试套件的各个部分，以及通过增量构建测试套件中生成的C ++代码的方式来减轻这种情况(这就是我们为带有IL2CPP的Unity项目提供一些增量构建可能性的方式，因此敬请关注)。 这样，单个测试的周转时间是合理的(尽管仍然没有我们想要的那么快)。

This heavy use of integration tests was a conscious decision though. Much of the code in IL2CPP looks different than it used to, even at our initial public releases in January of 2015. We have learned plenty and changed many of the implementation details in the IL2CPP code base since its inception, but we still have many of the original tests written years ago. After trying out tests at a number of different levels (including even validating the content of the generated C++ source code), we decided that these integration tests give us the best runtime to test stability ratio. Seldom, if ever, do we need to modify one of the existing integration tests when something changes in the IL2CPP code. This fact gives us tremendous confidence that a code change which causes a test to fail is really a problem. It also let’s us refactor and improve the IL2CPP code as much as we need to without fear.

但是，大量使用集成测试是一个有意识的决定。 即使在2015年1月的首次公开发布中，IL2CPP中的许多代码看起来也与以前不同。自IL2CPP代码库问世以来，我们已经学到了很多东西，并更改了许多实现细节，但是我们仍然有很多几年前写的原始测试。 在尝试了许多不同级别的测试(甚至包括验证生成的C ++源代码的内容)之后，我们认为这些集成测试为我们提供了最佳的运行时测试稳定性比率。 很少(如果有的话)，当IL2CPP代码发生变化时，我们是否需要修改现有的集成测试之一。 这一事实使我们充满信心，认为导致测试失败的代码更改确实是一个问题。 这也使我们可以重构和改进IL2CPP代码，而无需担心。

更大的测试 (Even larger tests)

Outside of IL2CPP itself, the IL2CPP code fits into the much larger Unity testing ecosystem. For each platform we ship supporting IL2CPP, we execute the Unity player runtime tests. These tests build up a single Unity project with more than 1000 scenes, then execute each scene and validate expected behavior via assertions. We usually don’t add new tests to this suite for IL2CPP changes (those tests usually end up being at a lower level). This suite serves as a check against regressions that we might introduce with IL2CPP on a given platform. This suite also allows us to test the code used in integration IL2CPP into the Unity build toolchain, which again varies for each platform. A typical runtime test suite completes on about 60-90 minutes, although we often execute individual tests locally much faster.

在IL2CPP本身之外，IL2CPP代码适合更大的Unity测试生态系统。 对于我们支持IL2CPP的每个平台，我们都执行Unity Player运行时测试。 这些测试将构建一个包含1000多个场景的Unity项目，然后执行每个场景并通过断言验证预期的行为。 我们通常不会在此套件中为IL2CPP更改添加新测试(这些测试通常最终处于较低级别)。 该套件用作对我们可能在给定平台上使用IL2CPP引入的回归的检验。 该套件还使我们能够测试将IL2CPP集成到Unity构建工具链中所使用的代码，该代码对于每个平台也有所不同。 典型的运行时测试套件大约需要60-90分钟才能完成，尽管我们通常会在本地更快地执行单个测试。

The largest and slowest tests we use for IL2CPP are Unity editor integration tests. Each of these tests actually runs a different instance of the Unity editor. Most of the IL2CPP editor integration tests focus on building a running a project, usually with various editor build settings. We use these tests to verify things like complex editor integration, error message reporting, and project build size (among many others). Depending on the platform, integration test suites run in a few hours, and usually are executed at least nightly, if not more often.

我们用于IL2CPP的最大，最慢的测试是Unity编辑器集成测试。 这些测试中的每一个实际上都运行Unity编辑器的不同实例。 大多数IL2CPP编辑器集成测试通常着重于使用各种编辑器构建设置来构建运行项目。 我们使用这些测试来验证诸如复杂的编辑器集成，错误消息报告和项目构建大小(等等)之类的事情。 根据平台的不同，集成测试套件会在几个小时内运行，并且通常至少每晚执行一次(如果不是更频繁的话)。

这些测试有什么影响？ (What is the impact of these tests?)

At Unity, one of our guiding principles is “solve hard problems”. I like to think about the difficulty of problems in terms of failure. The more difficult a problem is to solve, the more failures I need accomplish before I can find the solution.

在Unity，我们的指导原则之一是“解决难题”。 我喜欢从失败的角度考虑问题的难度。 解决问题的难度越大，找到解决方案之前我需要完成的故障越多。

Creating a new highly-performant, highly-portable AOT compiler and virtual machine to use as a scripting backend in Unity is a difficult problem. Needless to say, we’ve accomplished thousands of failures along the way. There are more problems to solve, and so more failures to come. But by capturing the useful information from almost all of those failures in a comprehensive and fast test suite, we can iterate very quickly.

创建一个新的高性能，高度可移植的AOT编译器和虚拟机以用作Unity中的脚本后端是一个难题。 不用说，我们已经完成了成千上万的失败。 还有更多的问题需要解决，因此会有更多的失败。 但是，通过在全面，快速的测试套件中捕获几乎所有此类故障的有用信息，我们可以非常快速地进行迭代。

For the IL2CPP developers, our test suite is not so much a means to verify bug-free code (although it does catch bugs), or to help port IL2CPP to multiple platforms (it does that too), but rather, it is a tool we can use to fail fast and solve hard problems so our users can focus on creating beautiful things.

对于IL2CPP开发人员而言，我们的测试套件不是验证无错误代码(尽管它确实捕获到错误)或帮助将IL2CPP移植到多个平台(也可以这样做)的一种手段，而是一种工具我们可以用来快速失败并解决难题，因此我们的用户可以专注于创造美丽的事物。

结论 (Conclusion)

We hope that you have enjoyed the IL2CPP Internals series of posts. We’re happy to share implementation details and provide debugging and performance hints when we can. Let us know if you want to hear more about other topics related to the design and implementation of IL2CPP.

我们希望您喜欢IL2CPP Internals系列文章。 我们很乐意分享实施细节，并在可能时提供调试和性能提示。 如果您想进一步了解与IL2CPP的设计和实现有关的其他主题，请告诉我们。

翻译自: https://blogs.unity3d.com/2015/07/20/il2cpp-internals-testing-frameworks/

il2cpp