一周内如何测试Unity？

That is the question we constantly face as Quality Experts in Sustained Engineering. Every week we release two new minor versions of Unity with fixes and improvements. A crucial aspect of our job is to assess the quality of these builds, identify possible issues, provide feedback to the development teams and give the green light for the update to be released. This question became even more prominent with the introduction of LTS, our Long Term Support version.  How can you test something as complex as Unity in a week though and make sure that what will end up in the hands of hundreds of thousands of users will not make everything turn into magenta pink?

作为持续工程质量专家，这是我们不断面临的问题。 每周我们都会发布两个新的Unity较小版本，其中包含修复和改进功能。 我们工作的一个关键方面是评估这些构建的质量，发现可能的问题，向开发团队提供反馈并为要发布的更新开绿灯。 随着我们的长期支持版本LTS的推出，这个问题变得更加突出。 但是，如何在一周内测试像Unity一样复杂的东西，并确保最终在成千上万的用户手中得到的东西不会使一切变成洋红色？

让机器人接管 (Let the robots take over)

The first part of the answer to this question is a bit obvious. Automation. Lots and lots of automation actually! Our builds have to pass around 58000 unique tests. Take note of the word “unique” because the actual number of tests that run is much bigger. A lot of these tests run on different configurations and different platforms.

这个问题的答案的第一部分是显而易见的。 自动化。 实际上有很多自动化！ 我们的构建必须通过大约58000个独特的测试。 请注意“唯一”一词，因为实际运行的测试数量要大得多。 这些测试很多都在不同的配置和不同的平台上运行。

So before it even reaches human hands an update release for LTS is getting hammered by tests to make sure that it works as expected. These tests come in a lot of different flavors. The majority of automated tests are Editor and Native tests. The editor tests are the ones that cover the editor’s functionality. The native tests cover the C++ side of Unity’s code. On top of those. we also have Playmode tests, Graphics tests, Performance tests, to name a few. Even if one of those tests fails the build goes back to cooking until we discover what is wrong with the failed test, incorporate the fix and create a new build.

因此，在LTS的更新版本发布之前，它就受到了测试的敲击，以确保其按预期运行。 这些测试有很多不同的口味。 大多数自动化测试是编辑器测试和本机测试。 编辑器测试是涵盖编辑器功能的测试。 本机测试涵盖Unity代码的C ++方面。 在这些之上。 我们还有Playmode测试，Graphics测试，Performance测试，仅举几例。 即使其中一个测试失败，构建也会恢复正常运行，直到我们发现失败的测试出了什么问题，然后合并此修复程序并创建一个新构建。

After the build passes all the tests, it lands in our hands and the manual quality assurance process starts. There are two main phases in this process, the case verification phase, where we verify that all bugs we believed are fixed are actually fixed, and the “no regression” phase, where we check if the fixes have created some new bugs. It would be good to note that the process we follow is the same for both the LTS and the Tech Releases, so they both adhere to the same standards of quality.

构建通过所有测试后，它就落在我们手中，手动质量保证过程开始。 此过程有两个主要阶段，即案例验证阶段(在此阶段我们验证我们认为已修复的所有错误均已实际修复)，以及“无回归”阶段(在此阶段我们检查这些修复程序是否已产生一些新的错误)。 值得一提的是，对于LTS和技术版本，我们遵循的过程是相同的，因此它们都遵循相同的质量标准。

案例验证 (Case verification)

All the fixes that are included in an update are getting verified by the person who is responsible for the associated bug. This could be anyone in the Unity organization: Embedded Quality Engineers who specialise  in certain areas like Graphics or 2D, Field Engineers who filed a bug while getting feedback from a client or, the QA Student Worker team who are the frontline of Unity’s QA, which is responsible for converting the hundreds of reports we get from Unity’s users into bug reports.

更新中包含的所有修复程序均由负责相关错误的人员验证。 这可能是Unity组织中的任何人：专门从事某些领域(例如图形或2D)的嵌入式质量工程师，在从客户那里获得反馈的同时提交错误的现场工程师，或者是Unity QA前线的QA学生工作者团队，负责将我们从Unity用户那里获得的数百个报告转换为错误报告。

Each one of the people above is responsible for checking if the bug has been fixed. Usually that is the second or third time that a bug fix is verified because someone usually also verifies the fix on a development branch before it lands in a particular release version.

上述每个人都有责任检查错误是否已修复。 通常，这是第二次或第三次验证错误修复程序，因为有人通常还会在开发分支将其修复到特定发行版之前对其进行验证。

The above process is how we strive hard to ensure that when we say something is fixed, it is actually fixed. If the fix doesn’t work, the bug goes back to the developer who originally added the fix.

上面的过程是我们努力确保当我们说某件事是固定的时，它实际上是固定的。 如果修复程序不起作用，则该错误会退回到最初添加此修复程序的开发人员。

The positive side effect of the above process is that we get a lot of of eyes on the build. Different people from different areas take a look at the build and assess the quality of it.

上述过程的积极副作用是我们对构建有了很多关注。 来自不同地区的不同人员会查看构建并评估其质量。

发布验收测试 (Release Acceptance Test)

This is our baseline test phase. It’s an all around manual testing phase that pokes different areas of Unity and makes sure that all the basic parts are working as they are supposed to work. It covers main areas like 3D, 2D, Animation, XR, and others.  We also test that everything is building and running correctly on all the popular platforms. We have tests for Windows, MacOS, Android, iOS, and consoles that we support.

这是我们的基准测试阶段。 这是围绕手动测试阶段的所有阶段，可进入Unity的不同领域，并确保所有基本部分均按预期工作。 它涵盖了3D，2D，动画，XR等主要领域。 我们还测试了所有流行平台上的所有东西都在正确构建和运行。 我们已经针对Windows，MacOS，Android，iOS和我们支持的控制台进行了测试。

Our Release Acceptance Test suite is evolving constantly and we change it depending on the changing feature set. Moreover, we do major overhauls from time to time based on how effective certain cases are in catching issues.

我们的发布验收测试套件正在不断发展，我们会根据不断变化的功能集对其进行更改。 此外，我们会根据某些案例在解决问题方面的效果，不时进行大修。

有针对性的探索性测试 (Targeted Exploratory Testing)

The Release Acceptance Test described above is our “nothing major is broken” phase. It’s usually the same across different builds and isn’t affected that much by the type of fixes that go into a release.

上面描述的发布验收测试是我们的“一切都不坏”阶段。 不同版本之间的情况通常是相同的，并且不受发行版中修复程序类型的影响。

In order to identify possible regressions that might come in a build with a fix, we have a targeted exploratory testing phase. Every week we have a meeting where we go over the different type of fixes that are going in an update and we assess them. Depending on this assessment, we decide which areas need a bit more poking around. If we see that a build has a major 2D fix that affects a lot of areas, we’re going to do a round of exploratory testing around the 2D features. Inn order to make it even more specific, the person responsible for testing a specific area is also taking a look at the fixes and the code that was added to make the testing more targeted.

为了确定带有修复的版本中可能出现的回归，我们有针对性的探索性测试阶段。 每周我们都有一个会议，讨论更新中要解决的不同类型的修复程序，并对它们进行评估。 根据这项评估，我们决定哪些地方需要更多检查。 如果我们发现某个版本的主要2D修复会影响很多区域，那么我们将围绕2D功能进行一轮探索性测试。 为了使它更加具体，客栈负责测试特定区域的人员还查看了修复程序和所添加的代码，以使测试更具针对性。

项目测试 (Project Testing)

We have Unity Projects of several popular games and applications that games studios were kind enough to make available to us. We use those to test that nothing is breaking when upgrading from one version to another. We import these projects into the new build and then we run them in the editor to make sure that we don’t get any errors. In most cases, we also create a build to verify nothing is broken across the Unity pipeline.

我们有几个流行游戏和应用程序的Unity项目，游戏工作室对此非常友善，可以提供给我们。 我们使用它们来测试从一个版本升级到另一个版本时没有任何损坏。 我们将这些项目导入到新版本中，然后在编辑器中运行它们，以确保没有任何错误。 在大多数情况下，我们还会创建一个构建以验证Unity管道中没有损坏。

Moreover, we test with various popular asset store packages, with content that our Content team has created, like 3D Game Kit and Tower defence template and also with packages that have been developed internally in order to test specific areas like Particles and 2D Physics.

此外，我们使用各种流行的资产商店程序包，内容团队创建的内容(例如3D游戏套件和塔防模板)进行测试，并使用内部开发的程序包进行测试，以测试特定区域，例如粒子和2D物理。

未来 (The Future)

All the above are the things that we do at the moment to make sure that the LTS is as stable as possible and moving across the LTS line will not create any issues to users. We’re doing fairly well but we’re trying to get even better. That is why whatever you read above might not be completely accurate even 3 months from now. We track how well we’re doing and adjust our process accordingly.

以上是我们目前为确保LTS尽可能稳定而要做的事情，并且跨LTS线移动不会对用户造成任何问题。 我们做的还不错，但我们正在努力变得更好。 这就是为什么即使从现在开始三个月后，您上面所读的内容也可能不完全准确的原因。 我们跟踪我们的工作状况，并相应地调整流程。

Our philosophy is to catch problems as early as possible and even eliminate the factors that might cause them if that is possible. Our end goal is to have “almost zero” regressions on an LTS release, no matter how many fixes land on it.

我们的理念是尽早发现问题，甚至消除可能导致问题的因素。 我们的最终目标是使LTS发行版具有“几乎为零”的回归，而不管其上有多少修复程序。

One example of the things we do to reach that point is a Root Cause Analysis. We identify the critical issues that we missed, try to figure out why we missed them, and more importantly, why the issue was created in the first place. Then depending on the findings, we optimize our process to handle similar issues better in the future.

我们要做到这一点的一个例子是根本原因分析。 我们确定了我们遗漏的关键问题，试图弄清为什么我们遗漏了它们，更重要的是，为什么首先创建了问题。 然后，根据调查结果，我们优化流程以在将来更好地处理类似问题。

A big chunk of our time is also spent identifying tools that might make the overall quality of every Unity release better. These could range from a tool that could give us valuable feedback when a developer commits code to data analytics tools that would make it easier for us to identify riskier areas of the code and optimize our testing. Some of these tools are already helping us, others are prototypes and others are still in our heads.

我们还花费了大量时间来确定工具，这些工具可以使每个Unity版本的整体质量更好。 这些工具的范围很广，从可以在开发人员提交代码时向我们提供有价值的反馈的工具到可以使我们更容易识别代码的高风险区域并优化测试的数据分析工具。 其中一些工具已经在为我们提供帮助，另一些则是原型，而另一些仍在我们的脑海中。

Unity is a complex piece of software that is used by millions of users in all sorts of exciting and creative ways. Is it even possible to test a piece of software like that in a week and make sure that the fixes you put in didn’t create any regression? It’s a hard problem that we are very eager to solve.

Unity是一款复杂的软件，数百万用户以各种令人兴奋和富有创造力的方式使用了Unity。 甚至有可能在一周内测试类似的软件，并确保您放入的修复程序不会造成任何回归？ 我们非常渴望解决这个难题。

翻译自: https://blogs.unity3d.com/2018/08/20/how-to-test-unity-in-a-week/