unity 持续集成_Unity Downloader：为编辑器提供持续集成依赖

unity 持续集成

Unity has been using Katana over the years for continuous integration (CI). Katana is used for building the releases and running all our internal test suites before we ship a release to you. It’s been working well for a number of years since most of Unity was in the same mercurial repository. 

多年来， Unity一直在使用 Katana 进行持续集成(CI)。 Katana用于构建发行版并运行我们的所有内部测试套件，然后再将发行版交付给您。 自从大多数Unity都在同一个软件仓库中以来，它一直运行良好。

Whenever anyone at Unity wanted to run the editor, it was then possible to either: 

每当Unity上的任何人想要运行编辑器时，都可以执行以下任一操作：

• go to an internal website to download the upcoming builds

  转到内部网站下载即将发布的版本

• build the editor from source code locally

  从本地源代码构建编辑器

• Or more recently, use the Unity Hub to get the latest published builds

  或更近期一点，使用Unity Hub获得最新发布的版本

All three available solutions were covering our internal needs very well until recently. 

直到最近，所有三个可用的解决方案都很好地满足了我们的内部需求。

然后来包 (Then came packages)

Then, we decided to evolve and start using packages. More and more of Unity features are being moved from the main Unity repository and stored into separate packages repositories. This has allowed us to speed up the delivery process and help take some Unity features and extensions to the next level. It was also a much-needed step on our roadmap to DOTS. 

然后，我们决定发展并开始使用 package 。 越来越多的Unity功能从主要的Unity存储库中移出并存储到单独的软件包存储库中。 这使我们能够加快交付过程，并帮助将一些Unity功能和扩展提升到一个新的水平。 这也是我们制定 DOTS 路线图上急需的步骤 。

But like any transition, this new technical paradigm came with a set of challenges. 

但是，像任何过渡一样，这种新的技术范式也带来了一系列挑战。

We, of course, wanted to be able to verify and publish these packages and make sure that each of them would play nicely with the different editor versions, and satisfy our internal quality requirements. Packages have their own repositories with a new CI system running on them. 

当然，我们希望能够验证和发布这些程序包，并确保每个程序包都能与不同的编辑器版本完美配合，并满足我们的内部质量要求。 软件包具有自己的存储库，并在其上运行新的CI系统。

For a package’s CI to be successful, the most important thing is its main dependency: the Unity Editor. We have had no seamless and unified way for the package CI to request a specific Editor build.

要使包的配置项成功，最重要的是它的主要依赖性：Unity编辑器。 对于包CI，我们没有无缝和统一的方法来请求特定的Editor构建。

So last year we ended up with a lot of the package repos having their own custom way of grabbing the editor. Some of them ended up running CI on physical machines that would just have the editor permanently installed on them with hard-coded paths to each version needed.

因此，去年我们最终获得了许多打包存储库，它们具有自定义的抓取编辑器的方式。 其中一些最终在物理计算机上运行CI，而该物理计算机将永久性地安装了编辑器，并带有指向每个所需版本的硬编码路径。

Others created custom tools which would trigger Katana builds and download them to the CI machines.

其他人创建了自定义工具，这些工具将触发Katana构建并将其下载到CI计算机。

All the solutions were quite hacky and not particularly generic.

所有的解决方案都是很棘手的，并不是特别通用。

So a couple of us sat down at the start of September 2018, using our 20% time that we get in R&D for freeform experimentation, which is also known as Fridays Are For Fun, and set about to try and create a generic way of getting editors for CI that would work for as many people as possible.

因此，我们中的一些人在2018年9月初坐下来，利用我们20％的研发时间进行自由形式的实验，这也被称为 Fridays For Fun ，并开始尝试创建一种通用的获取方式CI的编辑器，可以为尽可能多的人使用。

And thus, a couple of weeks later, the Unity Downloader was born.

因此，几周后，Unity Downloader诞生了。

什么是Unity Downloader？ (What is the Unity Downloader?)

The Unity Downloader is a command-line tool made in Python.

Unity Downloader是使用Python制作的命令行工具。

It allows our employees to download any editor (5.0+) for MacOS, Windows and Linux with any combination of components (WebGL, Android, etc).

它允许我们的员工为MacOS，Windows和Linux下载具有组件组合(WebGL，Android等)的任何编辑器(5.0+)。

The Unity Downloader supports this for any Unity version, branch and revision we have.

Unity Downloader支持我们拥有的任何Unity版本， 分支和修订版。

It’s obviously not just as simple as that, there is some heavy lifting going on in the background that you can’t see, like:

显然，这不仅如此简单，而且在您看不见的背景中还有一些繁重的工作，例如：

• What is the latest release of 2019.1?

  2019.1的最新版本是什么？

• What revision is it?

  它是什么版本？

• Do we have a build for it?

  我们有它的构建吗？

• How do we reduce unnecessary waiting when people request popular things?

  当人们要求流行商品时，我们如何减少不必要的等待？

So basically the downloader is a combination of these things:

因此，基本上下载器是这些东西的组合：

• The command-line interface (CLI) itself

  命令行界面(CLI)本身

  •  which can request, download and extract editor installers and combine them into a valid editor installation.

    它可以请求，下载和提取编辑器安装程序，并将它们组合成有效的编辑器安装。

  The command-line interface (CLI) itself

  命令行界面(CLI)本身

• Caches for previously requested editors

  先前请求的编辑器的缓存

  • We have three of those right now, in locations where it’s easier for our build systems to download them quickly.

    现在，在我们的构建系统更容易快速下载它们的位置，我们拥有其中的三个。

  • It also uses the public download links as a 4th source in the situation where someone requests a published build.

    在有人请求已发布的版本的情况下，它还将公共下载链接用作第四来源。

  Caches for previously requested editors

  先前请求的编辑器的缓存

• The backend

  后端

  • Which has a rest API the CLI communicates with to request new editor builds to be triggered in Katana.

    CLI具有与之通信的其余API，以请求在Katana中触发新的编辑器版本。

  • It also monitors the triggered builds and communicates progress back to the CLI. Then it puts the finished build in our caches so the CLI can download them.

    它还监视触发的构建并将进度传达回CLI。 然后将完成的构建放入我们的缓存中，以便CLI可以下载它们。

  The backend

  后端

• Pre-fetcher

  预取器

  • A small script that runs in a cron job which monitors our most popular branches (like trunk) and will ensure builds for these are triggered when a new batch lands. This is to minimize the waiting for when someone requests to download a trunk editor since it’s very likely to be requested frequently.

    在Cron作业中运行的一个小脚本，用于监视我们最受欢迎的分支(例如主干)，并确保在新批次到达时触发针对这些分支的构建。 这是为了尽可能减少有人请求下载中继编辑器的等待时间，因为它很可能经常被请求。

  Pre-fetcher

  预取器

• Janitor

  看门人

  • We can’t store the builds for all of eternity. They take a lot of space and at the same time, we tend to stop using the older revisions a couple of days after something new comes along. So this script just cleans up our caches, deleting files that haven’t been downloaded in the past 30 days.

    我们无法存储所有永恒的版本。 它们占用了大量空间，同时，我们倾向于在出现新情况的几天后停止使用旧版本。 因此，此脚本只是清理缓存，删除过去30天内未下载的文件。

  Janitor

  看门人

With all this in place, we could now suddenly run CI on ephemeral VMs which get destroyed and recreated in a clean state after a job ends.

完成所有这些操作后，我们现在可以突然在临时VM上运行CI，该VM在作业结束后会被销毁并在干净状态下重新创建。

It also easily lets our teams set up their package CI and run it against any supported Unity version they want to test against.

它还使我们的团队可以轻松地设置其程序包CI并针对他们要测试的任何受支持的Unity版本运行它。

If they need to do a bug fix in the editor itself for their package and ensure it will work in their package CI when it lands, then they can easily tell CI to run using that branch by modifying the Unity Downloader command line to something like this:

如果他们需要为自己的软件包在编辑器中进行错误修复，并确保其着陆时能在其软件包CI中运行，则可以通过将Unity Downloader命令行修改为类似的内容，轻松地告诉CI使用该分支运行。 ：

The flow for the above request would be as follows, behind the scenes:

上述请求的流程将在后台进行如下操作：

1. Since we requested a branch we need a revision. So the CLI will ask the backend for what the current latest revision is for the branch 2019.1/myFix, which might return abc123456789.

   由于我们请求分支机构，因此我们需要进行修订。 因此，CLI将向后端询问分支机构2019.1 / myFix的当前最新版本，该版本可能返回abc123456789。

2. The CLI now queries the caches to see if it has the Editor and the il2cpp build target for that revision. If it does then it would simply start downloading it and it would be done. However, if there’s a cache miss, it would continue.

   CLI现在查询高速缓存，以查看其是否具有该版本的编辑器和il2cpp构建目标。 如果确实如此，那么它将简单地开始下载它并完成。 但是，如果存在缓存未命中，它将继续。

3. It would now ask the backend to get the editor and il2cpp component for current OS for the resolved revision.

   现在，它将要求后端获取当前OS的编辑器和il2cpp组件以获取已解决的修订版。

4. The backend will now start by looking among the build artifacts of Katana to see if there already exists a build of it. If it doesn’t, then it will simply trigger it and communicate with the CLI what the estimated time is for the build to complete as well as giving it a Katana build URL in case a user would want to see why it’s taking so long.

   现在，后端将通过在Katana的构建工件中查找，以查看是否已经存在该构建。 如果没有，那么它将简单地触发它，并与CLI交流完成构建所需的估计时间，并为它提供一个Katana构建URL，以防用户想知道为什么花这么长时间。

5. When the build is done, the backend will upload it to the caches and then inform the CLI. that it can now retry its request from step 2.

   构建完成后，后端会将其上载到缓存，然后通知CLI。 现在它可以重试第2步的请求。

6. The CLI downloads and extracts the builds and the user can launch it at their leisure.

   CLI下载并提取构建，用户可以随意启动它。

That’s a simplified version of what it is and does.

那是它的本质和作用的简化版本。

当下 (Present)

All internal package CI is now using the unity downloader in some way. Most of our package repo owners don’t know about this since it’s seamlessly handled by our CI tooling to invoke the downloader before it starts building and running tests. The repo owners only need to indicate what version or branch they want and they get it.

现在，所有内部软件包CI都以某种方式使用统一下载器。 我们的大多数软件包库所有者都不知道这一点，因为它由CI工具无缝处理，可以在开始构建和运行测试之前调用下载程序。 回购所有者只需要指出他们想要的版本或分支，他们就会得到它。

Some statistics for the month of May 2019 is:

2019年5月的一些统计数据是：

• 42,000 editors downloaded

  下载了42,000个编辑器

• 18,000 downloaded components for above editors

  为上述编辑者下载了18,000个组件

• 80% of usage came from VMs in our build farm, the remaining 20% is from employee machines

  80％的使用量来自我们构建场中的VM，其余20％来自员工计算机

In total 300,000 files downloaded since we started using metrics in November.

自从我们从11月开始使用指标以来，总共下载了300,000个文件。

The graph shows the peaks and valleys of usage throughout a month. You can easily see the nightly jobs we have in CI (peaks) and the weekend lull (flat sections).

该图显示了整个月的使用高峰和低谷。 您可以轻松地看到我们在配置项(夜间)中的夜间工作(高峰)和周末的平淡(平坦部分)。

So for CI, the use case is pretty obvious. However, a significant chunk of the traffic is coming from actual humans. Some of us have augmented our workflows heavily around its usage.

因此，对于CI，用例非常明显。 但是，很大一部分流量来自实际的人。 我们中的一些人围绕它的使用极大地扩展了我们的工作流程。

One situation that the Unity Downloader has been useful for is when tracking down causes of bugs by means of bisection. Since we already tend to find out which published Unity version a bug was introduced in we can trivially get the range of revisions between when it was introduced and the release preceding it. With the Unity Downloader, you can then upfront trigger building as many intermediate revisions as your conscience will allow with something like this.

Unity Downloader有用的一种情况是通过二分法来跟踪错误的原因。 由于我们已经倾向于找出哪个发行的Unity版本引入了一个错误，因此我们可以轻易地获得引入它的时间与发布它之前的发行版之间的修订范围。 有了Unity Downloader，您就可以按照这样的意愿提前触发构建尽可能多的中间修订版本。

Setting the –skip-download flag and omitting the — wait flag is the quickest way of just making sure that things are cached and builds are triggered. So a quick succession of those calls will ensure anything that hasn’t been built will be triggered. That ensures that as the bisection progresses after the first couple of revisions tested, the remaining ones are most likely built and ready for download. Doing this in a more traditional sense and building each revision locally would be a very time-consuming task.

设置–skip-download标志并省略– wait标志是确保已缓存内容并触发构建的最快方法。 因此，这些调用的快速连续操作将确保触发尚未构建的任何操作。 这样可以确保在测试了前两个修订版后，随着二等分的进行，其余的修订版很可能已构建并可以下载。 从更传统的意义上讲，在本地进行每个修订都是一项非常耗时的任务。

Another scenario is to quickly test a Unity project locally. To download the Editor for the project and launch it, I can just do the following:

另一种情况是在本地快速测试Unity项目。 要下载该项目的编辑器并启动它，我可以执行以下操作：

Which will get the exact editor revision the project was made for and won’t end up triggering the script updater and so on.

它将获得针对项目进行的确切编辑器修订，并且不会最终触发脚本更新程序等。

So, given that it is now widely used in our automation and has helped our own workflows, I would say it has been a success so far.

因此，鉴于它现在已广泛应用于我们的自动化领域并已经帮助了我们自己的工作流程，我想说到目前为止，它已经取得了成功。

However, as with all solutions concocted in a couple of weeks, there are some issues with the design of the system which causes problems.

但是，就像在几周内完成所有解决方案一样，系统的设计存在一些问题，这会引起问题。

These problems are mainly:

这些问题主要是：

• The CLI has a lot of business logic in it, it knows about all known components and where they should go. This makes it so that we need to release a new version pretty frequently.

  CLI中包含许多业务逻辑，它了解所有已知的组件以及它们的去向。 这使得我们需要非常频繁地发布新版本。

未来 (Future)

Now that the Unity Downloader has reached enough maturity to be widely used internally (especially in CI), we would like to investigate options for making it available for Unity users. It improved our workflows a lot, and we’re confident that it could play a role in helping solve other problems, both internally and externally. 

现在，Unity Downloader已经达到了足够的成熟度，可以在内部广泛使用(尤其是在CI中)，我们想研究使其可供Unity用户使用的选项。 它极大地改善了我们的工作流程，我们相信它可以在帮助解决内部和外部其他问题方面发挥作用。

The plan is, over the next weeks and months, to do a complete refactoring of the Unity Downloader.

计划在接下来的几周和几个月内，对Unity Downloader进行完全重构。

So with that in mind, and with all the lessons learned along the way, we would like to take a step back and take a moment to involve our users, in order to make the new version of the tool more fit for external needs in case we find a way to expand its usage. 

因此，考虑到这一点以及在此过程中获得的所有经验教训，我们想退后一步，花点时间让我们的用户参与进来，以使新版本的工具更适合外部需求，以防万一。我们找到了扩大其使用范围的方法。

To make sure that the new Unity Downloader fits the needs of our users, especially regarding CI, we would love to get some feedback from you all.

为了确保新的Unity Downloader能够满足我们用户的需求，尤其是关于CI的需求，我们希望从大家那里获得一些反馈。

Do you feel like you would have a need for being able to download any or all published editor versions from a terminal of your choice?

您是否认为需要从所选终端下载任何或所有已发布的编辑器版本？

Do you have your own CI solution where this might come in handy, and if so then what technical requirements might you have to make it viable for you? 

您是否有适合自己的CI解决方案？如果可以，那么您可能需要满足哪些技术要求才能使其可行？

Even if you don’t see yourself using it we would like to know why you think you wouldn’t.

即使您看不到自己在使用它，我们也想知道为什么您认为自己不会使用它。

We are all ears.

我们都是耳朵。

翻译自: https://blogs.unity3d.com/2019/07/15/unity-downloader-powering-the-editor-as-a-continuous-integration-dependency/

unity 持续集成