katana owin_Katana：利用开源工具进行持续集成

katana owin

Hi!  I am Na’Tosha, and, although most people know me as one of the people on the Linux Platform team here at Unity, my main job is working as the Lead Software Developer on the Build & Infrastructure team in R&D.  Back in October of 2011, I wrote a blog post entitled Build Engineering and Infrastructure: How Unity Does It.  A lot has changed at Unity over the years, and how we do infrastructure has changed over the years as well.  The two biggest areas where we have made changes are in the tool we use for our Mercurial hosting and code reviews and in our automated build and continuous integration solution.

嗨！ 我是Na'Tosha，虽然大多数人都知道我是Unity的Linux平台团队的一员，但我的主要工作是担任研发Build＆Infrastructure团队的首席软件开发人员。 早在2011年10月，我写了一篇博客文章，标题为《 Build Engineering and Infrastructure：Unity做到这一点》 。 多年来，Unity发生了很多变化，多年来，我们的基础架构也发生了变化。 我们进行更改的两个最大方面是用于Mercurial托管和代码审查的工具，以及我们的自动构建和持续集成解决方案。

For a few years, Unity has used TeamCity from JetBrains for automated building and testing.  As the R&D team grew here at Unity, the demands on the build infrastructure grew on multiple axes (namely the number of users, the number of changesets, and the number of simultaneous branches).  We reached a point where we needed to accommodate several thousand builds per day, and we started seeing performance problems on multiple fronts: servers became slow to respond, we encountered unexplained errors that we could not fix, new changes were processed very slowly causing delays, webpages took several minutes to load, etc.  After a year of back-and-forth with the makers of TeamCity, and a progressively worsening state of our build infrastructure, I came to the conclusion that the best path forward for us was to switch to a solution that better suits our particular needs (obviously when you have a way of working that is as particular as ours, combined with our scale, extensibility and flexibility are a necessity in any tool . . . and both can be hard to get with off-the-shelf proprietary solutions).  Being a long-time open-source enthusiast, I felt this was a particularly good scenario to leverage the power of open-source to fix our problems.  After some research, I decided that we would build a custom solution on top of Buildbot — an open-source continuous integration framework used by Chromium, Mozilla, Python, and various other projects.  Buildbot is written in Python on top of the Twisted event-driven networking engine.

几年来，Unity使用JetBrains的TeamCity进行自动化构建和测试。 随着Unity研发团队的不断壮大，对构建基础架构的需求也从多个角度(即用户数量，变更集数量和并发分支数量)增长。 我们达到了每天需要容纳数千个构建的地步，并开始看到多个方面的性能问题：服务器响应速度缓慢，遇到无法解释的无法解释的错误，新更改处理得非常缓慢而导致延迟，网页需要几分钟的时间加载，等等。经过与TeamCity的制造商来回的一年的考察，以及我们构建基础结构的不断恶化，我得出的结论是，对我们来说最好的方法是转向一个更适合我们特定需求的解决方案(很明显，当您拥有与我们一样特殊的工作方式时，再加上我们的规模，可扩展性和灵活性在任何工具中都是必不可少的……两者都难以摆脱-现成的专有解决方案)。 作为长期的开源爱好者，我觉得这是利用开源功能解决问题的特别好方案。 经过一番研究，我决定我们将在Buildbot之上构建一个自定义解决方案，该bot是Chromium，Mozilla，Python和其他各种项目使用的开源持续集成框架。 Buildbot是在Twisted事件驱动的网络引擎之上用Python编写的。

回顾 (A Look Back)

阶段1：原型和概念验证 (Phase 1: Prototype and Proof-of-Concept)

It was now September of 2012, and, luckily for me, we had just expanded the Build Engineering team at this time with a new hire – Maria – who already had previous experience working with Buildbot.  We knew this would be a large project, so we started with a 2-month long prototyping/proof-of-concept phase where Maria explored various aspects of Buildbot to test its potential to scale in the future while maintaining the flexibility we needed for our complex build chains.  We knew we wanted the ability to decouple as many parts of the build infrastructure as possible to allow for easier maintenance and debugging.

现在是2012年9月，幸运的是，我们这次刚刚扩建了建筑工程团队，招募了新员工玛丽亚(Maria)，他已经在Buildbot上工作过。 我们知道这将是一个大型项目，因此我们从为期2个月的原型设计/概念验证阶段开始，在该阶段，Maria探索了Buildbot的各个方面，以测试其未来的扩展潜力，同时保持我们所需的灵活性复杂的构建链。 我们知道我们希望能够将构建基础结构的尽可能多的部分分离开来，以便于维护和调试。

An early design diagram for Katana.

片假名的早期设计图。

阶段2：需求收集和实施开始 (Phase 2: Requirements Gathering and Beginning of Implementation)

After around two months of prototyping and proof-of-concept work, we were confident the toolset we had chosen would work — with some serious investment.  The next phase of the project involved doing a feature comparison between Buildbot and TeamCity and an initial attempt to gather requirements for a system that could be used in production as a TeamCity replacement.  This part was hard and required some iteration, because we 1) were still learning about all of the capabilities and limitations of Buildbot, and 2) it was hard to figure out which features TeamCity had that were really useful to us and which ones we could live without.   We started with an initial project plan and schedule, which we revised along the way at regular intervals.  At this point, we brought our IT department in to provide estimates on the amount of hardware we would need to acquire to build a fully-functioning system without taking resources away from our production instances.

在进行了大约两个月的原型设计和概念验证工作之后，我们相信我们所选择的工具集可以投入大量的投资才能工作。 该项目的下一个阶段涉及在Buildbot和TeamCity之间进行功能比较，并初步尝试收集对系统的要求，该系统可以在生产中用作TeamCity的替代产品。 这部分很困难，需要一些迭代，因为我们1)仍在学习Buildbot的所有功能和局限性，以及2)很难弄清TeamCity的哪些功能对我们真正有用，哪些可以没有生活。 我们从最初的项目计划和进度表开始，然后定期进行修订。 在这一点上，我们调动了我们的IT部门，以估算在不占用生产实例资源的情况下，构建一个功能全面的系统所需的硬件数量。

阶段3：前端 (Phase 3: The Front-end)

The version of Buildbot we forked from (0.8.7) does come with a user interface, but coming from TeamCity, it was practically impossible to use, especially with the number of build configurations and number of builds we have.  Performance was of course also a concern; after our previous experiences, we knew the most important thing was that the UI was fast to load — everything else was secondary.  Therefore, we needed someone with UI expertise and a keen eye for design to produce a new UI for us.  We hired a front-end developer — Simon — who was experienced with websites where performance is the main concern.  He was tasked with creating a new Buildbot frontend.

我们从(0.8.7)中派生的Buildbot版本确实带有用户界面，但是来自TeamCity，实际上几乎无法使用，特别是在构建配置数量和拥有的构建数量方面。 性能当然也是一个问题。 根据之前的经验，我们知道最重要的是UI加载速度很快-其他所有内容都是次要的。 因此，我们需要一个具有UI专业知识并且敏锐的设计意识的人为我们提供一个新的UI。 我们聘请了一个前端开发人员Simon —他对网站的性能非常感兴趣，而网站的性能是最主要的问题。 他的任务是创建一个新的Buildbot前端。

阶段4：更多实施 (Phase 4: More Implementation)

This is where the bulk of feature implementation was done.  At one point during these months we did decide to reassess and extend the project schedule after discovering some significant work was needed on the Buildbot side to handle one of our use-cases, but overall, the project went well.  We ended up needing to do some work in our buildsystem (e.g., work around the fact that Python stores internally, and lists, environment variables all in upper-case) and our test frameworks (e.g., make all tests output a standardized XML file containing test results that we could parse) here and there.

这是大部分功能实现的地方。 在这几个月中，我们确实决定重新评估并延长项目进度，因为发现Buildbot方面需要做一些重要的工作来处理我们的一个用例，但总的来说，项目进展顺利。 我们最终需要在构建系统中做一些工作(例如，围绕Python在内部存储并以大写形式列出，列出环境变量这一事实)和我们的测试框架(例如，使所有测试输出包含以下内容的标准化XML文件)我们可以解析的测试结果)。

Towards the end of this phase, we transitioned some internal projects (for example, our internal builds of the Mono runtime and classlibs) from TeamCity to Katana.  This allowed us to gain valuable user testing and feedback in a real-world scenario.  We started an internal focus group of users who were using the “Guinea Pig” projects.  From this, we progressed gradually to a more well-rounded feature set.

在此阶段快要结束时，我们将一些内部项目(例如，内部的Mono运行时内部版本和classlib)从TeamCity过渡到了Katana。 这使我们能够在现实世界中获得有价值的用户测试和反馈。 我们启动了一个内部焦点小组，这些小组的使用者正在使用“几内亚猪”项目。 由此，我们逐渐发展为功能更完善的功能集。

阶段5：生产准备和推出 (Phase 5: Production Readiness and Roll-Out)

This is where we started counting down the list of to-do items before we could transition the main Unity project.  We use Trello for project management with Katana, and it works very well — in particular towards the end of this project where the team of people working on Katana had grown (by this point we had also added another member to our team — Daniel — who had started working on Katana, and I also had started working on Katana development and overseeing the configuration management).

在这里，我们可以在转换主Unity项目之前开始对待办事项列表进行递减计数。 我们将Trello与Katana一起用于项目管理，并且效果非常好-特别是在该项目的最后阶段，Katana的工作人员团队已经壮大(至此，我们还向团队添加了另一位成员Daniel-我已经开始在Katana上工作，而我也已经开始在Katana开发上工作并监督配置管理。

Katana’s Trello Board

武士刀的特洛洛董事会

We migrated the main project to Katana (which, because it was a manual migration and is a very large project, actually took quite some time) and invited users to use this alongside TeamCity for verifying branches to be merged to trunk.  During this time, we fixed more issues and gained more feedback.  In late January of this year, we switched our mainline to building officially on Katana instead of TeamCity.  We’ve been using it since then, and overall, we are very pleased with the improvements it has brought us.

我们将主项目迁移到了Katana(由于这是一个手动迁移，而且是一个非常大的项目，实际上花费了相当长的时间)，并邀请用户将其与TeamCity一起使用以验证要合并到主干的分支。 在这段时间内，我们修复了更多问题并获得了更多反馈。 在今年1月下旬，我们将主线改为正式在Katana而不是TeamCity上进行构建。 从那时起我们一直在使用它，总的来说，我们对它带来的改进感到非常满意。

现状 (The Current State)

Katana lives in our buildbot fork on GitHub under a GPLv2 license.  We are still actively developing it; just a few weeks ago we deployed a real-time updating solution that uses Autobahn.

Katana在GPLv2许可下住在GitHub上的buildbot分支中 。 我们仍在积极开发它； 就在几周前，我们部署了使用Autobahn的实时更新解决方案。

Among other things, we have a good overview of our build status on each branch:

除其他外，我们对每个分支的构建状态都有很好的了解：

And also an overview of what our buildslaves are doing:

同时还概述了我们的构建奴隶正在做什么：

We can see a detailed breakdown of a build or test process:

我们可以看到构建或测试过程的详细细分：

And we have a nice test report to help us when tests fail:

我们有一个不错的测试报告，可以在测试失败时为我们提供帮助：

Katana’s architecture has grown in complexity, but we have been mindful of what elements are important to us.  Katana architecture now looks more like this:

Katana的体系结构变得越来越复杂，但是我们一直在注意哪些元素对我们很重要。 Katana架构现在看起来像这样：

In general we have seen vast improvements in:

总的来说，我们在以下方面取得了巨大的进步：

结论 (Conclusion)

Overall, I consider Katana a roaring success — both in terms of the improvements it has brought to R&D at Unity and also as a shining example of how to leverage the power of open-source tools.  We’re proud to be so instrumental in keeping the wheels turning here in R&D at Unity and I hope you all take advantage of build automation in your own studios.

总体而言，我认为Katana取得了巨大的成功-无论是在Unity上为R＆D带来的改进，还是在如何利用开源工具的力量方面的光辉榜样。 我们为能在Unity的研发中保持轮子转动而发挥作用而感到自豪，我希望你们大家都能在自己的工作室中利用构建自动化的优势。

翻译自: https://blogs.unity3d.com/2014/06/02/katana-leveraging-open-source-tools-for-continuous-integration/

katana owin