idea深入调试跟踪_深入了解：按用户疼痛跟踪错误

idea深入调试跟踪

Continuing the Release Management Series, let’s talk about how we changed our bug prioritization system for 5.4 and why.

继续 发布管理系列 ，让我们谈谈我们如何更改5.4的错误优先级系统以及原因。

In recent times, we have been focusing a lot on quality. One common, though inaccurate, measurement of quality is the bug count. In the last two years, we’ve been crushing out over 11,000 bugs per year (!), but even so, we never came close to reaching zero bugs. We don’t make shrink wrapped software running on fixed hardware platforms with specifications that can be locked in. Aside from cases of our own optimizations or fixes creating new bugs, new issues arise daily. There are countless cases, some of them not even coming from our own code base.

最近，我们一直非常注重质量。 错误计数是一种常见的(尽管不准确)质量度量。 在过去的两年中，我们每年已经淘汰了超过11,000个错误(！)，但是即使如此，我们也从未接近实现零错误。 我们不会制作在固定的硬件平台上运行且具有可以锁定的规格的收缩包装软件。除了我们自己的优化或产生新bug的修复程序外，每天还会出现新问题。 有无数情况，其中有些甚至不是来自我们自己的代码库。

So, the question we ask ourselves is, “Which bug do you fix first?” Unity has so many dimensions to work with, so is it fair to compare a “small platform bug” against a “editor usability bug”? Under the old system, we would track a priority, but the teams defined the priorities individually, so it was decidedly inconsistent. What is a shipstopper for one group does not make it a shipstopper for all. While some teams with larger overviews and ability to cross spaces had a consistent priority system, most did not and what was often organizational team priority was assigned at a global scope. You might see how this comes to be a release management problem.

因此，我们问自己一个问题：“您首先修复哪个错误？” Unity有很多可用的维度，将“小平台错误”与“编辑器可用性错误”进行比较是否公平？ 在旧系统下，我们将跟踪优先级，但是团队分别定义了优先级，因此肯定不一致。 对于一个团体来说，什么是造船者？ 尽管一些具有较大视野和跨空间能力的团队拥有一致的优先级系统，但大多数团队却没有，而且通常在全球范围内分配组织团队的优先级。 您可能会看到这如何成为发布管理问题。

So, in comes “user pain”. The “user pain” idea can be found in a lot of places, but the seeds for us started with another Director of Development in R&D, Erik Juhl, recalling a methodology used during his MMO days where the team would assess the risk of fixing a production bug. It went something like this:

因此，带来了“用户痛苦”。 “用户痛苦”的想法在很多地方都可以找到，但对我们而言，种子发源于研发的另一位开发总监Erik Juhl，他回想起了他在MMO时代使用的方法，即团队可以评估解决问题的风险。生产错误。 它是这样的：

Risk Assessment = Tech Risk + QA Risk + Business Concern + Community Impact

风险评估=技术风险+质量保证风险+业务关注+社区影响

In that situation there was a threshold to reach in order for the whole group to believe a bug was worth fixing immediately. This worked for its purpose of “live” issues in a service and case by case analysis, but it doesn’t exactly scale to bug databases. However, the idea was something we latched on to.

在那种情况下，要使整个团队相信一个错误值得立即修复，就要达到一个阈值。 这样做的目的是为了“解决”服务中的“实时”问题，并进行逐案分析，但并不能完全扩展到错误数据库。 但是，这个想法是我们所坚持的。

寻找公式 (Finding A Formula)

With this example in mind, our release management sampled all the old priority system “Priority 1 (Drop everything now)” and “Priority 2 (Shipstopper)” bugs marked for fixing in 5.4, which amounted to some ~150 bugs at the time in mid-development.

牢记此示例，我们的发布管理对5.4中标记为已修复的所有旧优先级系统“ Priority 1(立即丢弃所有内容)”和“ Priority 2(Shipstopper)”漏洞进行了采样，当时大约有150个错误。发展中期。

We started with something like this:

我们从这样的事情开始：

User Pain = (5 – Severity) * IsRegression + QA Risk + Technical Risk

用户痛苦=(5 –严重程度)* IsRegression +质量保证风险+技术风险

Here, Severity and IsRegression were information that previously existed in our bug tracking system (see below about Severity changes). “QA Risk” was an attempt to describe the risk in the ability to test the bug. “Technical Risk” was attempting to capture the complexity and “halo effect” (the likelihood the fix will influence another part of the system).

在这里，严重性和IsRegression是我们的错误跟踪系统中先前存在的信息(有关严重性更改，请参见下文)。 “ QA风险”是试图描述测试漏洞的风险。 “技术风险”试图捕获复杂性和“晕轮效应”(修复可能影响系统另一部分的可能性)。

In looking at the sampling of bugs and calculated pain, the list ordering looked wrong from our perspective. Mainly, we decided to drop QA Risk & Technical Risk as they are things only for release assessment, “if we should accept a bug fix”, instead of user pain, which should represent “should I prioritize fixing this.”

在查看错误的样本和计算出的痛苦时，从我们的角度来看，列表顺序看起来是错误的。 主要是，我们决定放弃QA风险和技术风险，因为它们仅用于发布评估，“如果我们应该接受错误修复”，而不是用户的痛苦，它应表示“我应该优先解决此问题”。

So, we next refined the formula to something like:

因此，我们接下来将公式提炼为：

User Pain = ((5 – severity) * (Platform Importance * Prevalence) ) * (1 + IsEnterpriseSupport + IsUpcomingPlatform) * (1+IsRegression) + IsInstability * 100 + Votes

用户痛苦=((5 –严重性)*(平台重要性*患病率))*(1 + IsEnterpriseSupport + IsUpcomingPlatform)*(1 + IsRegression)+ IsInstability * 100 +投票

We introduce a number of things here:

我们在这里介绍一些事情：

• Platform Importance – to separate individual edge platforms spanning to cross platform

  平台重要性–分离跨平台的各个边缘平台

1 = default	Single platforms, i.e. Tizen, PS4, etc.
2 = Core Platforms	Desktops + iOS + Android + WebGL
3 = Editor	Greater than Core Platforms, because working on making the game trumps shipping a particular platform
4 = Cross-Platform	Most or all platforms

1 =默认	单一平台，即Tizen，PS4等
2 =核心平台	台式机+ iOS +安卓+ WebGL
3 =编辑器	比核心平台更强大，因为致力于使游戏胜于特定平台
4 =跨平台	大多数或所有平台

• Prevalence to consider how widespread it is

  普遍考虑它有多广泛

1 = Affects few or very select users

2 = Affects some, but not majority of users

3 = Affects a majority

4 = Affects everyone

1 =影响很少或非常选择的用户

2 =影响部分使用者，但不影响大多数使用者

3 =影响多数

4 =影响所有人

• IsEnterpriseSupport – addressing Enterprise Support requests and their importance

  IsEnterpriseSupport –解决企业支持请求及其重要性

• IsUpcomingPlatform – working with various partners, this is a way to highlight issues that might need to be tracked

  IsUpcomingPlatform –与各种合作伙伴合作，是一种突出显示可能需要跟踪的问题的方法

• IsInstability – is a bug causing internal development slowdowns by introducing an automated test instability. While not obviously user pain, it’s indirect and very very impactful which is why it gets such a high rating. These can cause significant slowdowns that impact how much we can fix or improve quickly.

  IsInstability –是由于引入自动测试不稳定而导致内部开发变慢的错误。 尽管显然不会给用户带来痛苦，但它是间接的并且非常非常有影响力，这就是为什么它获得如此高的评价。 这些会导致明显的速度下降，从而影响我们可以快速修复或改进的程度。

• Vote – a simple vote counter collection from the Issue Tracker

  投票–来自问题追踪器的简单投票计数器集合

This formula gave a nice spread of bugs that seemed to properly give weight to appropriate bugs, and we worked with this idea for a while. Over time we made the following adjustments:

这个公式很好地传播了一些错误，这些错误似乎可以适当地权衡适当的错误，因此我们花了一段时间研究了这个想法。 随着时间的流逝，我们进行了以下调整：

• IsUpcomingPlatform was dropped, as it would carry excessive weight beyond the life of the initial release. Plus it was expected that the team in charge of the platform would be owning, raising concern and chasing these bugs.

  IsUpcomingPlatform已删除，因为它将在初始发行版的生命周期之外承担过多的重量。 另外，预计负责该平台的团队将拥有，引起关注并追逐这些错误。

• IsEnterpriseSupport was dropped. This was in recognition that the Enterprise Support team in conjunction with our Sustained Engineering were already on top of these marked bugs. We were running into each other looking at the same bugs with extra engineers now looking at bugs that someone was already working on.

  IsEnterpriseSupport已删除。 这是因为认识到企业支持团队和我们的持续工程团队已经在这些明显的错误之上。 我们彼此相遇，寻找相同的错误，现在有额外的工程师正在研究某个人已经在处理的错误。

• “Release Manager” tag was added to give weight to items like Known Issues or other items showing up lower in the ranks that were of high importance to users, but scoring did not reflect so.

  添加了“ Release Manager”标签，以权衡诸如“已知问题”之类的项目或对用户而言重要性较高的其他排名较低的项目，但得分并未反映出来。

So now we’ve arrived at:

现在我们到达了：

User Pain = ((5 – severity) * (Platform Importance * Prevalence) ) * (1+IsRegression) * (1 + RelMgr) + IsInstability * 100 + Votes

用户痛苦=((5 –严重性)*(平台重要性*患病率))*(1 + IsRegression)*(1 + RelMgr)+ IsInstability * 100 +投票

We started with a threshold of 54, on the basis of:

我们从以下阈值开始为54：

• Severity (Major functionality impacted) >= 2

  严重性(影响主要功能)> = 2

• Platform Importance (at least Editor) >= 3

  平台重要性(至少是编辑器)> = 3

• User Prevalence (Majority) >= 3

  用户流行度(多数)> = 3

• Is Regression (True) = 1

  回归(True)= 1

However once we went on and removed the Enterprise weighting, we dropped the threshold to 48 by arbitrary inspection. This is the current state of things, but we recognize it’s a living idea and may change over time.

但是，一旦继续进行并取消了Enterprise的权重，我们便通过任意检查将阈值降低到48。 这是当前的状态，但是我们认识到这是一个鲜活的想法，并且可能会随着时间而改变。

The one other adjustment was made that our Severity classification was somewhat broken. It would allow minor issues to trump functional issues on the basis of “no workaround available.” Here’s the shift of mapping we decided to implement along side of things:

进行了另一项调整，使我们的严重性分类有所中断。 它将允许小问题在“没有解决方法”的基础上胜过功能问题。 这是我们决定沿着事物的侧面实现映射的转变：

Severity	Old	New
1	Crash or major loss of functionality	Crash, freeze, data loss or work stops
2	No workaround	Core functionality majorly impacted
3	Workaround is possible	Secondary functionality broken
4	Minor or cosmetic issue	Minor or cosmetic issue

严重程度	旧	新
1个	崩溃或功能严重丧失	崩溃，死机，数据丢失或工作停止
2	没有解决方法	核心功能受到重大影响
3	解决方法是可能的	次要功能坏了
4	轻微或装饰性问题	轻微或装饰性问题

采用 (Adoption)

Across R&D there’s been appreciation of the new approach. Many teams have adopted it and are quite pleased thus far. QA members and release management have gone through lots of the existing bugs to assign appropriate values for the newer variables we had to add.

在整个研发过程中，人们都对这种新方法表示赞赏。 到目前为止，许多团队都采用了它，对此感到非常满意。 质量检查成员和发布管理人员已经经历了许多现有的错误，为我们必须添加的新变量分配了适当的值。

With the adoption of this process, people are starting to step away from the old priority mechanisms and view the bugs in light of how they impact the users.

通过采用此过程，人们开始摆脱旧的优先级机制，并根据它们对用户的影响来查看错误。

What’s interesting about the graph above, is that painful bugs still counted for less than our per team assessments of priorities. The “Pain” line grew because we expanded scope of bugs being assessed far beyond 5.4 P1 + P2. It eventually encompassed any bug listed from the past 5.0 to the future 5.5 (5.x) for all old prioritizations of P1 – P3, plus our bug triage queue. The numbers still were less than the “high priority bugs” of our past weighting. Part of the up and down was the incorporation of 5.3 items and then dropping of the IsEnterpriseSupport bugs which already had coverage.

上图有趣的是，痛苦的bug仍然比我们每个团队对优先级的评估少。 “痛苦”线之所以增加，是因为我们扩大了评估漏洞的范围，使其远远超出了5.4 P1 + P2。 最终，它涵盖了从P5到P3的所有优先级，从过去的5.0到将来的5.5(5.x)列出的所有错误，以及我们的错误分类队列。 这个数字仍然小于我们过去加权的“高优先级错误”。 上下波动的一部分是合并5.3项，然后删除已经涵盖的IsEnterpriseSupport错误。

Thus far, what we’re tracking with user pain plus a couple other items like upgrade issues is showing to be largely the main user concerns.The current feeling about the release is fairly positive without correlating as strongly to the number of P1s and P2s.

到目前为止，我们一直在跟踪用户的痛苦以及其他一些问题(例如升级问题)，这在很大程度上已成为用户关注的主要问题。当前对该版本的感觉是相当积极的，而与P1和P2的数量没有很强的相关性。

未来 (Future)

So, what’s next for the “user pain” system?

那么，“用户痛苦”系统的下一步是什么？

Well, currently it’s a fairly manual analysis for us, and not fully integrated into our bug system. Also, we need to take a pass at all the older bugs that haven’t been assessed for platform importance and prevalence.

好吧，目前这对我们来说是一个相当手动的分析，并且还没有完全集成到我们的错误系统中。 另外，我们需要通过所有尚未评估平台重要性和普遍性的较旧的错误。

Additionally, we still know there’s tweaking to the formula to consider. We may have missed some aspects to include or are still giving a lot of weight to area we shouldn’t. Also, there’s a great benefit to being able to change the formulas when needed, so it better reflects the issues of the moment. We know that the pain index is not an absolute measurement, but guidance to help show how we might consider prioritization.

此外，我们仍然知道需要对公式进行调整。 我们可能已经错过了某些方面的内容，或者仍在为我们不应该考虑的领域赋予很多权重。 此外，能够在需要时更改公式也有很大的好处，因此它可以更好地反映当前的问题。 我们知道，痛苦指数并不是绝对的衡量指标，而是有助于显示我们如何考虑优先次序的指南。

Lastly, we’ve started experimenting with the idea across other areas including the release risks for acceptance of fixes during RC/patch. We’re also thinking how this might be useful to prioritize features or re-factors.

最后，我们已经开始在其他领域进行试验，包括在RC /补丁程序中接受修订的发行风险。 我们还在考虑，这对于确定功能或重构的优先级可能有什么用。

It’s all still an experiment and thus far current results and perception is that a more unified system of prioritizing is helping. We’ll continue the experiment forward and update you some time in the future on how things keep going.

这仍然是一个实验，到目前为止，当前的结果和看法是，更加统一的优先级排序系统将有所帮助。 我们将继续进行实验，并在将来的某个时间为您更新情况。

For the next round, we’ll talk about some internal tooling changes.

在下一轮中，我们将讨论一些内部工具更改。

翻译自: https://blogs.unity3d.com/2016/08/17/a-look-inside-tracking-bugs-by-user-pain/

idea深入调试跟踪