分析自动化测试收益_分析自动化测试结果

分析自动化测试收益

So, you’ve set up a sweet automated test framework. You’re running it in your CI/CD pipeline, builds are rejected if tests fail, your regression tests are a fraction of the manual effort they once were, and you’re releasing to prod whenever the h*ck you want! Life is good. Except for that one darn test that fails every few runs. Oh, and the other one that takes a full 4 minutes to execute on Thursdays. Well, I guess there’s the few that have never, ever failed either…

因此，您已经建立了一个很好的自动化测试框架。 您正在CI / CD管道中运行它，如果测试失败，构建将被拒绝，您的回归测试仅是以前的手动工作的一小部分，并且您会在需要时发布产品！ 生活很好。 除了该织补测试每隔几次运行都会失败。 哦，另一只花了整整4分钟才能在星期四执行。 好吧，我想有一些从来没有，也从未失败过的人……

There’s a ton of untapped info to gain from taking a holistic, tilt-your-glasses-down and hold-your-paper-away-from-your-face view of our automation runs over time.

随着时间的流逝，对我们的自动化系统进行全面，倾斜的玻璃放置并从表面上拿开您的纸，可以获得大量未开发的信息。

Running automated tests usually goes something like this: Run tests. Look at the failures (maybe run the build again & cross your fingers). Re-run failed tests on your machine. Check a few selectors, execute a test or two manually, up a timeout value, fix it — someone hard coded a date, whoops— call it a day. Less often, though, is anyone looking at the broader picture. Let’s talk about analyzing automated test results.

运行自动化测试通常是这样的：运行测试。 查看故障(也许再次运行构建并用手指交叉)。 在计算机上重新运行失败的测试。 检查几个选择器，手动执行一个或两个测试，增加一个超时值，然后解决它—有人用硬编码的日期将其命名为“ day”。 但是，很少有人会看到更广阔的前景。 让我们谈谈分析自动化测试结果。

Why do we care? There’s a ton of untapped info to gain from taking a holistic, tilt-your-glasses-down and hold-your-paper-away-from-your-face view of our automation runs over time. Here are a few metrics we can look at to refine and continuously improve our automated test suite.

我们为什么在乎？ 随着时间的流逝，对我们的自动化系统进行全面，倾斜的玻璃放置并从表面上拿开您的纸，可以获得大量未开发的信息。 以下是我们可以用来改进和不断改进我们的自动化测试套件的一些指标。

1.片状测试！ (1. Flaky Tests!)

You know ’em, you don’t love ’em, they’re a pain in everyone’s ass. The test that fails every other run. Or every third run. Or whenever there’s a full moon and it’s a Sunday and it’s NOT daylight savings. Flaky tests can be symptoms of many things, but knowing exactly which tests are flaky and when/how often can help pinpoint why.

你知道他们，你不爱他们，这是每个人的痛苦。 每隔一次运行失败的测试。 或每三个运行一次。 或者，每当有满月，这是星期日，而不是夏令时。 片状测试可能是许多事情的症状，但是准确地知道哪些测试是片状以及何时/多频繁可以帮助查明原因 。

2.始终绿色测试 (2. Always Green Tests)

So you have 20 automated tests. 16 of them pass every time. Every. Single. Time. They have NEVER failed. They’re perfect, we love them, they don’t make us want to scream. Beautiful angel baby *chef’s kiss* tests. But… if they’ve never *ever* failed, are they really that useful? They can tell us a few different stories. One story is that the areas of the application they run against is hardly ever touched. Fine. We want to ensure that regression bugs aren’t seeping into random areas of the app, sure. Another story that they hopefully aren’t telling is that the tests, well, aren’t actually testing anything. Just like false negatives can exist in testing, false positives can too. There is value in purposefully trying to make your tests fail (aka testing your tests). Take the time to change expected values or assert false instead of true, just to see what happens.

因此，您有20个自动化测试。 每次有16个通过。 每一个 单。 时间。 他们从未失败过。 他们是完美的，我们爱他们，他们不想让我们想要尖叫。 美丽的天使宝贝*主厨之吻*测试。 但是……如果他们从未*从未*失败过，它们真的有用吗？ 他们可以告诉我们一些不同的故事。 一个故事是，几乎没有触碰过针对它们运行的​​应用程序区域。 精细。 我们希望确保回归错误不会渗入应用程序的随机区域。 他们希望不告诉的另一个故事是测试实际上没有测试任何东西。 就像测试中可能存在误报一样，误报也可能存在。 有目的地尝试使测试失败(也就是测试)是有价值的。 花些时间更改期望值或断言false而不是true，只是为了看看会发生什么。

Another point of consideration for green tests is that you may not need to run them with every build, if those tests can take a while to run. Utilize test tagging to specify which tests to run when, and shorten the feedback loop.

绿色测试的另一个考虑因素是，如果这些测试可能需要一段时间才能运行，则可能不需要在每个版本中都运行它们。 利用测试标记来指定何时运行哪些测试，并缩短反馈循环。

3.始终红色测试 (3. Always Red Tests)

Inevitably, you’ll have failing tests. Sometimes failures are valid, like when a selector changes or someone forgot to tell the automation team that there were UI copy updates. Such is life. But repeatedly-failing tests can indicate problems in the automation framework, automation code, or the application itself. Taking the time to investigate why these tests are always failing will save time and frustration in the long run. Don’t be afraid to deem a problem-child test be run manually. Evaluate time, effort, and value of automation vs. time and effort of manually executing a test. You’ll always have some tests that don’t make sense to automate and this metric can help point those out.

不可避免地，您将无法通过测试。 有时失败是有效的，例如选择器更改或有人忘记告诉自动化团队有UI副本更新时失败。 这就是人生。 但是反复失败的测试可能表明自动化框架，自动化代码或应用程序本身存在问题。 花时间研究为什么这些测试总是失败，从长远来看将节省时间和沮丧。 不要害怕认为有问题的子级测试可以手动运行。 评估自动化的时间，精力和价值与手动执行测试的时间和精力。 您总会有一些自动化测试没有意义，而该指标可以帮助您指出这些问题。

工具类 (Tools)

Modern tools such as Jenkins and Cypress offer built-in test result analyzers, but if your team isn’t exactly using the latest-and-greatest tools yet, some simple test reporters + formatting + Excel can do the (less-fancy) trick.

詹金斯(Jenkins)和赛普拉斯(Cypress)等现代工具提供了内置的测试结果分析器，但是，如果您的团队尚未完全使用最新，最先进的工具，则一些简单的测试报告器+格式+ Excel可以解决这个问题。

Jenkins测试结果分析器插件 (Jenkins Test Results Analyzer Plugin)

I think at this point it’s rare that a team *isn’t* using Jenkins, so this could be a really accessible option for yours. It isn’t the prettiest, but it’s somewhat customizable and gives us what we’re looking for. Test runs can be visualized over N number of builds, and you can see the execution status or the runtime. This makes it easy to pick out which tests have a tendency to fail or take a long time to execute.

我认为目前很少有团队*不*使用詹金斯，因此这可能是您的一个真正可访问的选择。 它不是最漂亮，但可以自定义，并为我们提供了所需的东西。 可以在N个构建中可视化测试运行，您可以看到执行状态或运行时。 这样可以很容易地找出哪些测试有失败的趋势或需要很长时间才能执行。

Top row = build numbers, left column = test names

第一行=内部编号，左列=测试名称

赛普拉斯测试运行分析 (Cypress Test Run Analytics)

Deservedly so, Cypress is quickly becoming a test automation framework darling among the testing community. I’ll be the first to admit that no matter what framework I’m using at my job, I always set up some little tests for myself in Cypress because it’s fun and so FAST. Anyway. This pretty little dashboard is shiny and new and I’m jealous of anyone that’s using it. Their analytics are more robust than the Jenkins plugin, and notably easier on the eyes. In the left column, you’ll see options to view Top failures and Slowest tests. These metrics can be deduced from the Jenkins plugin data as well, but it’s quite nice that Cypress pulls that together for you.

因此，赛普拉斯理所应当地Swift成为测试社区中的一个测试自动化框架宠儿。 我将第一个承认，不管我在工作中使用什么框架，我总是在Cypress中为自己设置一些小测试，因为它很有趣而且非常快捷。 无论如何。 这个相当小的仪表盘闪亮而新颖，我嫉妒正在使用它的任何人。 与Jenkins插件相比，他们的分析功能更强大，并且在外观上更加轻松。 在左列中，您将看到查看最高失败次数和最慢测试的选项。 这些指标也可以从Jenkins插件数据中推导出来，但是赛普拉斯为您提供帮助非常好。

As with any software, there are more options that’ll provide us the data we’re looking for. Feel free to leave feedback in a comment below of tools you use and metrics you look at for your test runs!

与任何软件一样，还有更多选项可以为我们提供所需的数据。 请在下面的评论中留下您对您使用的工具和测试运行指标的反馈！

翻译自: https://medium.com/software-qe/analyzing-automated-test-results-8ca8a0f7811

分析自动化测试收益