众包软件的两种测试方法

编者按：这篇文章来自于Markus Steinhauser，他是测评鸟公司（Testbirds）的合作创始人和首席通讯官，测评鸟公司是一个软件测试公司。Steinhauser 阐述了托管和自助服务两种众包软件测试方法的不同。你可以在测评鸟公司的网站上找到最新的信息。

近几年众包计划的辐射范围和软件提供者数量都有极大的增长。你可以设计自己的产品，建立并测试你的网站去卖这些产品，然后翻译你的网站去击败其他网站——所有的的这些都可以来自大众的力量。

根据不同的任务，一些平台提供为其访问者提供了两种模式：自助服务，托管服务。哪种更好呢？看情况而定！

即使是在同一领域，不同类型的众包方法使用的商业模式也是不同的，甚至商业模式有极大的差别。文章下面将集中精力讨论众包测试, 即利用实际消费者和相关专家去测评软件。

让我们假设一个公司准备开发一个新的app应用程序并决定使用众包测试作为质量保证的一部分。这个公司的产品经理要知道两件事：

1、我们目标群体对于这个app的看法？

2、它是否能够在多种设备上运行？特别是IOS和Android?

首先，测试需要设置。这部分包括定义项目的范围，选择一个特定的目标群体，确定需要回答的问题。

为了众包测试项目能够成功，在测试过程中详细的项目管理是至关重要的。测试人员可以远程作业，在家，在工作单位，在火车上，甚至当地的咖啡店里进行工作。他们需要通过书面文件，截图，或截屏记录工作。这点非常重要，有两个原因：

由于测试员不是内部人员，记录相关工作是确保有足够测试覆盖率的唯一途径。是不是所有的东西都是起初想去真正测试的？

如果发现有问题，他们必须是可追溯的和可重复的。如何让用户知道到底是什么问题？

一旦所有参与测试的人员完成了他们的工作，结果需要进行汇总和分析。根据项目的目的，每一个步骤所需的资源可能会有所不同。这一部分通常是费时，甚至更小的测试。

产品经理可以选择托管式的众包测试或自助式的众包测试。由于托管式测试可以大大节省时间和人力成本，它相对也就更贵一些。后者自助式的测试则需要安装、项目管理和评估全部自己完成。

如果不考虑项目的具体细节，很难去说哪种方式更好。然而，有一些基本的准则可以遵循。因为所有的参数起初都要设定，单独的项目需要最多的资源。相反，重复出现的项目设定所花费的时间较少。如果一个产品在一段时间里的发布特别频繁，比如每个月几次，前面所述的这一点就特别相关。此外，管理一大群人远程工作，若为实现高质量的结果则需要大量的经验。

因此，以下几个方面可以帮助你决策是选用托管式的还是自助式的：

1、这些文件是很重要的吗？

2、我们是否有经验和团队去完成这个项目？

3、我们能够管理他们吗?

4、我们有做这一切的内部资源吗?

5、预算是否允许进行托管式测试？

如果把所有因素都考虑进去，一般至少需要一个托管式的测试来作为开始。然后可以考虑精力和费用去选择是否采用自助式的测试。如果这个选择是长期的，使这种方式成为公司内部的一个流程也是一种挑战。从托管式工作的框架下获得经验，公司就可以考虑使用自助式的测试去满足那些重复出现的需求。在特定的情况下，他们也可以利用供应商提供的平台去使用统一的费率进行众包，进而取代原来按项目分别付款的形式。

在作出决定之前，项目的所有方面和目标都需要十分明确。通过了解项目所有者的专业知识，在关于自助服务模式与托管服务的利弊方面，众包测试的供应商之后可以为他们提出宝贵意见。

关键词：大众资源 众包测试 众包 软件测试 测评鸟公司（Testbirds）

TWOAPPROACHES TO CROWDSOURCED SOFTWARE TESTING

Editor's Note: Thefollowing comes to us from Markus Steinhauser, co-founder and head ofcommunications at Testbirds,a crowdsourced software testing company. Steinhauser discusses thedifferences between managed and self-service crowdsourced testing solutions.You can find out the latest news about the company on Twitter @Testbirds.

The spectrum of crowdsourcing projects and providers has seen a tremendousincrease in recent years. You can now fund your product, build and test thewebsite to sell that product, and then translate the site to conquer othermarkets – all using the power of the crowd.

Depending on the task, some platforms offer access to their crowd througha self-service model, a managed service by the provider, or both. But which isbetter? Well, it depends!

Different types of crowdsourcing use different business models for theirservice and even within the same field, these might vary. The following articletherefore concentrates on this question using the case of crowdtesting, whichutilizes actual consumers as well as experts to test software.

Let’s assume that a company is in the process of developing a new app anddecides to use crowdtesting as part of their quality assurance. The productmanager of this company wants to know two things:

1.       What does mytarget group think about the app?

2.       Does it workon the multitude of devices out there, especially iOS and Android?

First, the test needs to be set up. Part of this includes defining thescope of the project, selecting a specific target group and defining questionsthat need to be answered.

In order for a crowdtesting project to be successful, detailed project managementduring the test is crucial. Testers can work remotely from their homes, thetrain, or even their local coffee shop. They are asked to document their workthrough written documentation, screenshots, or screencasts. This is importantmainly for two reasons:

1.       Since thetesters are not in-house, documentation is the only way to ensure that there issufficient test coverage. Was everything that was part of the initial testreally tested?

2.       If issues arefound, they have to be traceable and reproducible. How did the user get to thispoint and exactly what is the issue?

Once all the participating testers have done their work, the results needto be aggregated and analyzed. Depending on the focus of the project, theresources required for each step may vary. This is usually time-consuming evenfor smaller tests.

The product manager needs to decide whether to use a managed crowdtestingservice or to do the management within his own team. While a managed servicewill save considerable time and manpower, it is more expensive than using aself-service. With the latter, the setup, project management and evaluation allneeds to be done internally.

It is impossible to say which option is better without taking all specificdetails of the project into consideration. There are, however, some generalguidelines that apply. Single setups require the most resources, because allthe parameters need to be defined first. In contrast, recurring projectsrequire less time to set up each time. This is especially relevant if the releasecycles of a software are happening at frequent intervals, for example severaltimes each month. In addition, managing a crowd of people who are workingremotely requires a lot of experience in order to achieve high-quality results.

Consequently, the following aspects can be part of the decision process inbetween managed and self-service:

1.       Isdocumentation necessary?

2.       Do we haveexperience in our team setting up projects?

3.       Are we ableto manage them?

4.       Do we havethe resources to do all of this internally?

5.       Does thebudget allow for a managed test?

Taking all these aspects into consideration, it usually makes sense tostart out with at least one managed crowdtesting project. It is then possibleto compare the effort and cost with a test done in self-service. A typicalscenario would be that a company tries out crowdtesting with a managed serviceat the beginning. If it turns out to be a long-term option, the service has tobe integrated into internal processes, which can also be a challenge. From theexperience gained through working together in a managed framework, the companycan think about switching to a self-service model to cover recurring testingneeds internally. In certain cases, they can also use a flat rate access to thecrowd though the provider’s platform instead of paying per project.

Before making a decision, all aspects and goals of a project need to bedefined. Through their expertise, crowdtesting providers can then offervaluable advice on the pros and cons of a self-service model versus a managedservice.

Tags: crowd sourcing, crowdsourced testing, Crowdsourcing, software testing, testbirds

www.crowdsourcing.org/

http://www.crowdsourcing.org/editorial/two-approaches-to-crowdsourced-software-testing/28918

Markus Steinhauser