自动化测试tips

 

why automation?

We have to read a lot of documentation every day. Bad experience made us suspicious about the correctness of any external documentation. We don't really like writing documentation ourselves because we know that it is only a matter of time until it is out of sync with the system and all our effort will be wasted. If the documentation is done via automated tests, it is assured to be up to date, making it a reliable source of information. We are much more motivated to invest our time for this.

 

 

We advise to do manual checks for everything that cannot break after it worked once. Everything else should be automated if the automation can be done without excessive costs.

 

测试用例文档管理有一个问题。如2.5版本对2.4版本添加了新功能，我们可能会针对新功能写一个用例文档，但是有部分是对2.4版本的修改。我们是应该 将2.4版本的所有用例全部放到2.5版本并进行更新吗？理想状态下应该全部更新文档并将原有用例全部跑一遍。但要是手工来完成回归工作量太大。要是将所 有的测试用例组织成自动化用例而不是用例文档就简单多了。这是我们为什么用webtest, selenium等自动化测试工具的原因之一。

 

data driven or record/replay

Functional testing can be classified as being either data driven or record/replay. Canoo WebTest follows the data driven approach. Record/replay is appealing at first, because you can create a lot of tests in a short time. A proxy logs what pages you request and stores the results. It can then replay the requests and compare the results against the stored ones. You typically have to tweak this procedure to tell the program what parts of the page are expected to change. The actual date and time are the most obvious examples. Every small change to your webapp causes a lot of these tests to fail. These failures must be manually processed to separate the "real" failures from the "false negatives". Doing this is almost as tedious and error prone as the manual testing and is therefore discouraged e.g. by the Automated Testing Specialists group.

Record/Playback is the least cost-effective method of automating test cases.

Keith Zambelich
Co-author of the WinRunner ATS Toolkit.

Functional tests and unit tests.

Functional tests do not replace unit tests. They work together hand-in-hand. Consider the following example: Your Webapp displays an html table that is filled with data from the database. The maximum number of rows should be 20 and if there is more data available, a link should be shown that points to the page that contains the next 20 entries. If there is no data, no table should be shown, but the message "sorry, no data". We would test this with a) no data b) one row c) 5 rows d) exactly 20 rows e) 21 rows f) 40 rows g) 41 rows. A naive way of testing this would be to manipulate the database (maybe by using an administration servlet that we can call via "invoke") prior to calling the page. But this is not only very slow but also a little dangerous. What if two tests run concurrently against the same test database? They will mutually destroy their test setup. What if the test run breaks? Is the state of the test database rolled back? The whole job is difficult to do for a functional test, but easy and quick for a unit test. A unit test can easily call the table rendering and assert the proper "paging" without even having a database! What is left for the functional test is to assert that the table rendering logic was called at all.

 

（from http://webtest.canoo.com/webtest/manual/whitepaper.html )

 

 

-----------------------------------------------------------------------------------------------------------------------------

 

 

一、 在 web 自动化测试思路上，我觉得 selenium 非常成熟。

selenium 能够很好地模拟人工测试 web 系统的过程。即人工能在页面完成的操作 selenium 基本能够通过录制来完成，人能够完成的验证 selenium 也能够很好地来验证。实际上 selenium 定义的一种语言，这种语言由很多命令组成。命令分成三种风格动作（ Actions ） , 获取因子（ Accessors ） 和断言（ Assertions ）。 Actions 实际上是模拟人工在页面上所进行的操作，如输入表单，点击链接等。 Accessors 用来获取页面上某部分的数据，它可以通过 id, xpath 来获取页面上任何地方的数据。 Assertions 用来检查实际所取得的结果与期望结果是否一致。

但是在实现程度上，无论是对浏览器的支持，还是和其它语言集成，或是缺陷方面，我觉得这个产品还不是太成熟。这可能是由于页面技术发展太快的原因。所以我们还有很大的对这个工具改善并应用于我们日常测试的空间。

 

二、通过在性能管理平台中运用 selenium 来测试，我总结出在设计 testcase 时应该注意以下事项：

 

1.       每个 testcase 不能包括多个功能点的测试。因为这种情况下如果前一个功能点 fail 会造成后面的 case 无法正常运行（特别是在某一个动作无法执行时）。所以，建议一个 testcase 只测试一个功能点 。如新增用户一个 case ，查询用户一个 case 。而将用户操作相关的 case 组成一个 suite.

2.       在需求描述，设计等阶段就应该明确细化功能点。这样有利行测试用例的设计和录制。

3.       case 之间应该保持独立性。在理想状态下，我们期望所有的 testcase 完全独立。如在每次执行 testcase 前初始化数据库状态。这样我们就能单独选择部分 case 来执行。

4.       先录制查询 case, 再录制 update case. 初始化数据库状态的成本比较高，如果执行每个 case 都初始化一次会让 case 运行很慢。我在测试性能管理系统时做了这种尝试。我们先备份一个初始数据库，这中间存放着一定量的数据，这些数据能满足测试查询功能。我们录制 case 时可以先录制查询的 case. 以后在跑 case 时先基于这个数据库跑查询的 case, 这样我们能基于一个初始数据库来完成查询的测试。在录制 update 的 case 时，我们尽量让它们相互独立，即后一个的状态不会依赖于前一个。我们希望 selenium 能确定 testcase 的执行顺序，这样我们就能确保先运行查询的 case 。

---------------------------------------------------------------------------------------------------------------------------------