by Adam Bender This article was adapted from a GoogleTesting on the Toilet (TotT) episode. You can download a printer-friendlyversion of this TotT episode and post it in your office. An end-to-end test tests your entire system from one end to the other, treatingeverything in between as a black box. End-to-end tests can catch bugsthat manifest across your entire system. In addition to unitand integration tests, they are a critical part of a balanced testing diet,providing confidence about the health of your system in a near production state.Unfortunately, end-to-end tests are slower, more flaky, and moreexpensive to maintain than unit or integration tests. Considercarefully whether an end-to-end test is warranted, and if so, how best to writeone. Let's consider how an end-to-end test might work for the following "login flow":
In order to be cost effective, an end-to-end test should focus onaspects of your system that cannot be reliably evaluated with smallertests, such as resource allocation, concurrency issues and APIcompatibility. More specifically:
  • For each important use case, there should be one correspondingend-to-end test. This should include one test for each important classof error. The goal is the keep your total end-to-end count low.
  • Be prepared to allocate at least one week a quarter per test to keepyour end-to-end tests stable in the face of issues like slow and flakydependencies or minor UI changes.
  • Focus your efforts on verifying overall system behavior instead ofspecific implementation details; for example, when testing loginbehavior, verify that the process succeeds independent of the exact messages orvisual layouts, which may change frequently.
  • Make your end-to-end test easy to debug by providing anoverview-level log file, documenting common test failure modes, and preservingall relevant system state information (e.g.: screenshots, database snapshots,etc.).
End-to-end tests also come with some important caveats:
  • System components that are owned by other teams may change unexpectedly, andbreak your tests. This increases overall maintenance cost, but can highlightincompatible changes
  • It may be more difficult to make an end-to-end test fullyhermetic; leftover test data may alter future tests and/or productionsystems. Where possible keep your test data ephemeral.
  • An end-to-end test often necessitates multiple test doubles (fakes or stubs)for underlying dependencies; they can, however, have a high maintenance burdenas they drift from the real implementations over time.