视觉测试工具箱

The most common use-case for visual testing is regression testing using baseline images. However, there are different aspects of visual testing worth discussing too. We will cover template matching (using OpenCV), layout testing (using Galen) and OCR (using Tesseract) and show how to seamlessly integrate these tools into existing Appium and Selenium tests. We use Java (and the Java-wrappers for OpenCV and Tesseract) but similar solutions can be achieved with other techstacks.

视觉测试最常见的用例是使用基准图像进行回归测试。 但是，视觉测试的不同方面也值得讨论。 我们将介绍模板匹配(使用OpenCV )，布局测试(使用Galen )和OCR(使用Tesseract )，并展示如何将这些工具无缝集成到现有的Appium和Selenium测试中。 我们使用Java(以及OpenCV和Tesseract的Java包装器)，但其他技术栈也可以实现类似的解决方案。

This is a companion-article for a lighnting talk given at Taqelah in Singapore in September 2020. For fully-functional demos and further details, please refer to www.justtestlah.qa.

这是 2020年9月在新加坡塔克拉(Taqelah) 进行 的绑架演讲 的配套文章 。 有关全功能演示和更多详细信息，请访问 www.justtestlah.qa 。

I hope this summary helps you choose the tools most impactful for your use-case and gives you some ideas on how to integrate them into your own toolbox!

希望本摘要有助于您选择对用例最有影响力的工具，并为您提供一些有关如何将其集成到自己的工具箱中的想法！

模板匹配 (Template matching)

The task template matching aims to solve is finding a given image (“template”) on the current screen.

匹配的任务模板旨在解决的是在当前屏幕上找到给定的图像(“模板”)。

Appium docs Appium文档

For mobile testing, Appium added this feature in the form of an -image locator strategy in their 1.9 release. The documentation can be found here and an early tutorial here. The idea is to pass a Base64 encoded String representation of the image to the WebDriver.

对于移动测试， Appium在其1.9版本中以-image locator策略的形式添加了此功能。 该文件可以发现这里和早期的教程在这里 。 这个想法是将图像的Base64编码的String表示形式传递给WebDriver。

Using the image locator, you can interact with the resulting element like any other WebElement . For example:

使用image定位器，您可以像其他任何WebElement一样与结果元素进行交互。 例如：

WebElement element = driver.findElementByImage(base64EncodedImageFile);
element.click();

or

要么

By image = MobileBy.image(base64EncodedImageFile);
new WebDriverWait(driver, 10).until(ExpectedConditions.presenceOfElementLocated(image)).click();

The approach the developers took is to add the functionality as part of the Appium server and use OpenCV (which becomes a dependency on the instance running the Appium server) to power the actual image recognition.

开发人员采取的方法是将功能添加为Appium服务器的一部分，并使用OpenCV (它成为运行Appium服务器的实例的依赖项)来增强实际的图像识别能力。

Interestingly, there is an extra round-trip between client and server as the flow looks like this:

有趣的是，客户端和服务器之间的流程如下所示：

1. Request a screenshot from the Appium server
   从Appium服务器请求屏幕截图

2. Send both the screenshot and the template to the Appium server for matching

   将屏幕截图和模板都发送到Appium服务器进行匹配

This does not feel 100% elegant, especially if we want to match multiple templates on the same screen.

这并不是100％优雅的感觉，尤其是当我们要在同一屏幕上匹配多个模板时。

When I first implemented template matching back in 2018 (unaware that, at the time, it was already being developed by the Appium team), I also chose OpenCV but instead ran it on the client-side. Using the OpenCV Java wrapper the gist of my code looked something like this:

当我在2018年首次实现模板匹配时 (当时还不知道Appium团队已经在开发模板匹配 )，我也选择了OpenCV，而是在客户端运行了它。 使用OpenCV Java包装器，我的代码要旨如下所示：

Mat result = new Mat(resultRows, resultCols, CvType.CV_32FC1);
Imgproc.matchTemplate(image, templ, result, Imgproc.TM_CCOEFF_NORMED);
MinMaxLocResult match = Core.minMaxLoc(result);
if (match.maxVal >= threshold) {
  // found
}

This approach doesn’t require the extra request to the Appium server mentioned above. In fact, it doesn’t require any functionality of the WebDriver other than the capability to take screenshots. It also works with both Selenium and Appium. That said, it also adds a dependency on OpenCV, this time, on the instance running the test execution.

这种方法不需要对上述Appium服务器的额外请求。 实际上，除了拍摄屏幕截图的功能外，它不需要WebDriver的任何功能。 它还可以与Selenium和Appium一起使用。 也就是说，这次，它还增加了对OpenCV依赖，即对运行测试执行的实例的依赖。

I wrapped both of the above approaches (client- and server-side execution) into a TemplateMatcher interface to show-case its usage (consider it a PoC).

我将以上两种方法( 客户端和服务器端执行)都包装到TemplateMatcher接口中，以展示其用法(将其视为PoC)。

You can find more details and examples here.

您可以在此处找到更多详细信息和示例。

布局测试 (Layout testing)

Another type of visual testing involves verifying the layout of a page or screen. You could do this by image comparison which, implicitly, also checks the layout. A simpler approach is using a specialized layout testing tool like Galen (in my opinion, one of the most underrated UI test frameworks).

另一种视觉测试类型涉及验证页面或屏幕的布局。 您可以通过图像比较来做到这一点，图像比较也隐式地检查了布局。 一种更简单的方法是使用像Galen这样的专用布局测试工具(在我看来，这是最被低估的UI测试框架之一)。

Galen uses specifications for each screen that define all (important) elements on a screen and their size and absolute or relative positions to each other.

Galen为每个屏幕使用规范，这些规范定义了屏幕上的所有(重要)元素以及它们的大小以及彼此之间的绝对或相对位置。

Let’s look at the Google search page as an example:

让我们以Google搜索页为例：

Google search page

谷歌搜索页面

We can represent it using the following spec:

我们可以使用以下规范表示它：

= Login =SEARCH_FIELD:
   below LOGO
   centered horizontally inside viewport
   visible
LOGO:
   above SEARCH_FIELD
   centered horizontally inside viewport
   width < 100% of SEARCH_FIELD/width
   visible
SEARCH_BUTTON:
   near LUCKY_BUTTON 20px left
   visible

Note, that the above uses the JustTestLah! framework’s syntax (referencing UI elements by a unique key defined in a YAML file of the page object). In pure Galen, these would need to be defined on the top of the spec file instead:

注意，上面使用了JustTestLah！ 框架的语法(通过在页面对象的YAML文件中定义的唯一键引用UI元素)。 在纯Galen中，这些需要在spec文件的顶部定义：

@objects
    LOGO          id        
    ...

There are different ways to execute these checks. What I prefer is a verify method as part of an abstract BasePage class:

有多种执行这些检查的方法。 我更喜欢将verify方法作为抽象BasePage类的一部分：

private T verify() {
  String baseName = this.getClass().getSimpleName();
  String baseFolder = this.getClass().getPackage().getName().replaceAll("\\.", File.separator);
  String specPath = baseFolder
              + File.separator
              + configuration.getPlatform()
              + File.separator
              + baseName
              + ".spec";galen.checkLayout(specPath, locators);
  return (T) this;
}

This way, we can easily call the verification from within the test whenever we first interact with a screen (btw, I use a similar approach to integrate Applitools for visual testing):

这样，每当我们第一次与屏幕交互时，我们都可以轻松地从测试中调用验证(顺便说一句，我使用类似的方法来集成Applitools进行视觉测试 )：

public class GoogleSteps extends BaseSteps {
  private GooglePage google;
  @Given("I am on the homepage")
  public void homepage() {google.verify().someAction().nextAction();
  }
}

光学字符识别 (OCR)

A different form of visual assertion is Optical Character Recognition, better known by its acronym OCR. This can be useful whenever, for whatever reason, text is rendered as an image and cannot be verified with the standard testing tools.

视觉断言的另一种形式是光学字符识别，其缩写为OCR。 每当由于某种原因将文本渲染为图像并且无法使用标准测试工具进行验证时，此功能将非常有用。

It might also be interesting for those who use Selenium for web scraping rather than testing as it is one of the counter measures website developers put into place to make this a little harder.

对于那些使用Selenium进行Web抓取而不是进行测试的用户来说，这可能也很有趣，因为这是网站开发人员采取的反措施之一，以使其变得更加困难。

We use Tesseract, an OCR tool originally developed by HP in the 1980s and currently sponsored by Google.

我们使用Tesseract (一种最初由HP在1980年代开发，目前由Google赞助的OCR工具)。

Our example is not the most realistic but it’s meant to showcase Tesseract’s power in detecting different types of fonts: We’ll verify that the Google logo actually spells out “Google”:

我们的示例不是最实际的示例，而是要展示Tesseract在检测不同类型的字体方面的强大功能：我们将验证Google徽标是否确实拼写出“ Google”：

public class GooglePage extends BasePage<GooglePage> {
  @Autowired private OCR ocr;
  ...
  public String getLogoText() {
    return ocr.getText($("LOGO"));
  }
}public class GoogleSteps extends BaseSteps {
  private GooglePage google;
  ...
  @Then("the Google logo shows the correct text")
  public void checkLogo() {
    assertThat(google.getLogoText()).isEqualTo("Google");
  }
}

The OCR service used looks like this:

使用的OCR服务如下所示：

public class OCR implements qa.justtestlah.stubs.OCR {
  private Logger LOG = LoggerFactory.getLogger(OCR.class);
  private TakesScreenshot driver;
  private Tesseract ocr;
  @Autowired
  public OCR(Tesseract ocr) {
    this.ocr = ocr;
  }
  /**
   * @param element {@link WebElement} element to perform OCR on
   * @return recognised text of the element
   */
  public String getText(WebElement element) {
    return getText(element.getScreenshotAs(OutputType.FILE));
  }
  /** @return all text recognised on the screen */
  public String getText() {
    return getText(getScreenshot());
  }
  private String getText(File file) {
    LOG.info("Peforming OCR on file {}", file);
    try {
      return ocr.doOCR(file).trim();
    } catch (TesseractException exception) {
      LOG.warn("Error performing OCR", exception);
      return null;
    }
  }
  /**
   * Usage:
   *
   * <pre>
   * new OCR().withDriver(driver);
   * </pre>
   *
   * @param driver {@link WebDriver} to use for capturing screenshots
   * @return this
   */
  public OCR withDriver(WebDriver driver) {
    this.driver = (TakesScreenshot) driver;
    return this;
  }
  /**
   * Usage:
   *
   * <pre>
   * new OCR().withDriver(driver);
   * </pre>
   *
   * @param driver {@link TakesScreenshot} to use for capturing screenshots
   * @return this
   */
  public OCR withDriver(TakesScreenshot driver) {
    this.driver = driver;
    return this;
  }
  private File getScreenshot() {
    return driver.getScreenshotAs(OutputType.FILE);
  }
  @Override
  public void setDriver(WebDriver driver) {
    this.driver = (TakesScreenshot) driver;
  }
}

This requires Tesseract installed on the instance running the test. For the full source code and demos, check out the JustTestLah! testframework.

这要求在运行测试的实例上安装Tesseract。 有关完整的源代码和演示，请查看JustTestLah！ 测试框架。

翻译自: https://medium.com/@mart.schneider/visual-testing-toolbox-f742d35d62bd