Sberbank在线iOS测试

application's code which is used by millions of iOS users and about the difficult path that our team finished in order to achieve stable code. 应用程序代码方面的实践经验，以及我们团队为获得稳定的代码而完成的艰难道路。

Let’s imagine: the developers successfully convinced themselves and the business of the need to cover the code with tests. Over time, in the project were created more than a dozen thousand unit- and more than a thousand UI-tests. Such a large test base araised several problems, and we successfully found the solution for them.

想象一下：开发人员成功说服了自己和企业，使他们有必要用测试覆盖代码。 随着时间的流逝，在该项目中创建了超过一万个单元测试和一千多个UI测试。 如此庞大的测试基础引起了一些问题，我们成功找到了解决方案。

In the first part of the article, we will get acquainted with the difficulties that arise while working with clean (non-integration) unit-tests, in the second part we will consider UI-tests closely.

在本文的第一部分，我们将了解在使用干净的(非集成)单元测试时出现的困难，在第二部分，我们将仔细考虑UI测试。

In an ideal world, with unchanged source code, unit tests should always show the same result regardless of the number and sequence of it’s starts and constantly falling tests should never pass through the Continuous Integration(CI) server barrier.

在理想的情况下，使用不变的源代码，无论测试的开始数量和顺序如何，单元测试都应始终显示相同的结果，并且不断下降的测试永远都不应通过持续集成(CI)服务器障碍。

In reality we may encounter the fact that the same unit-test will show either a positive or a negative result — “blinking” test. The reason for this behavior is a bad test implementation. Moreover, such test can pass CI with successful status, and later it will begin to fall on other people's Pull Request (PR). In such situation, there is a desire to disable this test or play trigger build-roulette and run the CI again. However, this approach is anti-productive, it undermines the credibility of tests and loads CI with meaningless work.

实际上，我们可能会遇到这样一个事实，即同一单元测试将显示阳性或阴性结果-“闪烁”测试。 此行为的原因是测试实现不正确。 而且，这样的测试可以通过具有成功状态的CI，然后它将开始落在其他人的请求(PR)上。 在这种情况下，希望禁用此测试或播放触发构建轮盘并再次运行CI。 但是，这种方法生产效率低下，破坏了测试的可信度，并给CI带来了毫无意义的工作。

This issue was highlighted yesteryear at Apple's WWDC international conference:

昨天在苹果公司的WWDC国际会议上强调了这个问题：

• This session discusses about parallel testing, individual target coverage analysis and test run process.

  本节讨论并行测试，单个目标覆盖率分析和测试运行过程。

• Here Apple narrates about testing network requests, mocking, testing notifications and the performance of tests.

  苹果在这里讲述有关测试网络请求，模拟，测试通知和测试性能的信息。

单元测试 (Unit tests)

To struggle blinking tests we use the following sequence of actions:

为了应对闪烁测试，我们使用以下操作序列：

0. Evaluate test’s code quality according to basic criteria: isolation, mocks’ correctness, etc. We follow the rule: with a blinking test, we change the test’s code, and never the code that we test.

0.根据基本标准评估测试的代码质量：隔离性，模拟的正确性等。我们遵循以下规则：测试闪烁时，我们更改测试的代码，而从不测试代码。

If this does not help, proceed as follows:

如果这样做没有帮助，请按照下列步骤操作：

1. Fix and reproduce the conditions under which the test falls;

1.修正并复制测试所处的条件；

2. Find the reason why it falls;

2.找到它掉下来的原因；

3. Change the test’s code;

3.更改测试代码；

4. Go to the first step and check whether the cause of the fall has been eliminated.

4.转到第一步，检查跌倒原因是否已消除。

重现秋天 (Reproduce the fall)

The simplest and most obvious option to reproduce the blinking fail is to run a problem test on the same version of iOS and on the same device, and usually in this case the test is successful! We have a thought: “Everything works for me locally, I will restart the build on CI”. But in fact, the problem has not been solved, and the test continues to fall on the build of another developer.

重现闪烁失败的最简单，最明显的选择是在同一版本的iOS和同一设备上运行问题测试，通常在这种情况下，测试成功！ 我们有一个想法：“一切都对我本地有效，我将在CI上重新启动构建”。 但实际上，该问题尚未解决，测试仍取决于其他开发人员。

Therefore, at the next step, you need to run locally all the unit tests of the application to identify the potential affect of one test on another. But even after this check, your test result may be positive, while the problem remains undetected.

因此，在下一步中，您需要在本地运行该应用程序的所有单元测试，以识别一个测试对另一个测试的潜在影响。 但是即使进行了此检查，您的测试结果仍可能是阳性的，而问题仍然未被发现。

So, if test run was successful and the expected fail was not caught, you can repeat the run many times.

因此，如果测试运行成功并且未捕获预期的失败，则可以多次重复运行。

To do this you need to run a loop in terminal with xcodebuild:

为此，您需要使用xcodebuild在终端中运行循环：

#! /bin/sh
x=0
while [ $x -le 100 ];
    do xcodebuild -configuration Debug -scheme "TargetScheme" -workspace App.wcworkspace -sdk iphonesimulator -destination "platfrom=iOS Simulator, OS=11.3, name=iPhone 7" test >> "report.txt";
    x=$(( $x +1 ));
done

This should be enough to reproduce the fall and move on to the next step — identifying the cause of the recorded fall.

这应该足以重现跌倒并继续进行下一步-确定记录下来的跌倒原因。

测试的跌落原因和可能的解决方案 (Test’s fall reasons and possible solutions)

Let’s dive deeper into the main causes of unit-tests blinking, which you may encounter, tools to identify problems, and possible solutions.

让我们更深入地探讨单元测试闪烁的主要原因，您可能会遇到这些问题，确定问题的工具以及可能的解决方案。

There are three main groups of reasons for blinking tests:

闪烁测试的原因主要有三组：

测试隔离不良 (Bad test isolation)

By isolation we mean a special case of encapsulation: a mechanism that allows restricting access of some program components to others.

隔离是指封装的一种特殊情况：一种允许限制某些程序组件对其他程序组件的访问的机制。

Isolation of the environment has an important role for the purity of the test, nothing should affect the tested entities. Particular attention should be paid to tests that are aimed at checking the code and use global state entities, such as: global variables, Keychain, Network, CoreData, Singleton, NSUserDefaults and so on.

隔离环境对于测试的纯度具有重要的作用，没有任何东西应该影响被测试的实体。 应特别注意旨在检查代码并使用全局状态实体的测试，例如：全局变量，钥匙串，网络，CoreData，Singleton，NSUserDefaults等。

Imagine that while creating a test environment, a global state is set, which is implicitly used in another test code. In such case, the test may start to “blink” — due to the fact that, depending on the sequence of tests, two situations can arise — when the global state is set and when not set.

想象一下，在创建测试环境时，设置了一个全局状态，该状态在另一个测试代码中隐式使用。 在这种情况下，测试可能会开始“闪烁”，这是因为，根据测试的顺序，可能会出现两种情况：设置全局状态和不设置全局状态。

Oftenly, the described dependencies are implicit, so you may accidentally forget to set / reset such global states.

通常，所描述的依赖项是隐式的，因此您可能会意外地忘记设置/重置此类全局状态。

To make the dependencies clearly visible, you can use the principle of Dependency Injection (DI): pass the dependency through the parameters of the constructor, or a property of an object. This will make it easy to substitute mock dependencies instead of a real object.

为了使依赖关系清晰可见，可以使用依赖关系注入(DI)的原理：通过构造函数的参数或对象的属性传递依赖关系。 这将使替换模拟依赖项而不是真实对象变得容易。

异步呼叫 (Asynchronous calls)

All unit tests are performed synchronously. The difficulty of testing asynchrony arises due to the call of the test method that “freezes” awaiting of the unit-test’s scope completion. The result will be a stable fail.

所有单元测试均同步执行。 异步测试的困难是由于调用“冻结”等待单元测试范围完成的测试方法而引起的。 结果将是稳定的失败。

//act
	[self.testService loadImageFromUrl:@"www.google.ru" handler:^(UIImage * _Nullable image, NSError * _Nullable error) {
		//assert
		OCMVerify([cacheMock imageAtPath:OCMOCK_ANY]);
		OCMVerify([cacheMock dateOfFileAtPath:OCMOCK_ANY]);
		OCMVerify([imageMock new]);
		[imageMock stopMocking];
	}];
	
	[self waitInterval:0.2];

There are several approaches to test such a case:

有几种方法可以测试这种情况：

1. Run NSRunLoop
   运行NSRunLoop
2. waitForExpectationsWithTimeout
   waitForExpectationsWithTimeout

Both options require to specify an argument with a timeout. However, it cannot be guaranteed that the selected interval is sufficient. Locally, your test will pass, but on a heavily loaded CI there may not be enough power and it will fall — thus “blink” will appear.

这两个选项都需要指定一个带有超时的参数。 但是，不能保证所选间隔足够。 在本地，您的测试将通过，但是在负载较重的配置项上，可能没有足够的功率而将其降低，因此将出现“闪烁”。

Let’s imagine that we have some kind of data processing service. We want to verify that after receiving a response from the server, it transfers this data for further processing. To send requests via the network, the service uses the client to work with it. This case can be written asynchronously using a mock server to guarantee stable network responses.

假设我们有某种数据处理服务。 我们要验证的是，在收到服务器的响应后，它将传输此数据以进行进一步处理。 为了通过网络发送请求，该服务使用客户端进行处理。 可以使用模拟服务器异步编写此案例，以确保稳定的网络响应。

@interface Service : NSObject

@property (nonatomic, strong) id<APIClient> apiClient;

@end

@protocol APIClient <NSObject>

- (void)getDataWithCompletion:(void (^)(id responseJSONData))completion;

@end

- (void)testRequestAsync
{
  // arrange
    __auto_type service = [Service new];
    service.apiClient = [APIClient new];

    XCTestExpectation *expectation = [self expectationWithDescription:@"Request"];

    // act
    id receivedData = nil;
    [self.service receiveDataWithCompletion:^(id responseJSONData) {
        receivedData = responseJSONData;
        [expectation fulfill];
    }];

    [self waitForExpectationsWithTimeout:10 handler:^(NSError * _Nullable error) {
        expect(receivedData).notTo.beNil();
        expect(error).to.beNil();
    }];
}

But the synchronous version of the test will be more stable and will allow you to get rid of working with timeouts. For this we need a synchronous APIClient mock.

但是测试的同步版本将更加稳定，使您摆脱使用超时的麻烦。 为此，我们需要一个同步的APIClient模拟。

@interface APIClientMock : NSObject <APIClient>
@end

@implementation

- (void)getDataWithCompletion:(void (^)(id responseJSONData))completion
{
  __auto_type fakeData = @{ @"key" : @"value" };
  if (completion != nil)
  {
    completion(fakeData);
  }
}

@end

Then the test will look simpler and it will work stabler.

这样，测试看起来会更简单，并且工作会更稳定。

- (void)testRequestSync
{
  // arrange
    __auto_type service = [Service new];
    service.apiClient = [APIClientMock new];

    // act
    id receivedData = nil;
    [self.service receiveDataWithCompletion:^(id responseJSONData) {
        receivedData = responseJSONData;
    }];

    expect(receivedData).notTo.beNil();
    expect(error).to.beNil();
}

Asynchronous operation can be isolated by encapsulating to a separate entity, which can be tested independently. Other part of the logic should be tested synchronously. Using approach you will avoid most of the pitfalls brought by asynchrony.

可以通过封装到单独的实体中来隔离异步操作，该实体可以独立进行测试。 逻辑的其他部分应同步测试。 使用方法可以避免大多数异步带来的陷阱。

As an option, in the case of updating the UI layer from the background thread, you can check whether it’s the main thread and what will happen if we make a call from the test:

作为一种选择，在从后台线程更新UI层的情况下，您可以检查它是否是主线程以及如果我们从测试中进行调用会发生什么情况：

func performUIUpdate(using closure: @escaping () -> Void) {
    // If we are already on the main thread, execute the closure directly
    if Thread.isMainThread {
        closure()
    } else {
        DispatchQueue.main.async(execute: closure)
    }
}

For a detailed explanation, see the article by D. Sandell.

有关详细说明，请参阅D.Sandell 的文章 。

测试代码超出您的控制 (Testing code beyond your control)

Often we forget about the following things:

通常我们会忘记以下几件事：

• the implementation of the methods may depend on the localization of the application,
  方法的实现可能取决于应用程序的本地化，
• there are private methods in the SDK that can be called by framework classes,
  SDK中有一些私有方法可以由框架类调用，
• the implementation of the methods may depend on the version of the SDK.
  方法的实现可能取决于SDK的版本。

These cases bring some uncertainty in the process of writing and running tests. To avoid negative consequences, you need to run tests on all locales, on all versions of iOS supported by your application. Separately, it should be noted that there is no need to test code whose implementation is hidden from you.

这些情况在编写和运行测试的过程中带来了一些不确定性。 为避免负面影响，您需要在应用程序支持的所有iOS版本上的所有区域设置上运行测试。 另外，应注意，无需测试对您隐藏了其实现的代码。

Finally, we want to complete this part about automated testing of the Sberbank Online iOS application, dedicated to unit testing.

最后，我们要完成有关Sberbank Online iOS应用程序自动测试的这一部分，专门用于单元测试。

The article was written with @regno — Anton Vlasov, head of iOS development.

本文由@ regno撰写-iOS开发负责人Anton Vlasov。

翻译自: https://habr.com/en/company/sberbank/blog/511384/