几种iOS 自动化测试框架对比

目的是引入自动化测试，帮助QA减轻iOS回归测试需要的人力和时间的负担。

网上搜了一番，挑了几个对比了一下，分别记录了优缺点。最终选定的是KIF，公司其他同事推荐的，貌似Zappos也是用它。

Automator in XCode

Pros:

1. support testing others (like memory) at the same time.

2. Save screen shot at the same time.

Cons:

1. lack of document. The "UI Automation JavaScript Reference" lacks many details, like specific methods we can call for some elements.
2. Unstable or not working on several methods on Simulator, for example, double tape, pinch.
3. Can't handle some UIs, for example, our search view object on iPhone just cannot be seen in code and then can't be handled, so how can we auto test the search feature? (对于比较复杂的图片层次，比如像我们iPhone app的Search界面是覆盖在原有的上面，则logElementTree()没有显示这个界面的元素，相信也没有什么途径能操作它们。很可能需要改动代码，使得这个view处在某个view controller下，层次对了才能发现。)

http://appium.io/

Pros: 

1. Test code can be written in many languages, so that JAVA engineer can write tests for both Android and iOS app.

Cons: 

1. Lack of documents and hard to setup. (It took me a while to make this Appium.app running, and will take me more time to learn how to write tests)
2. It uses UIAutomation, may have same cons as the Automation Instruments, for example, unstable on simulator, or can't find some UI elements on screen (haven't confirmed yet)
3. At least for iOS test, the test codes are not along with app source codes, so we have to build app and start Appium to test and go back to modify app source codes to help test, and back and forth, inconvenient and take time.
4. A test contains too many steps: write tests, build app, start appium and point to app address, start tests.

http://www.gorillalogic.com/monkeytalk

Note, following pros and cons are based on Monkey Talk Community Edition document, not based on real experience.

Pros: 

1. Separated good IDE to organize and add/edit test cases.
2. Support record test steps which is good for QA beginner.

Cons: 

1. Have to install an agent in app for test. (not big issue though, since iOS can have different schemes.)
2. Test coding languages are either monkey talk specific or javascript, not good for iOS or Android developers.
3. Test codes are organized in different places other than same project, not good for maintenance.
4. Monkey talk operates App through the agent in app via the network, I'm afraid that the agent can only do basic operations, not those sophisticated ones like checking element absence, or counting table view total rows.

http://calaba.sh

Note, the following pros and cons are based on document, not based on real experiences.

Pros: 

1. Easy setup? It can help us setup a scheme for calabash only automatically.

Cons: 

1. Have to deal with Ruby, and learn special languages to write test cases, not good for either iOS or Android developers.
2. It uses UIAutomation, that's not very good, see cons in Automation Instruments for details.
3. Similar to monkey talk, it uses http to communicate between calabash and the agent. Depends on the protocol, we may not have too complicated operation.
4. Test cases are not in same project as app source codes.

https://github.com/kif-framework/KIF

Pros: 

1. Can save screen shot
2. Easy to write test and have many helpful methods
3. Can integrate with CI (haven't tried yet)
4. Can run directly in Xcode, no need separated server or separated IDE/Project.
5. Active development

Cons: 

1. just small issues, for some complicated operations or validations, we may need to understand more in the source codes to write our own helpful methods.