Purpose：
Build up the Auto Test framework with the CI process;
With the Help of Auto Test, the KPI of DEV Team can increased with 50%;
Background:
In Q3 there are lots of new feature will be deliver in WeChat Store, the work load is over 2 QA Engineers' capacity;
Current Status:
For each sprint, there are about 30 points should be completed,
DEV engineers can only put 5% time for UT task, only the happy route can be covered;
QA engineer only has 8 working days for design cases, prepare test data and execution for about 10 stories;
And last 2 days for regression test;
Each point can only get about 2 hours for each test task, there are :
| QA Task | Weight | Note |
|---|---|---|
| Analysis Story | 10% | Human task |
| Plan Schedule | 5% | Human task |
| Design cases | 20% | Human task |
| Data prepare | 10% | can be Auto |
| ENV deploy | 10% | can be Auto |
| Execution | 40% | can be Auto |
| Report | 5% | can be Auto |
| ~65% | can be Auto |
DEV & QA engineers can force on core work in every sprint;
RISK: Currently, the DEV & QA Backend env are not stable, and non-independent, in some day, we can't login target page;
So the scripts coding are blocked;
Target:
The repeat Execution(40% workload), Data Prepare(10% workload), Env Deploy(10% workload) and part of report(5% workload) can be automatic,
the QA Engineer can focus on core work as Analysis and Design cases;
Because the Auto test tool can't design test cases as human for new story,
so we want to make the regression test can be executed automatic (should be ready in 2018/8);
After that we want to make the data prepare and execution for new feature can been automatic (should be ready in early 2018/9);
Evolutionary Process of the Framework:
| Step | Description | Note |
|---|---|---|
| 1 | Code the test scripts with some hard code when the elements or pages in product is untestable; | Done with CYPRESS; |
| 2 | Encapsulation for the repeatable step, parameter, variable and configuration; | Done with CYPRESS; |
WHAT SHOULD TEAM DO:
Considering the main workload(40%) is execution, the highest priority task is making execution been automatic;
Building up phase:
|
| Description | Status |
|---|---|---|
| 1 | Define the test scope which MUST/SHOULD/Can't be covered by Auto Test; | Done, E2E |
| 2 | Build up the Auto test Demo in Local Env for Test Framework verification and select the Framework with high applicability and generality, low maintain cost; | Done, need Manager confirm |
| 3 | Make the E2E test be covered by Auto Test; Auto Engineer complete the test scripts for E2E scenarios and guide documents for installation, execution and code scripts; | POC Done, Release Engineer are preparing for Pipeline |
| 4 | Modify the code for adding the test attitude, then expand the Framework usage to more scope in E2E/Regression test; | ToDo, Auto engineer is collecting the information; |
Stable phase:
|
| Description | Status |
|---|---|---|
| 1 | Auto Engineer code more scripts in Regression scope | Doing |
| 2 | Training the QA for how to mock the API response with auto test tool for reducing Data Prepare for IT; | ToDo, Auto engineer should make some example |
| 3 | Auto engineer help the QA to use the auto tool in daily work and training the scripts coding; | ToDo |
| 4 | When analysis story, Auto Engineer should discuss with DEV for add unique attitude for element location; | ToDo |
| 5 | When there are enough scripts are ready, DEV team can make UT with lower cost and more scenario; | ToDo |
There are 2 kinds of ways for Preparing test data(10% workload):
1. MOCK:
There are 2 way for MOCK Backend API;
a) CYPRESS support this, samples as;
it('cy.route() - route responses to matching requests', () => {
// https://on.cypress.io/route
let message = 'whoa, this comment does not exist in real response'
cy.server()
// Stub a response to PUT comments/ ****
cy.route({
method: 'PUT',
url: 'comments/*',
status: 404,
response: { error: message },
delay: 500,
}).as('putComment')
// we have code that puts a comment when
// the button is clicked in scripts.js
cy.get('.network-put').click()
cy.wait('@putComment')
// our 404 statusCode logic in scripts.js executed
cy.get('.network-put-comment').should('contain', message)
cy.get('.some-element-appear-with-response-code-404').should('be.visible',{timeout: 1000)})
})
I can add the lines as "flag4MOCKstart","flag4MOCKend" to around the MOCK declaration and implementation in scripts,
and use other maintain scripts to annotation/un-annotation the test scripts when we run it without/with MOCK;
b) RAP from Alibaba support the same MOCK, Backend Team(Micheal Shen) will release the feature before 2018/8/13,
but FrontEnd team need maintain the related data by they-self; The maintain cost need to be estimated;
2. Set the real data of account/address/history/etc. in DEV/QA/PreLIVE Environment; The backend tool HAWKEYE already cover this feature, I will make some scripts for typical scenarios;
After read some documents from eBay, the 70~80% cases of component/model layer have been covered by auto test;
In E2E test scope, the manual test still cover more than 50% cases; Especially in the Exploratory Testing, human work are irreplaceable.
Then Why E2E test:
We want to cover the Happy Path for Basic Flow, then we can know which part of project are not available for Auto test;
Because we hope all test for single Sprint can be finished in 4 hours with high quality and enough coverage;
But the GUI test is not stable when involved with Mobile and App, so GUI can't be the main point, just the main Business Flow - E2E can be covered;
|
| WeChat Store drived by Real Mobile | H5 Page without Auth Limitation running in WeChat SDK | E2E test with Tools as cypress |
|---|---|---|---|
Advantage | Real customer angel view, involve all the environments : network, Mobile, Mobile OS, WeChat APP, etc. | Normal browser as Chrome/FF can handle the H5 page test, selenium can handle the test quickly; | Quick RUN, working with Backend API directly; |
Defect | WeChat internal browser block the Web View handle; | Without WeChat, the result is incomplete; WeChat Auth block the H5 page running in Chrome/FF; | Without WeChat, the result is incomplete; Scripts based on JS, can't be sync with the existing HAWKEYE; |
Plan | Appium drive the webview | Selenium drive the Web View; | CYPRESS or MOCHA; |
Status | Blocked, only location can be used to tap the screen, this kind of "Blind" operation can't check the flow step; | Trying the non-Auth web login, but the Order/Bag pages are still blocked; | Waiting for H5 page isolated with WeChat version ready, but the Order/Bag pages are still blocked; |
Compare several E2E Test Framework:
TOOL | Element operation | Variables & Aliases | assertion | Test Cases Running | Condtional Testing | ENV config | Dependency | Platform | Script Maintain Cost | LOG & Project Organization | Code Coverage |
|
| |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CYPRESS | Actionability， Visibility， Disability， Animations， Covering， Scrolling，Forcing, etc. Suitable for Mobile; Can get element by Text Content from human's view OR CSS SELECTOR; | yes | should-end cases; expect-end cases; timeout(before/after)-auto repeat then fail and end cases; | not support parallel; Only back end/network down will cause skip all following cases; Without script grammar check before execution; | yes | yes, simply K-V format; 2 Layer with overrule mechanism; we can use this as NATIVE Product Branch/ENV controller. | NO | MAC,LINUX,WINDOWS | Midle, simple assert, flex var, smart screen with case step tree, video; | Yes, but stored in 3rd, we can get private space with pay; Support 3rd reporter as JUnit for TeamCity/Jenkins; | Not Supported yet | | ||
NightWatch | Actionability, Visibility, Disability; CSS/JS related, suitable for Web Page; Only support attributes/property of JS/CSS OR XPath to get the element; | yes | assert-end case; verify-log & continue case; expect-end cases; timeout(before/after)-auto repeat then fail and end cases; | support parallel; Exception/Failure will cause skip all following cases; Without script grammar check before execution; | yes | yes, JSON format(); config for Product Branch/ENV should be maintain in the same file with different Domain; | Selenium for non-Chrome | MAC,LINUX,WINDOWS | High, simple assert, complicated var, screen without case step, no video; | Support 3rd reporter as JUnit for TeamCity/Jenkins; | Not Supported yet | | ||
| MACACA
| Based on web driver and selenium, so ID, CLASSNAME, XPATH, CSS SELECTOR, etc. | yes, based which Language you selected: JS, JAVA, Python | Default JUnit, you can expand with other framework such as TestNG OR PyTest; | Dependent with which JUnit or other framework such as TestNG OR PyTest; | yes, based which Language you selected: JS, JAVA, Python | based which Language you selected: JS, JAVA, Python | Web Driver | MAC,LINUX,WINDOWS | High, simple assert, complicated var, screen without case step, no video; | Support 3rd reporter as JUnit for TeamCity/Jenkins; | Not Supported yet |
| ||
What should ENV Platform involve:
Browser(headless mode also need the core),
CI Platform (Jenkins OR Teamcity),
Test Tool (CYPRESS, Nightwatch),
Test Scripts,
ENV Config file, etc.
How the E2E Test tool work with CI pipeline:
In this image, the UT are executed with DEV team with their own code, but we hope the Auto Framework can help them to finish this task in future;
UAT mean the manual test and the remaining part of the Regression test;
| Feature Layer | INPUT | OUTPUT | Description |
|---|---|---|---|
| 1 | Merge Request | Gitlab event | Someone in DEV team push some change as Merge Request to the FEATURE Branch; |
| 2 | Gitlab event | Build event | CI Server (Jenkins + Gitlab Merge Request Builder) configured with GitLab Project Path, Builder detail, test start command, etc.; When the MR arrived, the Build task will start; |
| 3 | product package | in location | If the build completed successfully, the product will be stored in Library; |
| 4 | Test scripts | Test tool and scripts ready | CI Task for build up the TEST Tool will install the Test Tool and deliver the ENV config files and scripts to the position under the Project Path; |
| 5 | command line for start sample test | sample test result | Start the sample test with default scripts with sample Product in test Tool to confirm the deliver, ENV config and dependency works well; |
| 6 | product package in speical location | product deliver to the target location | CI Task for build up the TEST ENV will get the package from Library and deliver to the TEST ENV; GIT Clone to the Project Path; |
| 7 | command line for start E2E test | Test log, etc. | CI Task for execution the test will start the local Node for WeChat Store and modify the test ENV configuration file, then start the Test Tool with selected scripts; |
| 8 | Test report | Merge job complete | If the Test cases are all passed, the FEATURE Branch will include the code change with the last MR; Otherwise, the MR will be rejected and the DEV will be alarmed; |
| 9 | Success notice to DEV team | Then the DEV Team can push the code change to TRUNK now; |
| Sprint Layer | INPUT | OUTPUT | Description |
|---|---|---|---|
| 1 | Merge Request | Gitlab event | Someone in DEV team push some change as Merge Request to the TRUNK Branch; |
| 2 | Gitlab event | Build event | CI Server (Jenkins + Gitlab Merge Request Builder) configured with GitLab Project Path, Builder detail, test start command, etc.; When the MR arrived, the Build task will start; |
| 3 | product package | Stored in location | If the build completed successfully, the product will be stored in Library; |
| 4 | Test scripts | Test tool and scripts ready | CI Task for build up the TEST Tool will install the Test Tool and deliver the ENV config files and scripts to the position under the Project Path; |
| 5 | command line for start sample test | sample test result | Start the sample test with default scripts with sample Product in test Tool to confirm the deliver, ENV config and dependency works well; |
| 6 | product package in speical location | product deliver to the target location | CI Task for build up the TEST ENV will get the package from Library and deliver to the TEST ENV; GIT Clone to the Project Path; |
| 7 | command line for start E2E test | Test log, etc. | CI Task for execution the test will start the local Node for WeChat Store and modify the test ENV configuration file, then start the Test Tool with selected scripts; |
| 8 | Test report | Merge job complete | If the Test cases are all passed, the TRUNK Branch will include the code change with the last MR; Otherwise, the MR will be rejected and the DEV will be alarmed; |
| 9 | Success Test report | Push code change to MASTER | Then the CI Task will push the TRUNK to the MASTER; |
| 10 | Build script | Deliver to Pre-LIVE | The product will be built with MASTER and deliver to the Pre-LIVE ENV; |
| 11 | Deliver success notice | Manual test result | Manual test can be executed after CI send the notice; |
| 12 | Then if any other steps in the pipeline, continue until the end; | ||
| 13 | If Manual test passed, MASTER will be marked as “Ready to Deliver”; The Release Engineer and Head of DEV will receive the Notice; | ||
| 14 | Then the Deliver to LIVE can start; |
Test Platform File structura:
1.For more exact element location, DEV Engineer need add the special attribute for identify,What should DEV modify the Code for better Selecting Elements with CYPRESS:
Anti-Pattern: Using highly brittle selectors that are subject to change.
Without attribute for auto test
Best Practice: Use data-* attributes to provide context to your selectors and insulate them from CSS or JS changes.
With attribute for auto test
Every test you write will include selectors for elements. To save yourself a lot of headaches, you should write selectors that are resilient to changes.
Oftentimes we see users run into problems targeting their elements because:
Oftentimes we see users run into problems targeting their elements because:
- Your application may use dynamic classes or ID’s that change
- Your selectors break from development changes to CSS styles or JS behavior
Luckily, it is very easy to avoid both of these problems.
- Don’t target elements based on CSS attributes such as:
id,class,tag - Don’t target elements that may change their
textContent - Add
data-*attributes to make it easy to target elements
How It Works:
Given a button that we want to interact with:
<button id="main" class="btn btn-large" data-cy="submit">Submit</button>
Let’s investigate how we could target it:
| Selector | Recommended | Notes |
|---|---|---|
|
| Never | Worst - too generic, no context. |
cy.get('.btn.btn-large').click() | Never | Bad. Coupled to styling. Highly subject to change. |
cy.get('#main').click() | Sparingly | Better. But still coupled to styling or JS event listeners. |
cy.contains('Submit').click() | Depends | Much better. But still coupled to text content that may change. |
cy.get('[data-cy=submit]').click() | Always | Best. Insulated from all changes. |
Targeting the element above by tag, class or id is very volatile and highly subject to change. You may swap out the element, you may refactor CSS and update ID’s, or you may add or remove classes that affect the style of the element.
Instead, adding the data-cy attribute to the element gives us a targeted selector that’s only used for testing.
The data-cy attribute will not change from CSS style or JS behavioral changes, meaning it’s not coupled to the behavior or styling of an element.
Additionally, it makes it clear to everyone that this element is used directly by test code.
Did you know?The Selector Playground automatically follows these best practices.
When determining an unique selector it will automatically prefer elements with:
data-cydata-testdata-testid
2.The CYPRESS already support the dashboard for result presentation in organizations, next step is adding the item into TEAMCITY.
3.How to make CYPRESS scripts work smoothly with the different point between Branch/Version/Language:
There are 2 layer of Environment configuration files: cypress.json & cypress.env.json, the duplicated item in cypress.env.json will overwrite the same item in cypress.json;
In cypress.json, we define all the location str for related elements in the target Code Branch ;
In cypress.env.json, we define Test Platform Node IP, Port, User account info for different Test Env(as DEV, QA, Pre-Live, Live), different Text Content of element in different language;
And some element which work with different Test Env;
4.Performance monitoring can get the sample data with ST in pre-Live ENV, and compare with the real data in LIVE ENV;
But we still need Active measuring instrument in “Payment”, etc. before the WeChat Auth can be overruled.
AS Google Test Model, what should TEAM do & the Benefit:
| Role | Input | Output | Feature | Scenario | TC STEP | TC DATA | Benefit from Auto Test |
|---|---|---|---|---|---|---|---|
| PO | Requirement | Story | Define | Simple Define | Clear view of Scenario | ||
| DEV | Story | Design | Analysis, Implement | Define, Implement | Don't need to prepare the UI Test, just need maintain the Unit Test | ||
| QA(Feature Owner) | Requirement | TC | Analysis | Analysis | Define | Define | Low miantain cost for TC/Data, more effort for Intellectual work |
| QA(Executor) | TC | test result | Understand, simple analysis | Understand, simple analysis | Execute | Execute | Less time for manual test, focus on analysis the test report/log |
| Above Role don't need to change thedaily work significantly | Only QA need prepare the Test case and Test Data in new Formate; | ||||||
| Auto Test Engineer | Test Plan | Framework, script template | Understand | Understand, map to script group | map to Key word(every Key word including 1~several steps) | map to Data Group |
扫一扫
664
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
