Automation Project Management and Automation Architecture

                Automation Project Management and Automation Architecture

 

                                                   By

                                             M.N. Alam

                                          IMI Systems Inc.

                                             Dallas , Tx

 

 

Introduction

 

Numerous white papers and articles are written on test automation. Most are well researched and present a wide range of views about the benefits of using automation. Some of the articles are the collection of individual experiences (some are more painful than others) to make the automation work for them. In this paper, I would like to discuss automation from project management and architecture standpoint and hope this may help others to establish a process that fit their testing environment.

 

Since a great deal of important information is already available on cost/benefit analysis of automation, I will not attempt to engage in further discussion on this. But I will say that it is very important to do a cost/benefit analysis of the test cases that require automating. James Hancock wrote:   “When to Automate Testing - A Cost-Benefit Analysis” which is an excellent article on this subject.

 

Along with the cost/benefit analysis, detail planning must be completed. I concur with the experts that a great deal of planning is necessary before jumping onto this "automation boat".   If omitted, the testing journey could be very rough and possibly cause the fatality of the project. I was involved in a project where a good deal of time was spent debugging existing test scripts because minor changes were made to the Application Under Test (AUT). A small change in the application should not require a great deal of maintenance. I like to discuss the followings and will try to shed some light on the subject.

 

- Automation Project Management 

- Automation Architecture

 

Automation Project Management

 

Since automation is like a development environment, it should be treated like one. Keeping this in mind, let’s discuss some of the criteria of putting together an automation project team. At this point, let’s assume that the cost/benefit analysis is completed and the pay back period has been determined. Since this is a Quality Assurance (QA) project, it is recommended that the project leader comes from QA or understands QA and automation. I will discuss later in this section the danger (so to speak) of having a project leader with very little or no knowledge of  Q.A. ( particularly test automation). Once the project leader is selected, the selections of team members follow.

 

 Below are typical skills necessary for the members of an automation team.

 

- Development (Programming) skill

- Knowledge of the test script language (familiar with C syntax if you are using Mercury automated test tools WinRunner and XRunner; familiar with VB syntax if you are using SQA suite products)

- Ability to write and execute a test plan.

 

Let’s talk about these skill sets in some detail. Many test automation tool vendors will like you to believe that using their tool is so easy that a person with no programming skill should be able to do automation in no time. They are of course referring to Record/Playback feature of their test tool. In our discussion, let’s assume that Record/Playback is not a viable option of automation and I think we all know why (lack of reusable code). Record/Playback is acceptable if you are planning on using the script for a very short period of time. I personally think that it is nice to have this feature, but I will not use it unless I have exhausted all other means. This leaves us with the option to write test scripts programmatically. Familiarity with the test script language is also important. There is a learning curve for someone who does not have a prior experience with a particular test tool. Finally, an automation engineer will have to have prior experience of writing and executing test cases, because automation is no more than automating the existing manual test cases.

 

It is also important to select an automation lead person from the members. A lead person should have prior experience with the tool. If any of the members do not have prior experience, then my suggestion would be to hire a consultant who has an extensive scripting experience (no Record/Playback) with that particular test tool. A word of caution while selecting a consultant to be your automation lead - ask a lot of questions about their previous projects. For example, what were some of the problems they had to face and how did they get around it; what kind (if any) of architecture was in place? If you don’t have enough knowledge about the product or automation, get someone who does, because this lead person can make or break your project. It is a good idea to hire someone from a reputable consulting firm even if it costs more money. You probably discovered during the cost/benefit analysis that the biggest part of the automation cost is labor and not the product. It is worth paying big bucks to consultants to set up automation the right way. The alternative is poor setup with no architecture and no standards, which causes a lot maintenance headache down the road, turnover in the automation team, and eventually the abandonment of the automation. This is one of the dangers of having a project leader with little or no knowledge of test automation.

 

After selecting the project leader and the team members, our team is in place. At this point we are ready to do the design.

 

 

 

 

 

 

 

Automation Architecture

 

 “Improving the Maintainability of Automated Test Suites” by Cem Kaner is an excellent article where he talks about two different architectures, Data-driven architecture and Framework-based architecture as two viable options for automation. My design supports, for most part, the Framework-based architecture. The followings are the components of this architecture,

 

- Main Routine

- Environment Setup

- Screens Navigation

- Test Data

 

Main Routine

 

I like to call this the “Engine” of the architecture, because it drives the body. Here the engineers will organize the steps of execution. I will come back to this and explain what I mean by organization of steps. Here you can do the necessary environment setups, open output file/files, load and unload compiled modules. At the end, close the output file/files.

 

Environment Setup

 

In this section, you can set up the necessary system variables (speed, fonts etc.). Declaring global variables can also be a part of the environment set up. You can also call Common_Lib (a common library of reusable functions) in this section. The functions in the common library are reused over and over again by everybody in the automation team throughout the existence of automation. This can be dynamic and additional user defined functions can be added to the library as needed.

 

Example of reusable functions:

 

- Opening and closing of the output files

- Drag and drop of an object (location parameter driven - change the parameter according to the location of the object need to be dragged and dropped)

- Status Check (location parameter driven - change the parameter according to the location of the status bar on the screen)

- Single mouse Click (location parameter driven - change the parameter according to the location of the objects need to be clicked)

- Double mouse click (location parameter driven - change the parameter according to the location of the objects need to be double clicked).

 

 

 

 

 

 

Screens Navigation

 

 The purpose of this section is to navigate through a screen using a number of user-defined functions. It can be done many different ways and it is up to the engineers to decide the best way for their environment. In most cases, engineers write one function per screen. I prefer writing a number of small and simple functions for each field on the screen instead of writing one big function per screen.

 

For example, let’s assume we have a screen called “New Customer” with the following edit fields and push buttons:

 

- Customer name

- Street address

- City

- State

- Zip

- Telephone number

- Enter

- Cancel

 

I would write functions for each edit field and a single mouse click function for push buttons to navigate through the “New Customer” screen.

 

Below are some examples (Test script language for WinRunner and XRunner),

 

# Inserts customer name

function Cust_Name(in cust_name)

{

 Type_Text(cust_name);

 return;

}

 

# Inserts customer address(Street, City, state and zip)

function Cust_Street(in st_addr)

{

 Type_Text(st_addr);

 return;

}

 

function Cust_City(in city)

{

 Type_Text(city);

 return;

}

 

function Cust_State(in state)

{

 Type_Text(state);

 return;

}

 

function Cust_Zip(in zip)

{

 Type_Text(zip);

 return;

}

 

# Inserts customer telephone number

function Tel_Phone(in ph_no)

{

 Type_Text(ph_no);

 return;

}

 

# Clicks button (Right or left mouse click)

function Click_Button(in loc1, in loc2, in loc3, in button)

{

 if (!button)

  button = 0;

move_locator_abs (loc1, loc2, loc3);

if (button==0)

  click("Left");

else if (button==1)

 click("Right");

 return;

}

 

There are two advantages to use this over writing one function per screen. First, these are small and simple which makes maintenance easy and quick. But the biggest advantage is the code is reusable. If you have other screens that require customer name, address and telephone number, you should be able to use the same code with different test data.

 

 

Test data

 

This is a separate test script contains test data which are used to populate the edit fields. These are one-dimensional arrays. Below are some examples:

 

public  test_data []                    = {

            "Cust_Name"               = {"John Doe"},

            "Cust_Street"                = {" 1234 Main street "},

            "Cust_City"                  = { " Dallas "},

            "Cust_State"                 = {"Tx"},

            "Cust_Zip"                    = {"75240"},

            "Tel_Phone"                 = {"972-221-1212"}

            };

 

You may have a small or a large number of test data depending upon the nature of the test you are performing. It is better to have a large number of test data than a large number of test scripts because it is easier to maintain test data than test scripts.

 

Automation is easy if you have to navigate through the same screen with different test data. In our example above, we can easily automate a test case that checks to see the maximum number of customers can be created (boundary test). In this case we will use the same code over and over with a different set of test data. But in most cases, it requires the use of different set of code with a different set of test data. This is where the organization of steps comes in.

 

The following scenario may help you to understand,

Let’s say in order to automate a test case (create a customer) you need to navigate through three different screens with different edit fields and they are in following order:

 

- New Customer (Customer name, Street address, City, State, Zip, Telephone)

- Contact Information (Contact name, Street address, City, State, Zip, Telephone)

- Bank information (Bank name, Street address, City, State, Zip, Telephone)

 

First of all we need a number of Test Data for these edit fields. We will follow above example (one-dimensional arrays) to set up the test data.

We then need to write a number of user defined functions to navigate through these screens (Screens Navigation). You may want to either write a single function for each screen or write a number of small functions for each edit field in the screen. I found the later more useful.

Test data and all the functions written in Screens Navigation are compiled modules. The advantage of using compiled modules over regular test scripts is that execution of compiled modules is faster. But loading too many compiled modules also causes performance degradation. So, it is a good idea to unload each compiled module after the execution of that module.

The organization is usually done in Main Routine. In Main Routine we will,

-         Call the test script to set up the environment

-         Load compiled modules

-         Open output files

-         Call the first test script (let’s call it ‘Create_Customer’)

If no error run New_Customer(1) à indexed (pointing to a particular set of test data) from ‘Create_Customer’ test script.

If no error run Contact_Information(2) à indexed (pointing to another set of test data) from ‘New_Customer’ test script.

If no error run Bank_Information(3) à indexed (pointing to another set of test data) from ‘Contact_Information’ test script.

-         Unload Common library

-         Unload compiled modules

-         Close output files

 

. One could make the following arrangement in Main Routine,

-         Call the test script to set up the environment

-         Load compiled modules

-         Open output files

-         Call the first test script (Create_Customer)

-         If no error

-         {

            Run New_Customer(1)

            If no error

               {

                     Run Contact_Information(2)

                     If no error

                        {

                             Run Bank_Information(3)

                        }

                }

            }

 

-         Unload Common library

-         Unload compiled modules

-         Close output files

 

In both cases if there is an error, print the appropriate error message to the output file and exit out of that particular routine. While writing results to an output file, it probably is a good idea to have enough information (eg: Test case number, Test case objective, Objective of a particular step etc.) so that in case of a “Fail” status you will know exactly where to look in your test scripts. This helps a great deal in debugging the test scripts.

 

 

 

Conclusion

 

If this seems like a lot of work, it is. It takes time to set up architecture, but once the standard and the architecture is in place, the subsequent development time is considerably shorter. This architecture may not suit every environment, because it requires test engineers to have some degree of programming skill. I don’t know any effective automation process that does not require some degree of programming skill. The best part of this architecture is more reusability and less maintenance. Reusability and maintainability go hand in hand. The more code can be reused the less code need to be written and less the code, less the maintenance. One thing to remember is that reusability and complexity (complexity of code) also go hand in hand. The more reusability you want the more complex code you need to write. It is not always easy to maintain a complex set of code when you don’t have skilled personnel. The reality is you are probably not going to find a large number of testers who have sophisticated programming skill. So, it is safe to suggest writing simple code such that a person with little programming skill can maintain these scripts fairly easily.

 

 

NOTE: I have written this paper based on my experience with a number of automation projects I was associated with. There is no one way to do automation. Every project is different and every place has their own way of doing things. I have found this particular architecture seems to work provided you have skilled personnel and the management who understands and supports the process. It is not to say that other form of architecture or Record and Playback will not work. No matter what form of architecture you decide to work with make sure that most everybody in the group is comfortable with it, because it is the people (not the tool[tool helps]) makes a product as good as it can be(or in most cases as market allows it to be).