Software Tools. Build vs Buy

This is a fairly long article that consolidates the common reasons associated with either buying or building software tools. The article also lists the factors to be considered when estimating costs involved with either of the decisions - to build or buy.

Software Tools are of various types and can be used either by a specific function or cross-functionally across the organization. In the case of software testing, Tools is a broad term that encompasses automation and allied testing tools used as part of testing software. These could be defect tracking tools, reporting tools or software test management tools. When an organization or a group within a larger organization decides that it needs a testing tool, a likely question that can crop up would be – can we build it in-house or do we buy a commercial testing tool?

Not every group will have to make this choice. Sometimes, the decision's been made, probably by a central entity who has ordained that a specific tool be used. However, this is a common question encountered by most organizations when they need software tools. Both of the choices above have some common characteristics.

• Require investment of time, effort and money
• Have implications that are long lasting
• Shape the way we do our jobs

While making choices on whether to build or buy software tools, we must weigh our options, not rely on assumptions and make well informed decisions. The repercussions of a choice can be long lasting. There is no such thing as an option that can easily be rolled back in case the choice turns out to be less than expected. In this article, let us examine both the build and buy choices and some of the popular reasons and assumptions around them.

Building a software tool may often seem to be the easier route to take. There does not seem to be a “steep” upfront cost involved as in buying a testing tool. Organizations reason that they can put something together quickly using their available resources. Here are some reasons put forth for building software tools in-house.

Reason: We just need something for the short term, a temporary solution, something quick for the time being. We are hard pressed for time and cannot spend time evaluating a lot of choices. Let us quickly put together something now and later explore more longer term solutions.

We must realize that when we decide to build something, even if it is supposed to be a temporary solution, we will be investing money, time and resources to create, use and maintain the testing tool. The software tools we employ to facilitate our work, will soon mold our work habits around them. Sooner  than later, the software tool will become a part of your regular job and exercise a certain degree of inertia towards any change. When the tool falls short in certain areas, you will (have to) come up with workarounds to continue using the software tool and over time it becomes harder to change established work patterns and software tools learnt.

Lets take the example of a software test automation tool / harness that your in-house resources put together. Once you have reached a certain level of automation using the software tool, how easy will it be to scrap the testing tool and opt for a new one ? If your software tool is lacking in certain features, would you spend the time and effort to enhance the existing tool or would you ditch it to go for an external software tool that offers those extra features ?

In my experience, I have come across software tools that were home-grown and met the basic requirements of the group. However, the software tool paled in comparison to some of the externally available software tools in terms of features and options available. Since the group had invested enough time and resources on the tool, it became hard to let go or even migrate to another software tool without incurring significant additional investments of money, time and resources – which they were reluctant to do.

In essence, you cannot afford to justify a decision based on it's perceived or assumed short-term need. Every decision carries a set of ramifications with it and it is rare that you can easily back out with minimal damage, if things do not work out as expected.

Reason: We have the talent (technical / functional) and can understand our needs better than any external vendor. We can develop a software tool that best matches our requirements,  better than what a commercial software tool vendor may be able to provide.

Whom are we kidding ? Unless we are talking of really simple software tools, the likelihood of internal groups being able to achieve the above and meeting requirements to the desired extent is when the particular internal group that is expected to be building the software tool – has software tools building as it's main responsibility, has demonstrated expertise in successfully building software tools, has the time / money / resources / skills to commit to building a software tool that matches all requirements and can maintain the same on an on-going basis.

While it is easy to commit to developing software tools, it is often hard to cater to all of the requirements and commit to sustain/maintain the software tool for the longer term. Most commercial software tool vendors have specialized groups dedicated to developing specific software tools and have made large investments towards building tools. These software tool vendors have built the expertise to address challenges and have the resources available to support the software tool on an on-going basis. Usually, when in-house software tools are developed, it is the case that a set of engineers are assigned to work on the software tool along with their regular tasks. Divided attention is hardly beneficial. If the software tool developers have pressure to work on other tasks, they can easily lose focus and attempt to cobble up a software tool that at best, barely meets requirements. When developers are assigned to work full time on creating software tools, unless they have the aptitude and skills required – they can give up soon or even look at cutting/watering down requirements to complete their assignment and move on to other projects. Internal groups often face the challenge of maintaining and enhancing the software tools that are developed. Another element to consider while choosing to develop software tools in house is software testing. Like any software product that is developed, the software tool too must go through the same standard development processes and test efforts to be useful, which can prove to be a significant added investment.

So, unless you are willing to commit the resource (time, money, effort) to develop, test and sustain a software tool, it might not be a great idea to build it in-house.

Reason: Building a software tool in-house saves a lot of money when compared to purchasing the tool externally. Since we are using our available resources, things would work out much cheaper than buying a software tool.

How easy is it to estimate costs accurately for a development project ? Try estimating the costs for your planned software tool development project. A fairly detailed list of items to consider when estimating the cost of building a software tool, is provided towards the end of this article.

Include the cost of resources involved in software development (gather requirements, analyze, design, implement, localize if needed, migrate data), documentation, software testing (unit testing, functional testing, performance testing, etc), training users, performing on-going maintenance and enhancements of the software tool. A common perception when developing in-house software tools is that users of the software tool will be more forgiving and adjusting of the tool's failings and they would have reduced expectations from the software tool. The fact is – users not only demand similar standards as external software tools, they generally have more (rather than less) expectations from the in-house developed software tool. After all, the developers are in-house and accessible. We can always request for customizations and changes whenever we want.

Cost calculations must also consider the opportunity cost involved with having both Developers & Management staff working on the software tool development project vs working on other revenue opportunities. We must not under-estimate the cost of software tool maintenance. In many cases, on-going maintenance costs (defect tracking, regular updates / patch releases, support) can turn out to be a significant recurring investment similar to the licensing costs involved when buying a software tool.

Basing the software tool build vs buy decision purely on cost is not really a feasible option. What may seem to be a less expensive choice could turn out to be a less than feasible alternative.

Reason:  We should build it since we just need a simple solution which our engineers should be able to quickly put together; our software testing requirements aren't too complex; we know what we need and figure it should not take much time or effort to build a  tool.

Organizations find it tempting to put together or build simple software tools to address a specific problem or area without considering, at the risk of making a cliched statement, the “larger picture”. If your software tool is not easily enhance-able/maintain-able and you are limited to the functionality you built-in initially, you can be sure that sooner than later, your needs will outgrow what your software tool offers. The tool will soon hinder than aid your efforts. I have seen different groups create such “simple” software tools to address specific pain points. Over time, various such “simple” software tools were created and put to use by different groups. Since little thought was given to integrating the various software tools or making them work together, the proliferation of these “simple & standalone” software tools posed a challenge to the organization's productivity.

From experience, such software tools tend to get discarded after some amount of usage and a suitable replacement is found. However, there are cases where these software tools continue to be used, since the organization has already invested significantly in the existing software tool and may not be willing to incur the additional costs associated with a migration to a new software tool.  In either case, the organization has already incurred costs on the simple software tool and will incur more costs to procure and move to a replacement tool. It would be better to consider the larger picture when making software tool decisions. Look beyond the confines of your group; examine (best) practices from other groups and software tools they use; explore any possible changes to your current processes/additional requirements for the software tool you intend to build and consider if your software tool can handle them.

It need not just be new or changed requirements that can make your software tool redundant. Simple scalability issues can cause your simple software tool to pose complex issues. If your tool is designed to handle the needs of your current group, consider if the same tool can handle the requirements of a larger group - will your software tool then scale or crawl ?

Does your software tool consider all audiences who might potentially use/want/need to use it? Some software tools are created to solve pain points that a section of engineers may encounter. Their usage, output and reporting are often put together to provide the basic information that those engineers would need. However, when the same software tool needs to provide information to a wider audience or someone in the business side of things, a different usage and reporting paradigm emerges. If the software tool cannot meet the needs of the different audiences, current as well as potential future ones, it can again be a drag. What if the products you are working on changes significantly so that the tool also needs to be changed to handle newer operational/functional needs?  Can your software tool adapt to changes?

Reason: Software Tools we build can be fully customized when compared to commercial software testing tools

This is one argument that's hard to beat. It is true that software tools you build will be more customizable than anything you can get externally.

However, that is just one part of the equation. We need to look at the emphasis on customization. Beyond a reasonable degree, the need for customization may be viewed as a factor emerging from an organization's fluid and yet-to-be-clearly-defined set of methods and processes that makes defining a set of requirements for a software tool, hard. The preference in such as situation is to go with a software tool that can theoretically be customized infinitely !

While the ability to customize is very good and useful, it must be viewed in the light of costs involved and time to deploy the software tool. Here's a very simplistic example – let us assume that your organization wants a GUI test automation tool. One possibility is to buy a commercial tool such as SilkTest or WinRunner. The other option is to build a software tool that is similar to these software tools but can be customized to any degree you would like.

Here, you need to consider the following.

• The costs involved – you might end up spending huge sums of money (many times more than it would cost to buy licenses) to build such a software tool and much more to maintain it for as long as your organization needs to use it
• The time to deploy – assuming your organization is willing to invest the money, an important factor to consider would be the time to deploy the tool. Building a software tool from scratch takes time, lots of it in fact. Factor this in while making your choice

Having looked at some of the common reasons stated in favor of building software tools, let us examine some of the common reasons stated in favor of buying software tools.

Reason: Buying will lead to faster deployment of the tool; a software tool that is purchased will be faster to deploy.

Yes, when you consider that buying a software tool cuts out the development costs involved in building it. Yet, the extent of quickness in deployment depends on your particular situation. 

There still needs to be due diligence towards selecting the right software tool. Once that is done and a software tool is procured, any needed customizations are to be performed. The software tool then needs to be setup and configured, following which a detailed migration exercise has to be performed to move away from the existing software tools and users need to be trained to work with the new tool. Time taken to accomplish the various tasks involved in deployment is also dependent on the extent of complexity. A factor to consider would be the human acceptance of the software tool. Just because a new software tool is procured and setup, users & others associated with the roll-out will not automatically accept it. When a new software tool replaces existing software tools that are being used widely, you can expect some degree of resistance to the new tool. Comparisons are inevitable and in the initial days there are bound to be comments on the new tool vis-a-vis the existing software tools. Despite any shortcomings of an existing tool, users can be reluctant to adopting a new software tool; more so, if they were involved in or had a stake in developing the existing tool. Customization, migration and acceptance are chief areas that can consume a lot of time.

I have come across teams that were reluctant (resistant would be more like it) to adopting a new software tool. The new software tool could potentially help overcome drawbacks which the existing software tool had. But folks had gotten used to the current software tool, warts and all of it and were pushing back wider implementation of the new tool. They would find ways to delay implementation and after several deadlines for deployment had passed, the organization settled for a dual systems approach (use the old software tool for some tasks and the newer one for some others). In some organizations, the transition to the new tool does happen but at a much later date than what was targeted.

The above factors will influence the amount of time it takes to deploy the software tool you buy. Will it be faster than building a software tool ? There is no definite answer. In some cases, it might be faster to build a software tool while in other situations, buying a software tool and deploying it will be quicker.

Reason: The software tool we buy will be maintenance-free – after all that's what the vendor claimed. All we need to do is to buy the software tool, pay up the annual maintenance fees and bingo, the vendor will take care of the rest.

Wishful thinking ! There's hardly any software tool of reasonable complexity that is maintenance-free. Maintenance includes the effort required from the organization's side towards setting up the tool, configuring it and regularly performing administrative activities. 

If customizations have been performed, we can expect additional maintenance requirements. This is more so when updates/patches/upgrades to the software tool are received. Applying an update or upgrading to a later version could involve significant work including backing-up and re-applying any customizations, testing and re-deploying the tool. Even when there are no customizations, moving to a later version can pose compatibility issues and requires testing before deploying widely. While the vendor personnel may offer support, the organization will also need its own resources to work with the vendor on software tool maintenance activities and support.

Summing up, there are situations where it makes more sense to build your software tool, while there are situations where it might be better to go out and buy a software tool and customize it if need be. Sometimes, it can be hard to find software tools externally that meet your specific requirements. With software tools you buy, you will need to adapt to the tool and the way it works. With a software tool that you build, you can integrate it with your current way of working.

Irrespective of the choice you make – to build or buy – there are aspects that you must consider while estimating costs involved with either decision. Given below, is a listing to help in calculating costs associated with building and buying software tools.

1. Costs to consider while deciding to build a software tool in-house.

• Administrative costs involved in Defining the problem
  • Collecting requirements
  • Development environment and method selection
  • Developing and administering training for users
  • Managing the project and people involved

• Production costs involved in
  • Planning the Tool development project
  • Architecture and HLD (High Level Design)
  • Detailed Design
  • Coding / Implementation
    • Client / Web
    • Server / Back-end infrastructure
    • Module integration
    • Migration
  • Software Testing
    • Unit
    • Integration
    • System
    • Performance
    • Acceptance
  • Debugging
  • Documentation
  • Deployment

• Support Costs involved in
  • Ongoing maintenance activity
  • Developing and Delivering upgrades/patches
  • Defect tracking and reporting
  • Ongoing support activities, such as running a help desk or similar mechanism

In addition, the opportunity costs involved must be considered. The cost of time spent by employees (technical and business staff) along with any delays in schedules must be taken into account.

2. Costs to consider while deciding to buy a software tool.

• Administrative costs involved in
  • Defining the problem
  • Collecting requirements
  • Evaluation of various tools, Proof-of-Concept and Selection
  • Managing Vendors
  • User training

• Procurement costs involved in
  • Cost of licenses
  • Customizations
  • Cost of tool deployment – setup, configuration, integration
  • Tool testing – integration, compatibility, performance, acceptance
  • Cost of migration

• Support Costs involved in
  • Annual Maintenance costs
  • Costs involved to upgrade across versions
  • Additional internal support costs

Also, as in the case of building a software tool, the opportunity costs involved must be considered. The cost of time spent by employees (technical and business staff) in the above activities along with any delays in schedules must be taken into account.

Stay tuned for more upcoming posts on various facets of Software Testing, QA and Development.