“If you automate a mess, you get an automated mess.” (Rod Michael)

Virtually all software development projects today contain an element of test automation. If an initiative is not already underway chances are there is one under consideration. The software engineering groups are generally convinced of the benefits – greater test coverage achieved faster being the most apparent. That being said as the business owner how does the economic argument in favour of test automation stack up?

Before that- is there any value, though, in getting into ROI discussions? The point being that in all software development, testing is a given. Despite that has there ever been any meaningful attempt to try & define the ROI of the testing effort? The fact is that the risk of sending an untested product into the market is just too great for the business to bear so an ROI argument is generally not necessary. So let us start with the base assumption that a certain cost for testing is inevitable – this would likely be the cost allocation made towards the efforts the testers put in to achieve the desired quality standards before the product or software release.

Let us consider the various cost elements of a test automation initiative.

• You will likely have to spend for the software that you use to build your test automation suite or framework. Since we are talking about the lifetime cost of ownership, even assuming you choose to go open-source there will likely be a cost associated with ongoing support, maintenance, and possible customization.
• Then there will be an allocation towards the effort it takes to use these tools and build your test automation suite. The skills needed to build test automation are generally different and in many cases more expensive, than those needed for doing manual testing so there is that to consider as well.
•  Remember that the automation suite is also a product that will need ongoing maintenance and support – the cost for that effort should also be factored into the total cost of ownership.
• Some effort will also be expended in running the test automation suite and that has a cost too.

The cost of the automation effort is a combination of all of these and the cost to be considered in any, even notional, ROI discussions would be that, less the cost allocation that would have been made towards a traditional manual testing effort. Are you with us so far?

Having dealt with the, I in ROI let’s move towards the R. We spoke of the principal benefits from automating your software testing – more coverage and faster testing. What are the possible economic benefits that can accrue from these, though?

First, let’s look at the greater coverage. The conventional wisdom is that greater code coverage will uncover more defects and potentially earlier in the development cycle.

• The earlier in the cycle defects are detected, theoretically, the easier they are to fix. There are less things to check for in previous versions, regressions are simpler and so on. There are models out there that show how much more it costs to correct defects later in development cycles – that is a good place to start if you want to find how much money you are saving.
• Can you put a cost to a better quality product reaching the customer? Perhaps not but there are possible ways to allocate notional costs to when the reverse happens. Customer-found defects cost money in replacements, recalls and under-warrantee fixes not to mention the loss of reputation and potential loss of costumers and market share. Greater coverage that catches such defects before such an eventuality occurs is a well-made argument.

What is the value of testing faster? It seems reasonable to assume that faster testing will shorten development cycle time and, therefore, allow for faster releases, more product iterations within a shorter time and so on. How does that look on the balance sheet?

• A faster release translates into a faster market entry. Addressing the available market opportunity on or before schedule has undeniable business benefit – first-mover advantage, early market validation or feedback and an opportunity to occupy unoccupied market positions can all be achieved and models exist to allocate, in one way or the other, monetary values to these benefits.
• Automation would allow software development teams to spend less time on testing – effort saved is money earned of-course. Then there is the opportunity cost – potentially the time saved on testing could be spent profitably doing something else of value.

One way to look at this argument is that it is too complex to look at purely from a “cash in, cash out” kind of viewpoint so you should consider only the engineering value. The variables are many but isn’t that in many ways true of software testing in general. Remember the words of James Bach,

“Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous.”