Automated Testing ROI

Part 1. When is Automated Testing More Profitable Than Manual One?

What are the major objectives of automated testing? How can you bring the costs and number of running cycles down and coverage and effectiveness up? And finally, does automated testing always pay off better than manual testing?

Automated testing is sure not a сure-all solution. It often implies considerable expenses in the beginning which should be considered rather as a long-term investment into your project under development. But at some point in this ballgame this investment will start covering the expenses spent on manual testing.

Where exactly is this very point? How can you catch it?

To get this question answered, we may compare ROI (return on investment) in automated and manual testing and make our mind what is better for us.

For starters, we should determine the crucial parameters of our project.

Example 1: Basic Test Set Usage

Let’s fancy we have a regular task to work out a set of basic tests which we are going to perform in every new product build. Our goal is a minimal quality assessment and Non-GUI tests running.

For instance, assumptions for our project are:

• Daily builds and test runs (5 times a week)
• Manual tests take 20 days to design, 8 hours to run
• Only half the manual tests would be run on any given day (4 hours) with the other half run the following day
• Automated tests take 60 days to design and implement, automatically run (zero cost)
• Automation is done with batch scripts and integrated into the build process. Test server requires $1,500 in added hardware, with a useful life of 3 years
• Automated tests need to be maintained every 20 runs, 3 days of work required
• Periods of time (t) selected: 6 months (125 days) and 18 months (375 days)
• Average instinctools QA engineer cost for our client $14 per hour = $112 per day = $26880 per year

ROI (in 6 months) = $7000 / $6830 = 1.025 [about break even]
ROI (in 18 months) = $21000 / $11530 = 1.821 [~80% return]

As we see above, the investment made in an automated testing approach starts paying off already in the 6th month.

Example 2: the Full-Scale Set of GUI Tests

Let’s presume we have a goal to quality test the product functionality via GUI.

In this case our assumptions will look like the following:

• A new product with all new tests
• 1 people-years developing manual tests, 3 for automated tests
• 1 person maintenance after 1st year for automated tests
• 3 people full time running manual tests, 1 person for automated
• Fixed costs for automated tests (new hardware, tools licenses) of $15,000 with useful life of 3 years
• Periods of time (t) selected: 12 months (250 days) and 24 months (500 days)
• People cost $14 per hour = $112 per day = $26880 per year

ROI (in 12 months) = $53760 / $58760 = 0.915 [small loss]
ROI (in 24 months) = $107520 / $90640 = 1.186 [~20% return]

In this case, even one year later manual testing stays more profitable and effective.

The bottom line

We have specially reviewed 2 examples in order to show that sometimes ROI starts in the 6th month after the work went underway, sometimes even 1 year is not enough. The calculation above shows how much the assumptions and input parameters influence ROI.

We hope our examples and experience will help you calculate your ROI and build up more effective solutions.