How to decide what tests to automateEdit

There are four distinct factors that goes into deciding which manual tests ought to be automated:

  • How long it takes to run the test manually.
    • Automating tests that are more difficult to run manually increase the value of the automation.
      • Example: Make sure the software will install on a new OS.
  • What is the cost of automating the test.
    • Those tests that are harder to automate, may not be the best choice for automation
      • Example: Pulling the network cable from the computer would not be trivial to automate.
  • How many times is the test expected to run in a project.
    • Those tests that are going to be executed a lot because of their critical nature, would gain from being automated.
      • Example: Ensuring that runtime data is being read and written correctly.
  • How often is the incidental execution of the test area in the day-to-day testing and usage of the product.
    • If the code won't execute incidentally as part of other tests automating that test would ensure that the test is executed more often and prevent us from releasing a bad bug inadvertantly
      • Example: Partial migration

In the end, the above factors should be taken into account when determining which tests should be automated, and in what order, and contribute the total cost savings of SW Test during the project run.

How to determine the percentage of tests which are automatedEdit

Traditionally, the % of number of test cases are used to determine which tests are automated. A lot of times, this isn't a very good measure because the point of automating test is to save cost. Thus, we should look at what is the gain from automation and use a metric that takes into account the cost and value of each test that's to be automated.

Effort in Manual DaysEdit

Given a product with a set of test cases, if none of the test cases were automated, than the effort to run those tests is called the "Effort in Manual Days", (EMD)

Effort in Actual DaysEdit

Given a product with a set of test cases, some of which are automated, and some are not, the total effort required to run the automated tests and the manual tests is called the "Effort in Actual Days", (EAD)

EAD/EMD CalculationEdit

The ratio of EAD to EMD is a good way to calculate the percentage of automated testing in the system. For instance, given a product that will take 100 man days to test manually (EMD), but has some automated tests, enabling the test group to run through all thests in only 60 man days (EAD), then EMD/EAD would yield 60%. Note that this is only remotely related to the percentage of test cases automated (since the length of time that takes to run a test cases varies so greatly from test to test)