The heart of Scrum is a Sprint, a time-box of one month or less during which a “Done”, useable, and potentially releasable product Increment is created.

So why not release every sprint?

Releasing more often is easier said than done. The release frequency of a product depends on the time it takes to test it. Changing a line of code is easy. Testing that change is a lot more difficult. And it’s usually a bad idea to release untested software.

Implementing test automation in an existing codebase can take serious effort. It does not only change the code, but it also changes the way of working. So where to get started?

Know why to automate!

In a waterfall project, testing is about discovering and validating new behavior of the system, documenting it and judging the correctness of that behavior.

Designing a system in a waterfall project takes a long time. First, functional design will be created, then you will need a detailed technical design. Such a process can take weeks or months. Then, it takes months to build it. That results in lots of new behavior being introduced to the software after the build phase has been completed. Afterward, the intended behavior needs to be validated. That’s a lot of checking and exploratory testing that needs doing and this too, is a time-consuming process.

In an Agile project, the development process looks pretty much the same. You’ll still need to design what you’ll build. (Agile is not an excuse to skip the design phase.) Next? Building and testing it. The difference between Agile and Waterfall is that in an Agile process the entire process is completed in a matter of days instead of months, and is repeated continuously.

Significantly less new behavior will be introduced in newer versions, and that requires a different way of testing. There’s less exploratory testing to be done and more validating needs doing to make sure no existing behavior has been changed unintentionally. Validating that same thing over and over again takes too much time, and takes the fun out of the game.

Know what tests to automate!

Identify trivial checks that are done repeatedly and start with automating those. Validating lots of things manually, in a short period of time leaves little time for proper exploratory testing. Automating trivial checks creates time for testers to do what they’re best at Finding all sorts of brilliant scenario’s you could have never imagined.

Find the key test-cases to automate, by increasing the releasing frequency (bit by bit)

Test automation done wrong may cause a longer time to market. Keep the end in mind: Test automation is not a goal. Neither is coverage. Shipping valuable new features to the customer, quicker, is!

Defining (and executing) the right test-cases is the key to high-quality software. Automating them is just a way to shorten the execution time. Test automation is just a tool. So is unit testing. What tool applies, and how itapplies depends on the problem at hand. Shipping the application more often helps to pinpoint the things that need to be tested automatically, first.

Figure out how to automate tests, in a cost-efficient way

Don’t waste lots of time on building test-automation that doesn’t catch any bugs. Instead, spend little time on one (or a few) that do.

Some parts of the software change often. Others are rarely ever touched. As a result, some things are never checked, others are, constantly. Use this formula to determine whether it makes sense to automate a test:

(C * T$) > (B + M$)= Automate that test!!
C = Times the unit under test is changing
T$ = Cost of testing the unit under test, manually
B = Time needed to build test automation.
M$ = Maintenance cost of that test-approach (when code changes)

Try to automate as many tests as possible. If the formula shows test-automation is too expensive, consider a cheaper way to automate that test!

Know how to automate testing!

Test automation is not a goal. Shipping value quicker is! That’s easier said than done. A high frequency requires a different way of working. With that comes a different set of approaches, practices, and tools.

Different test-cases require different types of testing. Unit-testing is a well-known test-automation method. But unit-testing alone won’t cut it. You’ll need to apply a lot more test-automate techniques. Combine different types of testing to get an effective test automation strategy.

Automated tests are useless if they’re not run. And the faster a defect has been discovered, the easier it is to fix it. Run the automated tests as fast as possible and install the software on an environment automatically, if the tests succeed.

Constantly deploying stable, small increments of your software to production, within minutes, safely, can be achieved by applying a software engineering approach called “Continuous Delivery”.

Deploying software that often, requires the installation to be automated. This is typically done through the usage of delivery pipelines. These pipelines install the software on different environments, automatically and they run a set of automated acceptance tests. Deployments are canceled if one or more of the automated acceptance tests fail. This eliminates the need for manual testing done by the stakeholders.

Communicate the specifications of the new features to the stakeholders. Use Example Mapping to find the specifications in the refinement process, and write them down using the Gherkin syntax. This allows the team to create ‘living documentation’: Documentation that is an automated test.

Communicate about new specifications before they’re being built and validate them as early as possible. Use behavior-driven development to implement these specifications.

Releasing new versions, all the time, requires the software to be shippable at all times. That means you’ll need small, usable increments. Use Elephant Carpaccio to define the smallest possible, non-technical user stories that have business value.

Simplify!

The fewer teams needed to create or modify functionality, and the fewer systems that need to be modified in order to create new functionality, the faster you’ll create business value and the easier it gets to test functionality automatically and ship it autonomously. Do you want a shorter time to market? Design for it!

The more systems that need modifications to create or modify functionality, the harder it gets to create or change functionality. All teams that work on these systems need to fully understand the new requirements and somebody needs to test the feature altogether when all the teams are done implementing their part. That requires coordination.

If testability wasn’t an architectural driver in your codebase up till now, chances are, that big chunks of code are hard to test, and — because these parts have probably been built a long time ago — it’s probably unclear what the requirements of these parts of the software were in the first place. You’ll need to execute a chain-test to validate if it still works. These usually manifest in an end-to-end test, and those are known to be expensive, unreliable and slow.

Move towards independent feature teams and craft an (enterprise) architecture that is easily testable and releasable. Testability is an architectural driver. Think about how to create and test your functionality carefully, before even getting near a keyboard to write code. Take testability into consideration (on an enterprise level, too). Rearranging software and teams can shorten the time to market!


Summary

Test-automation done right can be pretty complicated and broken tests can cause serious impediments. Test automation requires a different way of thinking and lots of new things to learn. It takes time and effort to get there. Don’t expect to get it right, right away. Inspect and adapt. Learn from each other. Help each other and communicate.

Automating trivial, time-consuming processes that need doing all the time is considered a good practice. Testing could be one of those. Automating things makes a more efficient, more pleasant development process. It makes sense to automate testing and releasing if it needs doing all the time because it shortens the time to market.

Focus on the right things. Fix (small) things that hurt the most, first. Don’t waste too much time on it, either. It makes the team choose carefully how to apply the minimum amount of automation to speed up the development process as much as possible.

Read more »