Chapter 1

Objectives for Test Automation

Advantages of test automation

  • You can run more automated than manual tests because they’re faster to execute.
  • You can create tests that can’t be done manually. E.g. verifying response times down to the number of milliseconds for a web service. Humans can’t measure such precise timing without specialised tools, but a few lines of Ruby code can do it effortlessly:
start_time = Time.now
make_api_call
end_time = Time.now

response_time = end_time - start_time
expect(response_time).to be < 0.05  # 50 milliseconds
  • You can create tests that are more complex than manual ones. (Complex steps can be tedious, time-consuming and error-prone to execute manually).
  • Automated tests are less subject to human error. E.g. if you want to verify that a user’s session expires after exactly n minutes of inactivity, it’s less error-prone to get Cucumber to wait for the exact period of time:
Feature: Session Management

  Scenario: Session expiration after 15 minutes of inactivity
    Given the user is logged in
    And the user is inactive for 15 minutes
    When the user tries to perform an action
    Then the system should prompt for re-authentication
  • More efficient and effective in terms of test resources (faster execution, repeatability and parallelisation).
  • Provides quicker feedback regarding system quality.
  • Helps improve system reliability (availability and recoverability).
  • Improves consistency of test execution across test cycles.

Disadvantages of test automation

  • Additional project costs (hiring an automation engineer, training).
  • Initial investment to set up an automation solution.
  • Time to develop and maintain the automation solution.
  • Clear test automation objectives are needed for success.
  • Rigidity of tests and less adaptable to changes in the system.
  • Introduction of additional defects by test automation. E.g. running your Cucumber tests over a long period of time, without cleanups, could pollute the database, causing performance issues or incorrect analytics.

Limitations to test automation

  • Not all manual tests can be automated. E.g. features that verify that you are human, such as two-factor authentication or Captcha, can’t be tested with automation.
  • Verifying only what automated tests are programmed to do.
  • Some quality characteristics can’t be tested with automation. E.g. the visual styling of a web page, user experience, brand voice and tone.
  • Test automation can only check results that can be verified by an automated test oracle. An automated test oracle could be a script that verifies whether a web service returns the correct HTTP status code and response body for a given input. Here’s an example using Cucumber with Ruby:
Then(/^the response should be successful$/) do
  expect(@response.code).to eq(200)
  expect(@response.body).to include("Welcome, user!")
end

Automation in the SDLC

Waterfall model (linear and sequential)

  • Automation implementation happens in parallel to or after the implementation phase.
  • Test runs take place during the verification phase when the software components are ready for testing.

V-model (sequential)

  • Providing a test automation framework for each level is possible.

Agile (iterative)

  • Frequent automated test execution is possible.
  • Eliminating silos (the ‘whole team approach’) allows teams to cover all test levels with the appropriate amount and depth of test automation.

Selecting suitable tools

  • For testers with no programming knowledge – choose a no-code or low-code solution, like Robot Framework.
  • For technical testers – it can be helpful to select a tool whose language matches the system under test. E.g. if your application is also written in Ruby, you can use the in-process, as opposed to out-of-process, approach when testing a web service with Cucumber.