Chapter 2

Preparing for Test Automation

configuration of an infrastructure

Testability of the system (a non-functional requirement) should be designed and implemented in parallel with other system features. The software architect and automation tester identify areas where improvements can be made. Some solutions for better testability:

  1. Accessibility identifiers, e.g id HTML attributes, generated by the framework (e.g. Symfony/WordPress) or manually set by developers. Cucumber can target both ids and data-testids, which is far more reliable than using xpaths.
  2. System environment variables, e.g. TEST_MODE=true – this environment variable signals the application to operate in a controlled testing state during deployment. HEADLESS=true (on MacOS) can tell your tests to run in headless mode.
  3. Deployment variables – can be set before starting deployment.

Designing for testability of a system consists of:

  1. Observability – you need to be able to determine whether you got the correct result after running the test. E.g. feedback on a web application can be checked with Cucumber to compare the actual result with the expected result.
  2. Controllability – you need to be able to access interfaces to perform actions on the system. E.g. Cucumber can target many UI elements such as input fields, buttons, links etc.
  3. Architecture transparency – the architecture documentation should provide components and interfaces that give observability and controllability.

Most projects have one or more environments for testing:

Local development environment (white box testing)

  • Component testing
  • GUI testing
  • API testing

Build environment (low level testing)

  • Component testing
  • Component integration testing
  • Static analysis

Integration / test environment (black box testing)

  • Testing with a fully automated test suite. E.g. a Cucumber UI regression suite.

Preproduction / Staging environment (tests have less focus on functional aspects)

  • Performance efficiency testing
  • User acceptance testing

Production environment

  • Canary release: deploy to a small group of users (the ‘canaries’) for real-world testing, monitor the performance, errors and user feedback, then extend the rollout to more users.
  • Blue/green deployment: deploy the new version to the ‘green’ environment, test it, then switch traffic from the ‘blue’ server to the ‘green’ one with e.g. a DNS update.
  • A/B testing: two or more versions of a feature are deployed to determine which performs better.

Selecting the right tools and strategies

Analyse a system to determine the appropriate test automation solution.

  • Which test process activities (test management, test design, test generation, test execution) should be automated?
  • Which test levels (component/unit testing, component integration testing, system testing, system integration testing, acceptance testing) should be supported?
  • Which test types (functional, non-functional, black box, white box tests) should be supported?
  • Which test roles (e.g developers, test automation engineers, test analysts) and skill sets should be supported?
  • Which software products should be supported?
  • Which kinds of systems need to be compatible with the automation tool?
  • Avaiability of test data and its quality.
  • Ways to emulate unreachable cases (e.g. 3rd party applications involved).

Illustrate the technical findings of a tool evaluation using a comparison table, with tools in the columns, requirements in the rows, and cells containing the properties off each tool, including priorities.

Requirements to consider when comparing tools:

  • The language of the tool and the IDE tools. E.g. Cucumber by default uses Ruby as the programming language, but other languages, such as Java or JavaScript can also be used.
  • The ability to configure a tool. E.g. Cucumber can be configured to support different spoken languages for Gherkin syntax, by putting a # language: comment on the first line of a feature file.
  • The ability to manage test data within the tool. E.g. Cucumber has data tables, which allow you to add your test data right underneath a step (as opposed to in a separate file).
  • Different test types might need different automation tools. E.g. you may want to use Cucumber for the functional tests, and JMeter for the non-functional load tests.
  • The ability to provide reporting capability. E.g. Cucumber can provide reports in both json and html format.
  • The ability to integrate with other tools (CI/CD, reporting…). E.g. Cucumber can integrate with MS Teams and AIO Tests.
  • The ability to expand overall test architecture and assess the scalability, maintainability, modifiability, and compatibility of tools.