Chapter 4

Implementing Test Automation

Test Automation Development

Test automation pilot – quick to conduct and can have a big impact on the direction of the project.

Based on information gathered about the system and requirements of the project, evaluate the following technical aspects:

  • Programming languages that will be used.
  • Suitable (commercial / open source) tools.
  • Test levels to cover.
  • Test cases selected.
  • Test case development approach. E.g. You may consider testing by querying the database directly versus checking the UI for saved data. Given the difficulties in getting Cucumber to access a DB using Windows, you might opt for the latter.

Based on the above, several prototypes can be created, to show the pros and cons of the different approaches. During the pilot, try to integrate the solution into CI/CD, to expose issues early.

As the number of test cases grows, think about changing the initial CI/CD setup to run tests in different ways and at different times.

During the pilot, evaluate these non-technical aspects:

  • The knowledge and experience of the team members.
  • The team structure.
  • Licensing and organisation rules.
  • The type of planned testing and targeted test levels to cover during automation.

After the pilot project is complete, assess the success/failure and make an appropriate decision.

Risks associated with test automation development

Deployment risks can be identified from the pilot (e.g. firewall openings, resource utilisation like CPU and RAM).

Technical deployment risks can include:

Packaging

Testware may need to be uploaded into a version-controlled repository to share across an organisation.

Logging

Gives most of the information about test results:

  • Fatal – logs events that abort test execution. Caused by e.g. network misconfiguration, firewall blocking, or Selenium server not running. E.g. FATAL: Unable to establish TCP connection between Cucumber and Selenium WebDriver
  • Error – when a test case fails, caused by an assertion failure. E.g. ERROR Selenium [:cucumber_step_failure] Step ‘Then I should see the confirmation message’ failed: ElementNotFoundException
  • Warn – when an unexpected but non-fatal issue occurs. E.g. WARN Selenium [:selenium_manager] There was an error managing msedgedriver (error sending request for url (https://msedgedriver.azureedge.net/LATEST_RELEASE_139_MACOS): error trying to connect: dns error: failed to lookup address information: nodename nor servername provided, or not known); using driver found in the cache
  • Info (diagnostic message) – basic info about what is happening, e.g. INFO Selenium [:logger_info] Details on how to use and modify Selenium logger: https://selenium.dev/documentation/webdriver/troubleshotting/logging
  • Debug (diagnostic message) – execution-specific details not required for basic logs, but useful when a test case fails, e.g. DEBUG Selenium [:selenium_error] ElementNotInteractableException: element not visible
  • Trace (diagnostic message) – similar to Debug, but with even more information, and tracks every action, even successful ones. E.g. TRACE Selenium [:element_locator] Attempting to locate element by XPath: //button[@id=’submit’] TRACE Selenium [:element_locator] Element found: TRACE Selenium [:click_action] Initiating click on element TRACE Selenium [:click_action] Click successful

Test structuring

  • The most important part of the test automation solution is the test harness and test fixtures included in it (items that must be available for tests to run).
  • Test fixtures provide freedom in controlling a test environment and the test data.
  • Preconditions and postconditions can be defined for test execution. E.g. Cucumber has before and after hooks.
  • Test fixtures enable the creation of automated tests that are repeatable and atomic.

Updating

  • Risks with automatic updates on test harnesses and devices can be mitigated by having adequate power supplies, proper network connections & device configuration plans.

Test automation solution maintainability

Clean code principles:

  • Use a common naming convention for classes, methods & variables. E.g. Ruby uses snake_case.
  • Use a logical and common project structure. Cucumber has a pre-defined folder structure that separates feature files from step definitions.
  • Avoid hardcoding.
  • Avoid too many input parameters for methods.
  • Avoid long and complex methods.
  • Use logging.
  • Use design patterns where they are beneficial and required.
  • Focus on testability.

To ensure high quality code, use static analysers and code formatters to improve readability. Use an agreed branching structure and strategy in version control.