Flaky tests

TL;DR

  • Test data, improper initiation, timing of application (in Selenium — test-data-id, waits). Concurrency. SUT fails to respond.
  • When your tests are flaky, do some root cause analysis to understand why they're flaky.

Reasons

‣
The tests themselves can introduce flakiness.
‣
The test-running framework
‣
SUT and OS
‣
Large Selenium tests are flaky

Mitigation rules

  • Re-runs, and marking tests themselves as flaky
  • Monitor such tests: a test was flaky if it had at least one flaky run during the week.
  • Better parallel run
SuperMade with Super