Flaky tests

TL;DR

  • Test data, improper initiation, timing of application (in Selenium — test-data-id, waits). Concurrency. SUT fails to respond.
  • When your tests are flaky, do some root cause analysis to understand why they're flaky.

Reasons

The tests themselves can introduce flakiness.
The test-running framework
SUT and OS
Large Selenium tests are flaky

Mitigation rules

  • Re-runs, and marking tests themselves as flaky
  • Monitor such tests: a test was flaky if it had at least one flaky run during the week.
  • Better parallel run
SuperMade with Super