Flaky tests and failures

A test that passes on one run and fails on the next costs time. BenchFleet keeps the history of your test suites and workflows, so you can tell a flaky test from a real failure. Test suites and workflows are part of the Pro and Team plans.

Retry failed tests

Set Retries (0 to 5) in a new test run. Each failed Maestro flow or Espresso test runs again on the same phone, up to that many times. No extra approval is asked, and the phone's time limit covers the retries. Appium tests are not retried: your own test command drives the phone.

A test that fails and then passes shows Passed on retry, with what the failed attempts said. It is never shown as a plain pass and it counts towards the test's flaky score, but it does not fail the run. In JUnit it is a passing test with its failed attempts marked as flaky, which Jenkins shows as a flaky test.

Flaky scores

Every finished run is kept per test, phone model and app build. Name the build in Build when you start a run (for Espresso the app's version is used by default). A test is flaky when it passes and fails on the same build and the same phone model, or passes only on a retry. Failing on one model and passing on another is a model-specific failure, not flakiness.

A flow or test in a run's results, or a workflow in the Workflows list, that needs a look shows a badge: Flaky with its score out of 100, Some flakiness, or Failing. Hover over it to see why.

Give CI runs the same suite name every time, so their history adds up.

Quarantine

Quarantine next to a test in a run's results: from the next run, a failure of that test is still shown, marked Quarantined, but it does not fail CI. The run itself still says it failed. Release makes the test count again.

Compare with an earlier run

Under a finished run, Compared with shows what changed against The previous run of the suite, or a run you pick: New failures, Still failing, Fixed, Slower, New tests and Removed tests. A comparison never changes a result.

Videos of runs

BenchFleet records each phone's screen while a test suite or a workflow runs on it. By default the video is kept only when the phone failed or had no result: Watch the video plays it under the phone and Save video downloads it.

Settings → System → Videos of test and workflow runs decides:

• Keep the video: When the run fails (the default), Always or Never (don't record);
• Quality: Preview (light) or Full;
• Recordings at once, Keep for (14 days by default) and a Storage limit (5 GB by default; the oldest videos are deleted first).

Recording never holds up a run. When too many phones record at once or the computer is busy, a phone runs without a video and says why. Videos are recorded for phones attached to this computer.

Why it failed

Open a failed workflow run's replay and Why it failed comes first: a verdict (The app changed, Flaky, The workflow, Phone or Hub, A person is needed or Not clear), the facts behind it, the recent runs and what to do next. Propose a repair… can suggest a fix for the workflow.

This needs no AI model. For a summary in plain words, choose one of your models and Explain in plain words: it reads the analysis, the failed step and the texts on the failure screen, never the phone, the run's inputs or your secrets. A local model costs nothing, but may take a few minutes. Add models in Settings → Models (AI with your own model).

From CI

controlhub-ci (test suites from CI) takes the same options:

• --retries N retries failed tests; one that then passes is printed PASS … FLAKY.
• --app-build NAME names the build for the history (with --apk, a fingerprint of the APK by default).
• --compare-to previous (or a run's ID) prints new failures, fixed, still failing and slower tests.
• --videos DIR saves the kept videos into DIR and attaches them to their tests in the JUnit file, which Jenkins and GitLab show next to the test.

A quarantined failure is printed QUARANTINED and does not change the exit code; neither do flaky marks.

Related: Test suites, Screens and failure clusters.