Test suites

Your team already has tests. BenchFleet runs them unchanged on the phones in your fleet: Maestro flows, Espresso and other Android instrumentation tests, and Appium tests. You get a result per phone and per test, the runner's own output, screenshots and a JUnit report. Test suites are part of the Pro and Team plans.

Open Automation → Test suites and choose the runner: Maestro flows, Espresso tests or Appium tests. The page says whether that runner is ready on this computer.

Which phones

Test suites run on phones attached to the computer that runs the app. Phones of other computers (gateways) and cloud phones are not supported yet.

Each phone is leased to the run, so nothing else uses it meanwhile. Phones that are offline, in Manual or Paused mode, or with automation switched off cannot be picked. If you take control of a phone during a run, or a check only a person may pass appears (a CAPTCHA, a sign-in code), the tests on that phone stop at once and the phone waits for you.

Maestro flows

You need Java 17 or later and Maestro on the computer that runs the app (install it from maestro.dev). Then:

1. Choose folder… and pick the folder with your flows (the one you would give maestro test), or Choose zip…. A config.yaml at the top works as in Maestro.
2. Pick the phones. Each phone runs the whole suite.
3. Optional, under Tags, variables and secrets: Only tags and Skip tags; Variables as NAME=value, one per line (shown in the run, so not for passwords); Secrets as NAME=secret_name, where secret_name is a secret from Settings → Secrets. Maestro gets a secret as MAESTRO_NAME, so use ${MAESTRO_NAME} in your flows. Its value is never shown. You can also set a Time limit per phone.
4. Run.

JavaScript in your flows (runScript) runs on your computer, and requests it makes go out directly, not through the app's privacy controls.

Espresso and instrumentation tests

Nothing to install. Build the two APKs as usual (./gradlew assembleDebug assembleDebugAndroidTest), then:

1. Choose the App APK and the Test APK. BenchFleet checks that they belong together before anything starts.
2. Pick the phones.
3. Optional, under Filters, arguments, orchestrator and sharding: Classes, Skip classes, Packages, Annotations, Skip annot. and Size; Arguments as NAME=value (shown in the run, so not for passwords); split the tests across the phones (sharding), so four phones take about a quarter of the time; run each test in its own process with the AndroidX Test Orchestrator (upload its two APKs from your Gradle cache), and clear the app's data before each test.
4. Run.

Your APKs stay installed on the phone afterwards, as after connectedAndroidTest.

Appium tests

You need Node.js 20 or later and Appium 2 or 3 with the UiAutomator2 driver on the computer that runs the app (npm install -g appium && appium driver install uiautomator2). BenchFleet starts Appium itself for each phone.

Your tests run where you start them, on your computer or in CI, and only the server address changes:

1. Choose Appium tests, pick the phones and Run.
2. Once a phone is ready, the run shows its Appium address. Copy with key copies it with that phone's key; give it to your Appium client as the server URL. The key works for that phone only, and only until the run gives the phone back.
3. Run your tests. Every session goes to that phone, whatever udid your tests ask for. Install your build first, or upload the APK and use appium:app = artifact:<id>; a path or a download address is refused.
4. When they are done, Finish with JUnit report… with your framework's JUnit file gives the phone its results, one per test. Finish without report ends it without results.

The driver is UiAutomator2. Appium's Espresso driver is refused: run Espresso tests as an Espresso suite instead. A few commands that would reach beyond the phone are refused too, such as mobile: shell; the run's output names each one.

Results

Each phone shows Passed, Passed on retry, Failed (with the message) or Not run with the reason: the phone was offline, someone else had it, a person took it over, or an install failed. Ignored Espresso tests are Skipped. Under each phone:

• the results per flow or test, with the time each took;
• Output, the runner's own output; Phone log after an Espresso failure; JUnit for that phone; Screenshot files;
• the video of the run, when one was kept (videos of runs).

JUnit in the list of runs gives one report for the whole run: one test suite per phone, one test case per flow or test.

Device conditions

A run can set each phone up before the tests start, under Device conditions: language, time zone, dark mode, location…, and puts the phone's own settings back afterwards: after a pass, a failure, a cancel, and also after a crash or a restart.

Condition	What the phone gets	Needs
Language	that language for the apps under test (the flows' apps, the Espresso app, and any you add)	Android 13 or later
Time zone	that time zone	any Android
Clock	12-hour or 24-hour time	any Android
Dark mode	dark mode on or off	Android 10 or later
Font size	larger or smaller text	any Android
Animations off	no animations, for steadier tests	any Android
Keep the screen on	the screen stays on while the phone has power	any Android
Location	a mock GPS position: a city from the list, or latitude and longitude	Android 12 or later

If a phone cannot take a condition (its Android is too old, for example), it is left exactly as it was and the tests do not run on it; choose run with what it took to run them anyway. Each phone's result lists what it took and that its settings were put back. A phone a person controls is never changed back behind their back: its settings go back when they let it go. Workflows can set the same conditions with the step device.conditions.

From CI

controlhub-ci claims free phones, installs your build, runs the suite on each phone, writes a JUnit file and gives the phones back, whatever happened. Point it at the app with CONTROLHUB_URL and CONTROLHUB_TOKEN: an API token (Settings → Security → API tokens) allowed to run tests on those phones.

# Maestro flows on two phones
controlhub-ci --maestro .maestro/ --count 2 --apk app-debug.apk --junit maestro.xml

# Espresso tests split across four phones
controlhub-ci --espresso app-debug.apk app-debug-androidTest.apk --count 4 --shard --junit espresso.xml

# Appium tests: your command runs once per phone, on the CI machine
controlhub-ci --appium 'pytest tests/ --junitxml="$CONTROLHUB_JUNIT"' --count 2 --apk app-debug.apk --junit appium.xml

For Appium, your command gets the phone's address with its key in APPIUM_URL and writes its JUnit report to CONTROLHUB_JUNIT.

Maestro takes --include-tags, --exclude-tags, --env NAME=VALUE and --secret NAME=SECRET. Espresso takes --test-class, --not-class, --test-package, --not-package, --annotation, --not-annotation, --size, --instrumentation-arg NAME=VALUE, --orchestrator and --clear-package-data. Device conditions are --locale, --timezone, --clock, --dark-mode, --font-scale, --no-animations, --stay-awake and --location (add --conditions-continue to run with what a phone took). --retries, --app-build, --compare-to and --videos are on Flaky tests and failures.

Exit codes: 0 every test passed, 1 a test failed, 2 a phone had no result (refused, offline, stopped for a person), 3 the run did not end in time.

Not supported yet

Phones of other computers and cloud phones, Maestro web flows, network throttling, and iOS.

Related: Flaky tests and failures, Workflows and schedules.