Appearance
BenchFleet reliability report: 1,000 simulated phones and 17 real ones
In short: In BenchFleet's own tests on a Windows 11 computer, 1,000 simulated phones connected in 3.8 seconds, a batch of 1,000 jobs over the Hub and three gateways succeeded with none sent to the wrong phone, and 17 real phones ran a scheduled workflow every 10 minutes for 24 hours with all 2,465 jobs succeeding.
We test BenchFleet the way a device lab uses it: many phones, phones that drop off, links that fail and computers that restart. This report gives the results of those tests, taken from the project's own verification reports of 29 September to 1 October 2026 on a Windows 11 reference computer. Only results marked as met are here, each with its source, and the last section lists what has not been shown yet.
Free for up to 3 phones, with no account. Pro adds workflows, schedules and AI for up to 25 phones from £12 a month + VAT.
At scale: 1,000 simulated phones
Simulated phones answer at once, so these numbers measure BenchFleet's own work, not a phone's speed. Source: docs/verification/m7.3.md, scale suite, 1 October 2026.
| What we tested | Result |
|---|---|
| 1,000 phones connecting | All 1,000 connected 3.8 seconds after start |
| Listing 1,000 phones | 53 ms |
| A phone's status change reaching a live client (100 changes) | 95% within 5.7 ms; the longest 21.2 ms |
| BenchFleet's own time per command, 16 at once on 1,000 phones | 95% within 17.8 ms |
| One batch of 1,000 jobs over the Hub and three gateways, 250 phones each | 1,000 of 1,000 succeeded, 0 sent to the wrong phone, in 34.5 seconds (29 jobs a second) |
| A hung phone, with four clients sending it commands that all time out | The other phones' 95th percentile was 101 ms before and 80 ms during: no slowdown |
| Scrolling the Fleet page with 1,000 phones changing state | 95% of frames within 18.1 ms, about 55 frames a second |
When things go wrong
Source: docs/verification/m7.3.md, chaos suite and streamer test, 1 October 2026.
| What we tested | Result |
|---|---|
| 60 phones in a batch while, for 6 seconds, 12 phones dropped off or came back every 150 ms | 60 of 60 jobs succeeded, every phone kept its ID, 0 sent to the wrong phone |
| 3 of 20 phones hanging in the middle of a batch | Their 3 jobs failed cleanly; the other 17 succeeded; nothing went to the wrong phone |
| A gateway's link dropping 12 times while 40 jobs ran on its phones | 40 of 40 succeeded, each exactly once |
| BenchFleet's core process killed with 12 jobs running, then restarted | Jobs set to retry or resume finished successfully; jobs set to fail were marked as interrupted; none was lost |
| The video service killed with 20 live screens, twice | All 20 live again after 4.1 and 4.0 seconds |
On real phones
The 17 real phones are all the same model (Android 14), connected over Wi-Fi. Input went only to three test phones; the rest were read-only.
| What we tested | Result | Source |
|---|---|---|
| A read-only workflow scheduled every 10 minutes on all 17 phones for 24 hours | 2,465 jobs, all succeeded: 144 firings, none missed; 0 failures, 0 retries; median run 3.8 seconds | m3.2.md, 29 to 30 September 2026 |
| A screenshot of all 17 phones as one group action | 17 of 17 succeeded in 1.2 seconds | m3.3.md, 30 September 2026 |
| Mirroring one phone's swipe to two others with different video sizes | All three scrolled to the same entry; excluding one took effect on the next event | m3.3.md, 30 September 2026 |
| Restarting the adb server with the 17 phones connected | Every phone came back with the same ID, in 4.1 seconds | m7.3.md, 1 October 2026 |
What is not shown yet
- 72 hours without a restart. The first long run found memory growing without limit, which was fixed. The second run stopped early after 13.9 hours, with memory still growing by about 60 MB a day after the first hour. A full 72-hour run is still to do, so we make no claim about it.
- No network traffic in Local mode. A one-hour audit of connections found only connections to the same computer; the packet capture that would confirm it needs an administrator and is still pending.
- Real phones of several models. The real-phone results come from one model; other models are covered by the simulated phones and by people using the app.
What this means if you run a lab
- One phone's trouble stays its own. A phone that hangs or drops off fails or retries its own job; the others carry on, and nothing is sent to the wrong phone.
- Restarts lose nothing. Each workflow says what happens to a job that a restart interrupts: retry, resume from its last step, or fail; none is lost silently.
- No limit on phones. The plans set how many phones are active at once; the app itself has no device limit, and the tests run it with 1,000.
See how this fits a real lab in controlling many Android phones from one PC and Android app testing on real phones, or read about BenchFleet.
Questions
Were these tests run on real phones?
Partly. The scale and chaos tests use simulated phones, which answer at once, so they measure BenchFleet's own work. The 24-hour soak, the group screenshot and the adb server restart ran on 17 real Android phones of one model, over Wi-Fi.
Which version of BenchFleet was tested?
The versions of 29 September to 1 October 2026; the scale and chaos results are from commit 3f89b34 of 1 October. Each row names the report it comes from, with its date.
Has BenchFleet run for 72 hours without a restart?
Not shown yet. The second 72-hour soak stopped early after 13.9 hours, with memory still growing by about 60 MB a day after the first hour, so it does not count as a pass. A full 72-hour run is still to do.
What does "sent to the wrong phone" mean?
A command that reaches a phone other than the one it was meant for. In the 1,000-job batch every phone typed its own ID, so a job on the wrong phone would have shown up. The count was zero in every scenario reported here.
How were the real phones connected?
Over Wi-Fi, with adb's network mode, to the same Windows 11 computer that ran BenchFleet. Input went only to three test phones; the other phones were used read-only.
Start with the Free plan
Free for up to 3 phones, with no account. Pro adds workflows, schedules and AI for up to 25 phones from £12 a month + VAT.
- A Free plan with no time limit and no account
- Starts in Local mode, with no telemetry or analytics in the app