Android test matrix generator

In short: The Android test matrix generator turns the Android versions you support and conditions such as screen size, language, dark mode, font size and network into the rows to test. Pairwise mode covers every pair of values in a few dozen rows instead of every combination, counts the phone setups you need, and exports the matrix as CSV, all in your browser.

Lowest Android version (minSdk)API 37, Android 17API 36, Android 16API 35, Android 15API 34, Android 14API 33, Android 13API 32, Android 12 (12L)API 31, Android 12API 30, Android 11API 29, Android 10API 28, Android 9API 27, Android 8.1API 26, Android 8.0API 25, Android 7.1, 7.1.1API 24, Android 7.0API 23, Android 6.0API 22, Android 5.1API 21, Android 5.0

Highest Android version (targetSdk)API 37, Android 17API 36, Android 16API 35, Android 15API 34, Android 14API 33, Android 13API 32, Android 12 (12L)API 31, Android 12API 30, Android 11API 29, Android 10API 28, Android 9API 27, Android 8.1API 26, Android 8.0API 25, Android 7.1, 7.1.1API 24, Android 7.0API 23, Android 6.0API 22, Android 5.1API 21, Android 5.0

Rows

Pairwise: every pair of values at least once Every combination

Other factors (values separated by commas)

Screen needs another phone

Density needs another phone

Language

Theme

Font size

Network

Your own factor

37 rows instead of 1,944 combinations, covering every pair of values. They need 36 phone setups (Android version, screen); the other factors are settings you change on the same phone.

#	Android	Screen	Language	Theme	Font size	Network
1	API 28	small phone	English	light	100%	Wi-Fi
2	API 28	large phone	German (long words)	dark	130%	mobile data
3	API 28	foldable	Arabic (right to left)	light	200%	offline
4	API 28	tablet	English	dark	200%	Wi-Fi
5	API 29	small phone	German (long words)	light	130%	offline
6	API 29	large phone	English	light	100%	mobile data
7	API 29	foldable	English	dark	100%	offline
8	API 29	tablet	Arabic (right to left)	light	100%	Wi-Fi
9	API 30	small phone	Arabic (right to left)	dark	130%	mobile data
10	API 30	large phone	English	light	200%	Wi-Fi
11	API 30	foldable	German (long words)	light	100%	Wi-Fi
12	API 30	tablet	German (long words)	light	130%	offline
13	API 31	small phone	English	light	130%	Wi-Fi
14	API 31	large phone	Arabic (right to left)	dark	100%	offline
15	API 31	foldable	German (long words)	light	200%	mobile data
16	API 31	tablet	English	light	100%	mobile data
17	API 32	small phone	English	light	200%	Wi-Fi
18	API 32	large phone	German (long words)	dark	100%	mobile data
19	API 32	foldable	Arabic (right to left)	light	130%	offline
20	API 32	tablet	English	light	100%	Wi-Fi
21	API 33	small phone	English	light	100%	Wi-Fi
22	API 33	large phone	German (long words)	dark	130%	mobile data
23	API 33	foldable	Arabic (right to left)	light	200%	offline
24	API 33	tablet	English	light	100%	Wi-Fi
25	API 34	small phone	English	light	100%	Wi-Fi
26	API 34	large phone	German (long words)	dark	130%	mobile data
27	API 34	foldable	Arabic (right to left)	light	200%	offline
28	API 34	tablet	English	light	100%	Wi-Fi
29	API 35	small phone	English	light	100%	Wi-Fi
30	API 35	large phone	German (long words)	dark	130%	mobile data
31	API 35	foldable	Arabic (right to left)	light	200%	offline
32	API 35	tablet	English	light	100%	Wi-Fi
33	API 36	small phone	English	light	100%	Wi-Fi
34	API 36	large phone	German (long words)	dark	130%	mobile data
35	API 36	foldable	Arabic (right to left)	light	200%	offline
36	API 36	tablet	English	light	100%	Wi-Fi
37	API 29	small phone	English	light	200%	Wi-Fi

The same choices always give the same matrix. Nothing leaves your browser.

Running the rows is the slow part. BenchFleet shows every phone on one screen, and on Pro a workflow's Device conditions step sets language, dark mode, font size and other settings for a run and puts them back afterwards, on many phones at once. See workflows and schedules and managing a bench of Android test devices.

Free for up to 3 phones, with no account. Pro adds workflows, schedules and AI for up to 25 phones from £12 a month + VAT.

Download freeSee pricing

How to use it

1. Choose the lowest and highest Android versions you support: usually your app's minSdkVersion and targetSdkVersion. Every API level between them becomes a value.
2. Tick the other factors that matter for your app and edit their values; add one of your own, such as an account type.
3. Pairwise gives the shortest list that pairs every value with every other value at least once; every combination lists them all, up to 5,000 rows.
4. Export the rows as CSV for your test tracker, or print them.

How the matrix is built

Each new row starts from the first pair of values no row has yet, then gives every other factor the value that adds the most new pairs, taking the first on a tie. That greedy all-pairs construction is not always the smallest possible matrix, but it is close, it always covers every pair, and it gives the same rows for the same choices. The tests check the coverage for each preset.

The values for versions come from the Android API levels list. The mobile app testing checklist has the checks to run on each row, and Android's guidance on testing across screen sizes explains which screen classes matter.

Related free tools

• Android API levels and version listEvery Android version with its API level and code name, and the changes testers meet at each level.
• Device lab cost calculatorA cloud device farm or your own phones over 12 months, with dated list prices and every formula shown.
• Android permissions list and adb grant commandsEvery runtime permission and the special ones, with API levels, and the adb commands to grant, revoke or check them.

All free Android developer tools

Questions

What is pairwise testing?

Choosing test rows so that every pair of values from any two factors appears together at least once, such as Android 14 with dark mode or a tablet with Arabic. Most bugs that depend on settings depend on one or two of them, so pairs catch most of them in far fewer rows than every combination.

Why does the matrix count phone setups?

The Android version, the screen and the density belong to the phone, so each different combination needs another phone or emulator. Language, theme, font size and network are settings you change on the same phone between rows.

Does the same input always give the same matrix?

Yes. The construction is deterministic, so a matrix you export today is the one a colleague gets from the same choices tomorrow.

Is pairwise enough?

For broad coverage of settings, usually. Add rows by hand for combinations you know are risky, such as the oldest Android version with the largest font size, and test new features on every phone you support.

Does it send my factors anywhere?

No. It works in your browser, and nothing leaves your browser.

Start with the Free plan

Free for up to 3 phones, with no account. Pro adds workflows, schedules and AI for up to 25 phones from £12 a month + VAT.

Download freeSee pricing

• A Free plan with no time limit and no account
• Starts in Local mode, with no telemetry or analytics in the app