Android end-to-end testing

Why do we perform end-to-end testing?

Unit tests, screenshot tests, and integration tests all validate the app in isolation: the server it talks to is mocked or faked. End-to-end (E2E) tests close that gap by running the real app against a real Home Assistant instance, exercising the same path a user takes.

This is what catches breakages that no other test can see, such as an upstream change in Home Assistant Core or in the frontend that breaks onboarding, or a WebView behavior that only shows up on a specific Android API level.

The E2E workflow

E2E tests run through the e2e.yml workflow. Each run boots one emulator per API level we support, from 29 up to the latest, and drives the whole flow against a live Home Assistant instance on all of them at once. That breadth is what makes the tests able to detect API level-specific issues, but it also makes them expensive, so they are not part of the pull request pipeline. Instead, they run:

• Every night at 05:00 UTC on the main branch, using the latest Home Assistant dev image tag.
• Manually, through workflow_dispatch, on any branch carrying the workflow. The dispatch takes an optional home-assistant-version input, which is the Home Assistant image tag to test against (dev by default).

What the workflow does

1. Assembles the fullDebug APK for the emulator platforms, with StrictMode disabled.
2. Installs the Maestro CLI, verifying the download against the published checksum.
3. Builds the device list: one emulator per API level, from 29 up to androidSdk-target in gradle/libs.versions.toml.
4. Starts a Home Assistant container from the configuration in .github/e2e/homeassistant, and asserts that the mobile_app component is loaded before going any further.
5. Starts the emulators, installs the APK on each of them, and runs the Maestro flow sharded across all of them in parallel.
6. Uploads the artifacts needed to debug a failure, whatever the outcome of the run.

note

The device list is floored at API 29 even though the app supports older versions. On older API images, the bundled WebView renders the frontend, but its content never surfaces to Maestro's view-hierarchy polling, so every WebView assertion fails. The instrumentation tests in the pull request pipeline still cover the full range down to the app's minSdk.

note

The Home Assistant container runs on the GitHub Actions runner, while the emulators run on Emulator.wtf. The workflow bridges the two with an egress tunnel, and a DNS override that resolves homeassistant.internal to the forwarded IP inside the emulators.

Maestro flows

The flows live in the .maestro folder at the root of the repository. Today there is a single flow, described entirely in .maestro/onboarding.yaml: every step the test performs is written there, and that file is the one to edit when the flow needs to change. It covers the most critical path of the application:

• Authentication: the app goes through the real login page of the instance with real credentials, and completes the authorization flow.
• Device registration: the app registers itself on the instance, which is what everything built on the mobile_app integration depends on.
• The frontend renders in the app: reaching the Overview page proves the WebView loaded and displayed the frontend on that API level.
• The external bus works both ways: the frontend only offers the Companion app entry in its settings once the app has announced itself over the external bus, and tapping that entry makes the frontend ask the app to open its native settings screen.

The flow is parameterized so it can run against any instance:

• HOME_ASSISTANT_URL
• HOME_ASSISTANT_USERNAME
• HOME_ASSISTANT_PASSWORD

Editing a flow

The Maestro documentation describes every available command. A few things are worth keeping in mind when changing onboarding.yaml:

• The flow drives the debug application, io.homeassistant.companion.android.debug. Changing the appId breaks the run, since that is the APK the workflow installs.
• The flow runs on every API level from 29 up to the latest we support, so anything version-specific must stay conditional, the way the existing steps handle the permission dialogs that only exist from a given API level onwards.
• Steps match on the text displayed on screen, so renaming a label in the app means updating the flow in the same pull request.

Running a flow locally

Install the Maestro CLI, start an emulator or plug in a device, install the debug app on it, and run:

./gradlew :app:installFullDebug
maestro test \
  -e HOME_ASSISTANT_URL=http://homeassistant.local:8123 \
  -e HOME_ASSISTANT_USERNAME=<username> \
  -e HOME_ASSISTANT_PASSWORD=<password> \
  .maestro/onboarding.yaml

The flow launches the app with clearState, so it starts from a fresh onboarding every time. Point it at a throwaway instance rather than your own: it registers a new device on the server it connects to.

Debugging a failure

Every run uploads an e2e-artifacts artifact, whether it passed or failed. It contains the following files:

Path	Content
maestro-results/	Per-shard command trace, maestro.log, and the screenshots captured on failure
logcat-*.txt	Full device logcat, one file per emulator
homeassistant.log	Timestamped container log
homeassistant-config.json	The /api/config response: Home Assistant version and loaded components
homeassistant-container.json	docker inspect of the container, including the exact image digest

Read them in that order and stop at the first one that explains the failure. Most failures are already visible in the failing Maestro command and the screenshot taken at that point, which is the fastest way to tell a genuinely broken screen from an element that merely never reached the accessibility tree. Opening the Home Assistant log first usually wastes time.

Only once the app and the flow are ruled out is it worth looking upstream. The dev image moves every night, so the useful comparison is against the last run that passed: diff the two homeassistant-config.json files to get the Home Assistant version on either side of the break, then look at what landed in core or the frontend between those dates.

Automated triage

A nightly failure goes unnoticed unless someone checks the workflow results every morning. To avoid that, a failing workflow run dispatches an e2e-triage agentic workflow, and a successful run closes the issue a previous failure opened.

The triage workflow downloads the artifacts of the failed run, and those of the last successful run when one is available, then follows the procedure documented in the .agents/skills/ha-android-e2e-debugging skill, which is the source of truth for the triage order, for what each artifact contains, and for the known flake patterns.

The agent only diagnoses: it never modifies a file or opens a pull request. It reports into a single e2e-failure issue in the Android repository, commenting on it when one is already open and creating it otherwise. An upstream finding is reported there as well, including the proposed fix, for a maintainer to carry over to core or to the frontend.