Evaluate an Android agent by the task outcome and the evidence it leaves behind, not by the confidence of its final message. “The agent tapped Save” and “the draft can be found after reopening” are different observations. A useful evaluation makes that difference explicit before the run begins.
Keep the first suite small enough to inspect manually. Three carefully specified tasks can reveal more than dozens of vague “use the app” prompts, especially when you include failure and cleanup behavior rather than scoring only the best-looking success.
Write a task that can fail unambiguously
Use an owned app and staging account. Define the exact initial state, allowed actions, harmless input and completion condition. For a note task, a completion condition could be: a unique marker remains visible in the app after process termination and reopening, with no extra duplicate note created.
Record the build, runtime, agent/model configuration and any permitted tools. A task run with a saved login is not the same starting condition as a fresh install. Likewise, a screenshot-enabled agent and a text-only agent have different information available. Those differences belong in the result, not in an unexplained score.
Separate three failure layers
| Layer | Example | What to retain |
|---|---|---|
| Environment | Phone did not become ready | Phone status and failure field |
| Control | Tap target missing or action batch stopped | Per-action results and current screen |
| Task | Save action ran but record is absent | Outcome assertion and post-check evidence |
This avoids blaming the planner for an installation failure or credit limit. It also avoids counting a transport-level HTTP 200 as task success. Mobilebox action responses can contain failed actions; the evaluator must inspect them.
For the request contract, use the API reference. Android’s testing strategy guidance distinguishes different levels of checks and their costs. A mobile agent evaluation belongs alongside smaller deterministic checks, not as a reason to replace every existing test with a model.
Include recovery cases on purpose
Add a missing label, a denied permission and a changed screen after observation. Give the agent a precise recovery policy: observe current state, identify which earlier operations completed, and continue only from the remaining safe step. For any operation with an external side effect, require a fresh outcome check before another attempt.
Then test interruption. If the run exceeds its budget or a control call fails, can the harness leave the phone parked or delete its disposable phone? Does the final report say the task is incomplete? A clean failure with accurate evidence is preferable to a fabricated success followed by an orphaned running phone.
Track measurements with a denominator
Record attempts, completed tasks, incomplete tasks, environment failures and cleanup failures separately. Include the number of repeated runs and keep all attempts in the private evidence set. Do not report a single successful replay as a reliability rate.
Useful per-run fields include action count, observation count, elapsed time, billed session seconds and the outcome assertion. Model cost is a separate input; Mobilebox’s phone runtime rate does not include the agent’s reasoning provider. Avoid comparing two agents with different allowed tools or different starting state without stating that difference.
Ground the report in inspectable artifacts
For each run, keep the smallest evidence needed to reproduce the conclusion: initial condition, operation sequence, returned observations, final assertion and cleanup result. Use screenshots for visual claims and structured screen text for text/state claims. Redact test-account and credential material before sharing.
The Android replay shows one narrow recorded workflow with matched screenshots and responses. Its manifest is proof of that capture, not a substitute for repeated evaluations. Start with the CI smoke test, then add outcome checks and failure cases that correspond to the risks in your actual application.