Use the accessibility tree when the answer is expressed as Android UI elements. Use a screenshot when the answer depends on pixels: an image, chart, canvas, visual alignment, or a control with no useful accessible label. For an unfamiliar app, capture both once and compare what each observation actually tells your agent.
Mobilebox exposes these as separate requests. Neither is a universal replacement for the other. The useful question is which evidence lets the agent choose and verify its next action with the least ambiguity.
Start with the structured observation
curl --fail-with-body "$MSB/v1/phones/$ID/observe?format=text" \
-H "Authorization: Bearer $MOBILESANDBOX_API_KEY"The response is a numbered list containing element type, visible text, state and position. JSON observation adds fields an ordinary program can inspect. A text label or resource id is usually easier to target than a pixel coordinate, especially when a keyboard opens or a layout shifts.
Android’s accessibility testing guidance explains how inspecting a UI hierarchy reveals accessible properties and missing labels. That is the underlying distinction: the tree describes exposed semantics; it does not describe every visible pixel.
For example, a note editor can expose a TextField containing the saved note. An agent can assert the note text directly. If it only needs to confirm the phrase “Release checklist” remains after reopening, sending the whole screenshot to a vision model adds an interpretation step without improving that assertion.
Reach for pixels when semantics run out
curl --fail-with-body "$MSB/v1/phones/$ID/screenshot" \
-H "Authorization: Bearer $MOBILESANDBOX_API_KEY" -o screen.pngA map, drawn signature, image preview or custom-rendered chart may have little useful representation in the tree. In that case, inspect the screenshot and target a coordinate only after checking the current layout. Observe again after the action. A successful coordinate tap establishes input delivery, not that the intended object was selected.
Do not assume every blank screenshot is a broken screenshot endpoint. Android’s FLAG_SECURE contract allows an app to keep secure window contents out of screenshots. A protected view is a capability boundary to identify, not a reason to promise a capture bypass.
Choose evidence for the assertion
| Task | Primary observation | Additional check |
|---|---|---|
| Find a Save button | Text or resource id | Observe the resulting screen |
| Verify a note was saved | Saved text after reopening | Screenshot for a human review |
| Inspect a photo crop | Screenshot | Confirm the chosen image or saved output |
| Diagnose an unlabeled control | Tree and screenshot together | Check its actual effect |
| Review visual layout | Screenshot | Keep device and app settings fixed |
There is no general token-saving claim here. Image dimensions, tree size, model and prompt all affect the cost of a particular run. Measure those inputs for your workload rather than treating a shorter-looking UI as a benchmark.
Keep the pair synchronized
A screenshot from one screen and element references from another screen create a misleading evidence pair. Capture them near the same operation and record the phone id, app package and operation sequence. Pause your own actions while collecting a pair; an app may still update itself asynchronously.
In the recorded Android run, actual screenshots are paired with operation responses and hashes in the evidence manifest. That demonstrates one specific workflow, not universal app compatibility. For the targeting rules behind the next action, read screen references and recovery and the API reference.