For six weeks, developer ghorbelhamdi has been hardening Buddy, an open-source Android app that hands control of the device to coding agents like Claude Code, Codex, and OpenCode. The app exposes the phone’s accessibility service as a suite of ~50 tools over MCP on localhost, allowing agents to read the screen tree, tap, swipe, and type. It’s a wild experiment in letting LLMs operate a phone without root, using an Alpine Linux userland via proot, but the real breakthrough is the decision to run the inference model directly on the phone’s hardware.

The NPU Advantage

Cloud models are the default, but Buddy supports a fully offline mode by downloading llama.cpp’s official Android build and serving it locally. On a OnePlus 12 with a Snapdragon 8 Gen 3, running Gemma 4 E2B, the performance gap is stark. CPU inference hits 128 tokens/s for reading and 12 for writing, while the GPU (OpenCL) manages 691/18. The Hexagon NPU, however, crushes it at ~2,400 tokens/s for reading and ~23 for writing. That NPU speed turns a 'wait a minute' response into an instant interaction, all while working in airplane mode.

Engineering Around Android Restrictions

Getting this to run wasn’t straightforward. Android 10+ (targetSdk 29+) blocks executing binaries from the app’s data folder, forcing the developer to launch the server through /system/bin/linker64. Context window bloat was another killer: initial agent prompts ballooned to ~24k tokens due to verbose tool descriptions. The fix was a 'lite mode' that cuts the prompt to ~4k by loading only a dozen core tools, with a more_tools call for the rest. Additionally, llama.cpp’s default thinking mode on Qwen3.5 caused simple requests to burn 3,000 tokens; disabling --reasoning dropped step times to seconds.

Google Play’s Accessibility Block

Distribution hit a wall when Google Play rejected the app, citing that the 'Accessibility API cannot be requested for performing autonomous tasks.' This forced Buddy to GitHub and F-Droid. Later, adding SMS tools triggered Play Protect to flag the APK as harmful, mimicking banking malware patterns. The solution was architectural: Buddy now reads SMS and call logs via a background helper running as the shell user, so the main app holds no sensitive permissions.

Key Takeaways

  • Local NPU inference on Snapdragon 8 Gen 3 makes on-device agent loops viable for short tasks.
  • Google Play’s strict accessibility policies are pushing agent apps to F-Droid and GitHub.
  • Context engineering is critical: reducing tool descriptions from 24k to 4k tokens was essential for small models.
  • Android’s sandbox restrictions require creative workarounds like linker64 execution and shell-user helpers.

The Bottom Line

Buddy proves that on-device AI agents are finally practical, not just a novelty. While Google’s gatekeeping is annoying, the technical leap in NPU performance makes local inference a viable, private, and fast alternative to the cloud for daily smartphone tasks.