How to Choose the Right AI Model for an Android App

Your team has decided to add an AI feature. Someone suggests "the best model", meaning the one at the top of a benchmark. Three months later the feature costs more than expected, feels slow on mobile networks, can't be used offline, and legal is asking where user data goes. None of that was the model's fault. It was a selection problem.

This guide is for intermediate Android developers and tech leads who need to choose a model, or a combination of models, for a real feature. You'll get a six-question framework, a decision diagram, a comparison of the main options available to Android developers, Kotlin code for routing between models at runtime, and a lightweight way to evaluate quality before you commit.

The principle: choose for the task, not the leaderboard

Benchmarks measure general capability on standard datasets. Your users care about one specific task on their devices and networks. For many mobile features, a smaller, faster or on-device model is the better product choice, even if it scores lower on a leaderboard.

Six questions that decide the model

1. What kind of task is it?

• Fixed perception task (read text, scan barcodes, detect objects or faces, identify language): use task-specific ML such as ML Kit or a LiteRT model. An LLM is the wrong tool.
• Short text transformation (summarise, proofread, rewrite, describe an image): a small generative model, possibly Gemini Nano on-device.
• Open-ended reasoning, long context, multi-turn chat, tool use: a cloud LLM.

2. How sensitive is the data?

Private messages, health notes and financial documents argue strongly for on-device processing, or at least explicit consent before sending data to a cloud provider. Remember that your Play Store Data safety declaration must match what you actually do.

3. How fast must it feel?

Inline features such as autocomplete or smart replies need near-instant responses. Network round-trips plus model time may be too slow, especially on mobile networks. Features where the user explicitly waits, such as "summarise this document", tolerate a few seconds, especially with streamed output.

4. Must it work offline?

If yes, only on-device options qualify: ML Kit, LiteRT, or Gemini Nano on supported devices.

5. Which devices must it support?

Gemini Nano runs only on specific supported devices. If the feature must work for all your users, including budget phones common in markets like India, you need a cloud path or a smaller model you ship yourself.

6. What can you afford per user, per month?

Cloud models are billed per token. On-device has no per-request fee but costs engineering effort and is limited by device and quota. Estimate cost before building, as shown below.

Decision diagram

The options available to Android developers

Model families and names change frequently. At the time of writing, Google's documentation recommends its latest Flash and Flash-Lite models for new projects. Check the current models and pricing pages before you decide.

The same thinking applies if you compare providers, for example Gemini, OpenAI or open-weight models hosted on your own servers. Compare them on your task with your evaluation set, consider data-handling terms, and keep the provider behind an interface so you can switch. See Cloud AI vs On-Device AI for the deployment side of this choice.

Estimate cost before you build

Cloud LLM pricing is usually per million input tokens and per million output tokens, and output tokens often cost more. A token is roughly a few characters of English text. Estimate with your own numbers:

/**
 * Rough monthly cost estimate for a cloud AI feature.
 * Fill the prices from the provider's CURRENT pricing page; never hard-code guesses in production.
 */
data class CostInputs(
    val monthlyActiveUsers: Int,
    val requestsPerUserPerMonth: Double,
    val avgInputTokens: Int,          // prompt + context + chat history
    val avgOutputTokens: Int,
    val pricePerMillionInput: Double, // in your billing currency
    val pricePerMillionOutput: Double,
)

fun estimateMonthlyCost(c: CostInputs): Double {
    val requests = c.monthlyActiveUsers * c.requestsPerUserPerMonth
    val inputCost = requests * c.avgInputTokens / 1_000_000.0 * c.pricePerMillionInput
    val outputCost = requests * c.avgOutputTokens / 1_000_000.0 * c.pricePerMillionOutput
    return inputCost + outputCost
}

Two lessons usually come out of this exercise. First, input tokens dominate in chat, because the whole history is resent on every turn, so trimming or summarising history matters. Second, a small share of heavy users can dominate cost, so per-user limits are a product requirement, not an afterthought.

Implementing model choice in code

Treat the model as a replaceable dependency. Route each request based on capability, availability and user consent:

enum class AiTask { SUMMARIZE_SHORT, REWRITE, CHAT, LONG_DOCUMENT_QA }

interface AiEngine {
    val name: String
    suspend fun isAvailable(): Boolean
    fun supports(task: AiTask): Boolean
    suspend fun run(task: AiTask, input: String): String
}

class ModelRouter(
    private val onDevice: AiEngine,                   // e.g. Gemini Nano via ML Kit GenAI
    private val cloud: AiEngine,                      // e.g. Gemini via Firebase AI Logic
    private val cloudConsentGiven: () -> Boolean,     // user opted in to cloud processing
) {
    sealed interface Route {
        data class Use(val engine: AiEngine) : Route
        data class Unavailable(val reason: String) : Route
    }

    suspend fun route(task: AiTask): Route {
        if (onDevice.supports(task) && onDevice.isAvailable()) return Route.Use(onDevice)
        if (!cloud.supports(task)) return Route.Unavailable("Not supported on this device")
        if (!cloudConsentGiven()) return Route.Unavailable("Cloud processing is turned off")
        return if (cloud.isAvailable()) Route.Use(cloud) else Route.Unavailable("You're offline")
    }
}

The ViewModel calls route(), shows the reason when the result is Unavailable, and records which engine answered. That metric is invaluable when you investigate quality complaints later.

Evaluate quality with your own test set

Before committing to a model, build a small evaluation set: 30 to 100 realistic inputs, with notes on what a good output must contain. Run every candidate model over it and review the results, both manually and with simple automated checks.

data class EvalCase(val input: String, val mustMention: List<String>, val maxWords: Int)
data class EvalResult(val engine: String, val passed: Int, val total: Int, val avgMillis: Long)

suspend fun evaluate(engine: AiEngine, task: AiTask, cases: List<EvalCase>): EvalResult {
    var passed = 0
    var totalMillis = 0L
    for (case in cases) {
        val start = System.nanoTime()
        val output = runCatching { engine.run(task, case.input) }.getOrDefault("")
        totalMillis += (System.nanoTime() - start) / 1_000_000
        val words = output.split(Regex("\\s+")).count { it.isNotBlank() }
        val mentionsAll = case.mustMention.all { output.contains(it, ignoreCase = true) }
        if (output.isNotBlank() && mentionsAll && words <= case.maxWords) passed++
    }
    return EvalResult(engine.name, passed, cases.size, totalMillis / cases.size.coerceAtLeast(1))
}

Simple checks like these won't capture everything. Tone and factual accuracy still need human review, but they catch regressions fast, especially when a provider updates a model version.

Worked examples

Common mistakes

• Choosing by benchmark instead of by your own evaluation set.
• Ignoring device reach. An on-device-only feature that most of your users can't access.
• Hard-coding the model name in many places. Model names change, and older models get retired. Centralise it, ideally in remote config.
• Not budgeting output tokens. Set a maximum output length, and ask for concise answers.
• Forgetting the non-AI path. When no model is available, the app must still work.

Frequently asked questions

Conclusion

Choosing an AI model for Android is a product and engineering decision: task fit, privacy, latency, offline needs, device reach and cost, verified with a realistic evaluation set. Start small and private, put the model behind an interface, route at runtime, and move up only when the data says you should.

Next step: write your feature's answers to the six questions and estimate its monthly cost with the formula above. Then dig into the deployment trade-offs in Cloud AI vs On-Device AI for Android.