Cloud AI vs On-Device AI for Android

Where the model runs is one of the most consequential decisions in an AI feature. It determines who can use the feature, how fast it feels, what it costs per user, what your privacy policy says, and whether it works in a lift with no signal. Yet teams often treat it as an afterthought once they've picked "a model".

This article is for Android developers designing AI features. It compares cloud inference and on-device inference across seven dimensions, explains how Android's on-device stack (AICore, Gemini Nano, ML Kit GenAI) differs from the cloud path (Firebase AI Logic or your own backend), introduces Firebase's experimental hybrid inference, and shows Kotlin patterns for graceful fallback.

Definitions: what "on-device" and "cloud" mean here

• On-device inference: the model runs on the phone's CPU, GPU or NPU. On Android this means Gemini Nano via AICore (through the ML Kit GenAI APIs), task-specific ML Kit models, or your own model running in LiteRT.
• Cloud (server-side) inference: the app sends input over the network to a model running in a data centre, such as Gemini through Firebase AI Logic or any model behind your own backend, and receives the output.

Seven dimensions compared

On-device AI on Android in detail

Android's managed on-device generative stack has three layers. Gemini Nano is the model. AICore is the system service that distributes, updates and hardware-accelerates it. ML Kit GenAI APIs are what your app calls. Because AICore manages the model, your APK doesn't grow by gigabytes, and several apps share one model.

The constraints are part of the design, and you must handle them:

• Feature status varies by device. It can be unavailable, downloadable, downloading or available. Check it before showing UI.
• Foreground only. Inference is permitted only when your app is the top foreground app. Background use returns an error.
• Quotas. Too many requests in a short time return BUSY, and long-running heavy use can hit a battery-use quota.
• Beta APIs. At the time of writing the GenAI APIs are beta (speech recognition is alpha) and carry no SLA.

Outside Gemini Nano, LiteRT (formerly TensorFlow Lite) runs models you choose or train and bundle yourself. That gives you full control over the model and broad device support, in exchange for owning size, accuracy and updates.

Cloud AI on Android in detail

Cloud inference gives you the strongest models and the same behaviour on every device. There are two common architectures:

• Firebase AI Logic from the app. A Kotlin SDK calls Gemini directly, protected by Firebase App Check, which the documentation says is enforced for AI Logic from July 2026. It's the quickest path with no backend to run.
• Your own backend. The app calls your API, and your server calls the model provider. It's the right choice when you need multiple providers, retrieval over private data (RAG), server-side tools, strict per-user quotas, or audit logs.

Hybrid inference: the best of both, with care

Many real features should run on-device when possible and fall back to the cloud otherwise, or the reverse. Firebase AI Logic offers experimental hybrid inference for Android. You configure an inference mode when you create the model, and the SDK routes each request between Gemini Nano (via ML Kit's Prompt API) and a cloud Gemini model:

// Requires firebase-ai plus the experimental firebase-ai-ondevice artifact.
// Check the Firebase AI Logic hybrid docs for current versions and imports.
val model = Firebase.ai(backend = GenerativeBackend.googleAI())
    .generativeModel(
        modelName = "gemini-3.5-flash",   // cloud model used when routing to the cloud
        onDeviceConfig = OnDeviceConfig(mode = InferenceMode.PREFER_ON_DEVICE),
    )

suspend fun shortReply(message: String): String? =
    model.generateContent("Suggest a short, friendly reply to: $message").text

Know the current limits before you adopt it. The API is experimental, routing is rule-based, and the on-device side is focused on single-turn generation from text or a single image. Multi-turn chat still needs the cloud. And a "prefer on-device" feature may silently send data to the cloud on unsupported devices, so your consent and disclosure must cover that.

A provider-neutral fallback pattern

If you need more control, for example a consent check, a different cloud provider or backend mediation, write the routing yourself and expose a clear state to the UI:

sealed interface SummaryResult {
    data class Success(val text: String, val source: Source) : SummaryResult
    data class Unavailable(val reason: String) : SummaryResult
    data class Failed(val message: String) : SummaryResult
    enum class Source { ON_DEVICE, CLOUD }
}

class SummaryRepository(
    private val onDevice: Summarizer,          // wraps ML Kit GenAI Summarization
    private val cloud: Summarizer,             // wraps Firebase AI Logic or your backend
    private val settings: PrivacySettings,     // user's cloud-processing choice
    private val connectivity: Connectivity,
) {
    suspend fun summarize(text: String): SummaryResult {
        if (onDevice.isReady()) {
            return runCatching { SummaryResult.Success(onDevice.summarize(text), SummaryResult.Source.ON_DEVICE) }
                .getOrElse { SummaryResult.Failed("Couldn't summarise on this device. Try again.") }
        }
        if (!settings.cloudProcessingAllowed) return SummaryResult.Unavailable("Enable cloud processing in Settings to use this on your device.")
        if (!connectivity.isOnline()) return SummaryResult.Unavailable("Connect to the internet to summarise.")
        return runCatching { SummaryResult.Success(cloud.summarize(text), SummaryResult.Source.CLOUD) }
            .getOrElse { SummaryResult.Failed("Summary service is busy. Please try again.") }
    }
}

interface Summarizer {
    suspend fun isReady(): Boolean
    suspend fun summarize(text: String): String
}
interface PrivacySettings { val cloudProcessingAllowed: Boolean }
interface Connectivity { fun isOnline(): Boolean }

Returning the Source lets the UI label where the result came from, for example "Summarised on your device", which builds user trust, and lets your analytics measure fallback rates.

Real-world use cases

Pitfalls and troubleshooting

• Feature works on your Pixel, not on test devices. Gemini Nano device support is limited. Check feature status at runtime and test on the device list your users actually have.
• BACKGROUND_USE_BLOCKED errors. You're calling on-device GenAI from a worker or service. Move the call to a foreground interaction.
• BUSY errors under load. Per-app quota. Add exponential backoff and avoid firing requests on every keystroke.
• Different answers on different devices. Different Gemini Nano versions can produce different output for the same prompt. Design the UI for variation.
• Unexpected cloud bills. Fallback rates were higher than assumed. Monitor routing and set server-side limits.

Frequently asked questions

Conclusion

On-device AI wins on privacy, offline use, latency and per-request cost. Cloud AI wins on capability and reach. For many Android features the right answer is hybrid: on-device where supported, cloud with consent where not, behind a repository that reports which path ran.

Next step: build the cloud path first with Gemini API Integration in Android Using Kotlin, then add the on-device path with Gemini Nano and ML Kit GenAI on Android.