AI Tokens, Latency, and Cost Optimization for Mobile Apps

A chat feature costs almost nothing in testing with five users. At 50,000 daily users, each sending ten messages with growing conversation history, the bill becomes a product decision. Meanwhile, users judge the feature by how quickly the first words appear. Tokens connect both problems.

Tokens 101

• Models process text as tokens: chunks of characters. In English, a token averages a few characters. Other scripts, such as Hindi, may use more tokens per word.
• You pay for input tokens (system instruction + context + history + user message) and output tokens, often at different rates.
• Images, audio and documents also consume tokens.
• Latency grows with input size (processing) and output length (generation).

val tokens = model.countTokens(prompt)              // Firebase AI Logic: count tokens for a request
Log.d("AI", "input tokens=${tokens.totalTokens}")
// Response usage metadata reports prompt/candidate token counts when available:
// response.usageMetadata?.promptTokenCount, response.usageMetadata?.candidatesTokenCount

Where the tokens go in a chat

Cost levers, in order of impact

1. Model tiering: use the smallest model that passes your evaluation, such as a Flash-Lite-class model for classification, short rewrites and extraction, and reserve larger models for hard reasoning.
2. Trim context: keep the last N turns, and summarise older ones into a short memory. Send only relevant retrieved passages, not whole documents.
3. Cap output: set maxOutputTokens, and ask for concise formats ("3 bullets").
4. Cache: deduplicate identical requests (hash of prompt version + input). Use provider-side context caching for large repeated prefixes where supported.
5. Move work on-device: Gemini Nano for supported tasks and devices costs nothing per request.
6. Avoid accidental calls: no AI on every keystroke, scroll or screen open without user intent.

Latency: what users feel

• Time to first token (TTFT) matters most for perceived speed. Stream responses so text appears immediately. See Streaming AI Responses in Compose.
• Smaller prompts and smaller models reduce TTFT.
• Show progress states ("Thinking…") and allow cancelling.
• Prefetch or precompute where the action is predictable, for example summaries of items the user is likely to open, but only within budget.

Trimming chat history

data class Turn(val role: String, val text: String)

class HistoryTrimmer(private val maxTurns: Int = 10, private val maxChars: Int = 8_000) {
    fun trim(history: List<Turn>, memorySummary: String?): List<Turn> {
        val recent = history.takeLast(maxTurns).toMutableList()
        // Drop oldest until under the character budget (rough token proxy)
        while (recent.sumOf { it.text.length } > maxChars && recent.size > 2) recent.removeAt(0)
        return buildList {
            memorySummary?.let { add(Turn("user", "Summary of earlier conversation: $it")) }
            addAll(recent)
        }
    }
}

Rate limits and retries

suspend fun <T> withAiRetry(maxAttempts: Int = 3, block: suspend () -> T): T {
    var delayMs = 500L
    repeat(maxAttempts - 1) {
        try { return block() }
        catch (e: Exception) {
            if (e is CancellationException) throw e
            val retryable = e is IOException || e is QuotaExceededException || e is ServerException
            if (!retryable) throw e
            delay(delayMs + Random.nextLong(0, 250))        // backoff + jitter
            delayMs *= 2
        }
    }
    return block()
}

QuotaExceededException and ServerException are Firebase AI Logic types. Adapt the check to your SDK or backend errors. Don't retry invalid requests or safety blocks.

Budgets and limits

• Per-user daily request or token caps (server-side, or at least client-side for Firebase setups).
• Budget alerts in your cloud billing.
• Track cost per feature and per active user. See Monitoring AI.

Examples

Frequently asked questions

Conclusion

Tokens drive both cost and latency. Measure them, pick the smallest adequate model, trim history and context, cap outputs, cache, move suitable tasks on-device, stream for perceived speed, and enforce budgets and retries deliberately.

Next step: get better results from fewer tokens with Prompt Engineering for Android Developers.