Monitoring AI Latency, Errors, and Inference Costs

Tokens, Latency and Cost explains the concepts. This article is about operating AI in production: instrumenting calls, building dashboards, setting alerts and budgets, and connecting metrics to quality.

What to measure

Instrumentation overview

Client instrumentation

fun <T> Flow<T>.instrumented(feature: String, model: String, source: String, telemetry: AiTelemetry): Flow<T> {
    var start = 0L
    var firstTokenAt = 0L
    return this
        .onStart { start = SystemClock.elapsedRealtime() }
        .onEach { if (firstTokenAt == 0L) firstTokenAt = SystemClock.elapsedRealtime() }
        .onCompletion { cause ->
            val end = SystemClock.elapsedRealtime()
            telemetry.record(
                AiCallEvent(
                    feature = feature, model = model, source = source,
                    ttftMs = if (firstTokenAt > 0) firstTokenAt - start else null,
                    totalMs = end - start,
                    outcome = when (cause) {
                        null -> "ok"
                        is CancellationException -> "stopped"
                        else -> "error:${cause::class.simpleName}"
                    },
                )
            )
        }
}
// Usage: model.generateContentStream(prompt).mapNotNull { it.text }.instrumented("summarize", "gemini-3.8-flash", "cloud", telemetry)

Send these as analytics events or custom Performance Monitoring traces. Never include prompts or responses. See AI Privacy.

Token usage and cost

Responses usually include usage metadata (input and output token counts). Record it where you can trust it, ideally on your backend, and multiply by your model's current price:

data class ModelPrice(val inputPerMillion: Double, val outputPerMillion: Double)   // from your config

fun estimateCost(inputTokens: Int, outputTokens: Int, price: ModelPrice): Double =
    inputTokens / 1_000_000.0 * price.inputPerMillion +
    outputTokens / 1_000_000.0 * price.outputPerMillion
// Load prices from remote config/backend; check the provider's pricing page for current rates.

Firebase AI Logic also offers AI monitoring in the Firebase console, covering requests, latency, errors and tokens. Use it alongside your own feature-level metrics.

Dashboards, alerts and budgets

• Dashboards: p95 TTFT and error rate per feature and model; daily cost per feature; tier share.
• Alerts: error-rate spikes (for example 429s), latency regressions after model changes, cost above a daily threshold.
• Budgets: cloud billing budget alerts, plus per-user quotas on your backend to prevent abuse.

Quality signals

• Ratings, report rate, regenerate/retry rate, stop rate, and edit distance between suggestion and final text.
• Compare quality metrics when switching models or prompts (A/B tests with flags).
• Combine with offline evaluation sets. See Evaluating AI.

Acting on the data

Frequently asked questions

Conclusion

Operate AI like any production dependency, plus a few new metrics: TTFT and latency percentiles, error types, tokens and cost per feature and user, tier share and quality signals, with dashboards, alerts and budgets, and without logging user content.

Explore more in the Android Developer Blog hub.