LiteRT and LiteRT-LM on Android

LiteRT is Google's on-device inference runtime for .tflite models, the evolution of TensorFlow Lite. LiteRT-LM builds on it to run language models efficiently, with conversation handling, streaming, and GPU or NPU acceleration.

Where LiteRT fits

LiteRT for custom models

Typical uses are image classifiers, audio models or custom detectors exported to .tflite. The flow is: load the model, prepare input tensors in the exact shape and normalisation used in training, run inference, and post-process outputs.

class Classifier(context: Context) : AutoCloseable {
    private val interpreter = Interpreter(
        FileUtil.loadMappedFile(context, "classifier.tflite"),
        Interpreter.Options().setNumThreads(4),
    )

    fun classify(input: FloatArray): FloatArray {
        val output = Array(1) { FloatArray(NUM_CLASSES) }
        interpreter.run(arrayOf(input), output)          // input must match the model's shape
        return output[0]
    }

    override fun close() = interpreter.close()
}

LiteRT also offers a newer CompiledModel API with improved GPU and NPU acceleration. Check the docs for the version you use.

LiteRT-LM for language models

LiteRT-LM's Kotlin API follows this pattern: configure an Engine with a model file and a backend, initialise it off the main thread (loading can take several seconds), open a conversation, and stream responses as a Flow.

class LocalLlm(private val modelPath: String, private val cacheDir: String) : AutoCloseable {
    private var engine: Engine? = null

    /** Slow: loads the model. Call from a background coroutine. */
    suspend fun load() = withContext(Dispatchers.Default) {
        val config = EngineConfig(
            modelPath = modelPath,
            backend = Backend.GPU(),          // or CPU / NPU depending on device
            cacheDir = cacheDir,              // speeds up subsequent loads
        )
        engine = Engine(config).also { it.initialize() }
    }

    /** Streams the reply; collect in a ViewModel and append chunks to UI state. */
    fun ask(prompt: String): Flow<String> {
        val conversation = checkNotNull(engine) { "Call load() first" }.createConversation()
        return conversation.sendMessageAsync(Message.of(prompt))
            .map { it.toString() }
            .onCompletion { conversation.close() }
    }

    override fun close() { engine?.close(); engine = null }
}

Choosing and delivering a model

• Models: open models such as Gemma variants converted for LiteRT-LM. Smaller (around 1B parameters or less, quantised) models suit more devices.
• Size: LLM files are often hundreds of MB to several GB. Don't bundle them in the base APK.
• Delivery: download on demand over Wi-Fi with progress and checksums, or use Play for On-device AI for model delivery through Play.
• Licences: check each model's licence and usage terms.

Performance and devices

See Offline AI and Device Compatibility for gating features by device capability.

Use cases

Troubleshooting

• Wrong outputs from a custom model: input preprocessing (size, normalisation, channel order) doesn't match training.
• OOM or crash when loading an LLM: the model is too large for the device; use a smaller or more quantised model.
• UI freezes: initialisation or inference on the main thread.

Frequently asked questions

Conclusion

LiteRT runs your custom models; LiteRT-LM runs open LLMs on device with streaming and hardware acceleration. Load off the main thread, deliver models on demand, gate by device capability, and check model licences.

Next step: process images and text on device with On-Device Image and Text Processing.