Image Understanding and Description in Android Apps

"What's in this image?" has many answers depending on the task. Is there a barcode? What does the text say? What objects are present? What's happening in the scene? Is this rash serious? (Don't build that last one casually.) The right tool depends on which question you're asking.

Three levels of image understanding

Task-specific: image labeling

// implementation("com.google.mlkit:image-labeling:<version>")
val labeler = ImageLabeling.getClient(ImageLabelerOptions.Builder().setConfidenceThreshold(0.7f).build())

suspend fun labels(bitmap: Bitmap): List<String> =
    labeler.process(InputImage.fromBitmap(bitmap, 0)).await().map { it.text }   // e.g. ["Dog", "Grass"]

Labels are fast and free, which is good for tagging, search and filters. They don't describe relationships ("a dog chasing a ball").

On-device description

ML Kit's GenAI Image Description API produces a short natural-language description on supported devices. It's ideal for draft alt text that stays private. Code is in ML Kit GenAI APIs Explained.

Multimodal Gemini with structured output

When you need specific information extracted from an image, ask a multimodal model for structured JSON matching a schema rather than free text:

@Serializable
data class ReceiptInfo(val merchant: String?, val totalPaise: Long?, val date: String?, val category: String?)

class ReceiptExtractor {
    private val schema = Schema.obj(
        mapOf(
            "merchant" to Schema.string(nullable = true),
            "totalPaise" to Schema.long(description = "Total amount in paise", nullable = true),
            "date" to Schema.string(description = "ISO-8601 date", nullable = true),
            "category" to Schema.enumeration(listOf("food", "travel", "shopping", "bills", "other")),
        )
    )

    private val model = Firebase.ai(backend = GenerativeBackend.googleAI()).generativeModel(
        modelName = "gemini-3.8-flash",            // check current names
        generationConfig = generationConfig {
            responseMimeType = "application/json"
            responseSchema = schema
            temperature = 0f
        },
    )

    suspend fun extract(photo: Bitmap): ReceiptInfo {
        val response = model.generateContent(content {
            image(photo.scaleToMaxSide(1280))
            text("Extract the receipt details. Use null when a value is not clearly visible.")
        })
        return Json { ignoreUnknownKeys = true }.decodeFromString(response.text ?: "{}")
    }
}

Structured output reduces parsing errors, but values can still be wrong, so always let users review extracted data before saving. Schema helper names follow Firebase AI Logic's Kotlin SDK. Check the structured-output docs for your SDK version.

Hybrid pattern: OCR first, then reason on text

For documents and receipts, running on-device OCR (ML Kit text recognition) first and sending only the extracted text to a cloud model is often cheaper and more private than sending the full photo, and accurate enough for many tasks.

Privacy and safety

• Photos often contain people, documents and locations. Get consent before cloud processing, and strip EXIF metadata (GPS) before uploading.
• Don't build identification of real people from faces without strong legal and ethical review.
• Avoid medical, legal or safety-critical judgements from images without professional oversight.
• Downscale images: smaller uploads use fewer tokens, less data and less time.

Use cases

Frequently asked questions

Conclusion

Match the tool to the question: ML Kit vision for fixed perception tasks, on-device Image Description for private short descriptions, and multimodal Gemini with structured output for open questions and extraction. Combine OCR with text reasoning where you can, and keep users in the loop.

Next step: add voice input with Speech-to-Text and AI Voice Features.