RAG for Android Apps: Connecting AI to Your Own Data

Ask a general model "What's the return window for electronics in our app?" and it will produce a plausible answer that may be wrong for your policy. With RAG, the system first retrieves your actual returns policy, then asks the model to answer from it, and the answer can cite where it came from.

How RAG works

Server-side or on-device?

1. Chunking

• Split by structure (headings, paragraphs), with chunks of a few hundred words and small overlaps.
• Store metadata: source title, URL or section, updated date, and who may access it.
• Re-index when documents change.

2. Retrieval

Embed the question with the same embedding model used for chunks, retrieve the top matches, filter by the user's permissions, and optionally re-rank. Hybrid keyword + vector retrieval often improves results for names and codes. See AI-Powered Search.

3. Grounded prompt with citations

data class Passage(val id: String, val title: String, val text: String)

@Serializable
data class GroundedAnswer(val answer: String, val citedIds: List<String>, val answerable: Boolean)

class GroundedAnswerer(private val model: GenerativeModel, private val json: Json) {

    suspend fun answer(question: String, passages: List<Passage>): GroundedAnswer {
        if (passages.isEmpty()) return GroundedAnswer("I couldn't find this in the help articles.", emptyList(), false)

        val prompt = buildString {
            appendLine("Answer using ONLY the passages. If they don't contain the answer, set answerable=false.")
            appendLine("Cite the ids of passages you used. Return JSON {answer, citedIds, answerable}.")
            passages.forEach { appendLine("<passage id=\"${it.id}\" title=\"${it.title}\">${it.text}</passage>") }
            appendLine("<question>${question.take(800)}</question>")
        }
        val raw = model.generateContent(prompt).text.orEmpty()
        val parsed = runCatching { json.decodeFromString<GroundedAnswer>(raw) }.getOrNull()
            ?: return GroundedAnswer("Sorry, I couldn't answer that.", emptyList(), false)

        // Keep only citations that refer to passages we actually provided.
        val validIds = passages.map { it.id }.toSet()
        return parsed.copy(citedIds = parsed.citedIds.filter { it in validIds })
    }
}

Configure the model for JSON output with a schema where possible, and keep temperature low.

The Android side

interface HelpApi {
    @POST("v1/help/answer")                     // backend does retrieval + generation
    suspend fun answer(@Body req: HelpQuestion): HelpAnswerDto
}
@Serializable data class HelpQuestion(val question: String, val locale: String)
@Serializable data class HelpAnswerDto(val answer: String, val sources: List<SourceDto>, val answerable: Boolean)
@Serializable data class SourceDto(val title: String, val url: String)
// UI: show answer, then "Sources" chips linking to the articles; offer "Contact support" when !answerable.

Security and access control

• Authorise retrieval: filter chunks by the user's permissions before they reach the prompt. The model can't be trusted to keep secrets it was shown.
• Treat documents as untrusted: retrieved text can contain injected instructions. Keep the model's abilities narrow. See Prompt Injection.
• Privacy: don't index personal data unless necessary, and support deletion.

Evaluating RAG

• Retrieval quality: for test questions, are the right passages in the top-k results?
• Answer faithfulness: are answers supported by the cited passages?
• Refusal correctness: does it say "I don't know" when the docs don't cover the question?

See Evaluating AI Responses and Reducing Hallucinations.

Use cases

Frequently asked questions

Conclusion

RAG grounds AI answers in your data: chunk and embed documents, retrieve authorised passages, prompt the model to answer only from them with citations, and evaluate retrieval and faithfulness. Run it on the server for shared knowledge and on-device for private data.

Next step: let the model take safe actions with Function Calling and Tool Use.