Protecting User Data in AI-Powered Android Apps

A "summarise my chats" feature is helpful, and it also means private conversations may travel to a cloud model. Users deserve to know, to choose, and to trust that you sent only what was needed. Privacy-respecting AI is mostly about design decisions made early.

Principles

1. Minimise input

• Send the specific text needed (one note), not the whole database.
• Use on-device OCR or extraction first, and send structured fields rather than raw photos.
• Strip image metadata (GPS, device info) before upload.
• Downscale images and truncate long inputs. That's less data and lower cost.

2. Prefer on-device processing

Gemini Nano via ML Kit GenAI processes data on the device. Android's documentation states that AICore doesn't retain inputs or outputs after processing. For supported tasks and devices, that's the most private option. See Cloud vs On-Device AI.

3. Redact before cloud calls

/** Simple pattern-based redaction. Not perfect, but reduces accidental exposure of obvious identifiers. */
object Redactor {
    private val patterns = listOf(
        Regex("""\b[\w.+-]+@[\w-]+\.[\w.]+\b""") to "[email]",
        Regex("""(?<!\d)(?:\+?91[\s-]?)?[6-9]\d{9}(?!\d)""") to "[phone]",     // Indian mobile numbers
        Regex("""\b\d{4}\s?\d{4}\s?\d{4}\b""") to "[id-number]",               // 12-digit ID formats
        Regex("""\b(?:\d[ -]*?){13,19}\b""") to "[card]",
    )

    fun redact(text: String): String = patterns.fold(text) { acc, (rx, label) -> rx.replace(acc, label) }
}

// Usage: model.generateContent(Redactor.redact(userNote))

Pattern redaction catches obvious identifiers. It won't catch names or context, so combine it with minimisation and consent rather than relying on it alone.

4. Consent and disclosure

@Composable
fun CloudAiConsentDialog(onAccept: () -> Unit, onDecline: () -> Unit) {
    AlertDialog(
        onDismissRequest = onDecline,
        title = { Text("Use cloud AI for summaries?") },
        text = {
            Text("Your device can't summarise this offline. If you continue, the note text is sent to " +
                "Google's Gemini service to create a summary. We don't store the note text on our servers. " +
                "You can turn this off anytime in Settings → AI features.")
        },
        confirmButton = { TextButton(onClick = onAccept) { Text("Allow") } },
        dismissButton = { TextButton(onClick = onDecline) { Text("Not now") } },
    )
}

• Ask before the first cloud processing of personal content, and remember the choice in DataStore.
• Make the setting reversible and visible.
• Keep your privacy policy and Play Data safety declarations accurate, including data sent to AI providers.

5. Provider terms and configuration

• Read the AI provider's data usage terms for your tier. Free and paid tiers may differ in how inputs can be used.
• Prefer enterprise or paid tiers with clear data handling for production user data, where appropriate.
• Choose regions and data residency options if regulations require them.

6. Retention and logging

• Don't log raw prompts or responses by default. Log metadata such as tokens, latency, model and error codes.
• If you store AI outputs (summaries, chats), treat them as user data: protect them, allow export and deletion, and apply retention limits.
• Make account deletion remove AI-derived data too. See AI Privacy and Data Retention.

Examples

Frequently asked questions

Conclusion

Privacy-respecting AI minimises data, prefers on-device processing, redacts before cloud calls, asks for informed consent, understands provider terms, and limits retention and logging, all reflected honestly in your disclosures.

Next step: protect your backend from abuse with Play Integrity API Explained.