AI document chatbot security checklist

• Conversations are private to their owner (and org admins by policy); collections are private or org-wide; every retrieval query is RLS-scoped and carries an explicit collection_id/org_id predicate, and the document ids in citations are re-authorised before a source link is shown, so no answer can quote a document the viewer cannot open.
• Embeddings are customer data: they never leave the tenant database, are never pooled across tenants for caching or training, and are deleted with their source document; embedding_cache is tenant-local.

• Token and USD budgets fail closed: the provider is called only after an atomic reservation succeeds against both the user and the org counters; platform default limits apply when none are configured; a missing counter, a database error or a gateway outage refuses the call instead of allowing it.
• Cost-exhaustion defences: max_output_tokens and input token caps per request, per-user concurrent stream limit, captcha or verified email before free-tier use, per-IP limits for anonymous demos, ingestion caps (documents, pages, chunks, embedding tokens) per org, and anomaly alerts on spend per user per hour.

• Indirect and direct prompt-injection defences: retrieved chunks and user uploads are wrapped as untrusted data with explicit instructions never to follow them; no side-effecting tools in phase 1; outputs strip markdown images and links that are not citations (blocks data exfiltration via URLs); a canary token in the system prompt detects leakage; documents flagged by an injection heuristic at ingest are marked and down-ranked.
• Answer rendering escapes all model output (no raw HTML), code blocks are inert, citation links open only same-origin source viewers, and a system-level refusal policy covers disallowed content with logged refusal counts.

• Prompts, documents and answers are personal data: optional PII redaction at ingest and before the provider call, provider-side retention and training opt-out required by the gateway contract, prompts and answers excluded from application logs (hashes, lengths and ids only), and export and erase endpoints for conversations and documents.

• Document uploads are untrusted: allow-list of types, size and page caps, magic-byte check, virus scan, parsing in a resource-limited sandbox with no network, zip and decompression-bomb limits, and no execution of embedded macros or scripts.

• Gateway credentials and signing secrets are env-only and server-side, never in client bundles or prompts; the model is never given secrets or environment dumps as context.

• Users are told when they are talking to AI, answers carry a 'check the sources' affordance, high-stakes domains (medical, legal, financial) require disclaimers and human-escalation paths, and data-processing terms with the model provider are recorded.

• GDPR / data protection (applies to EU or UK users): Conversations and uploaded documents are personal data and often contain third-party data: DPA with the model provider, no training on customer data, retention limits, export and erase across messages, chunks, embeddings and caches, and a documented sub-processor list.
• EU AI Act transparency (applies to EU users): Disclose that users interact with an AI system, label AI-generated content where required, and keep logs sufficient to explain answer provenance (citations, prompt version).
• Regulated data (HIPAA, financial, legal) (applies to health, finance or legal content): Do not ingest PHI or regulated data unless the provider offers a BAA/equivalent and the plan enables it; add redaction, access logging and disclaimers; this playbook does not make an app HIPAA compliant by itself.
• Copyright and content licensing (applies to third-party documents): Customers must hold rights to uploaded documents; answers quote bounded excerpts with attribution; takedown path for ingested content.
• Children (applies to minors may use the app): Age gate or restrict the product where minors may use AI chat; apply stricter content filtering.
• Accessibility: Screen-reader friendly streaming (aria-live politeness), keyboard-operable citations and WCAG 2.2 AA contrast.

• llm usd per user day: DEFAULT_USER_DAILY_USD_MICROS and budget_limits; chat refused before the provider call at 100%.
• llm usd per org month: DEFAULT_ORG_MONTHLY_USD_MICROS and budget_limits; every call refused at 100%, alert at 80%.
• tokens per request: MAX_INPUT_TOKENS and MAX_OUTPUT_TOKENS enforced by the gateway wrapper; oversize input refused, not truncated silently.
• embedding tokens per org month: per-plan embedding budget reserved per batch; ingestion fails the document closed when exhausted.
• document storage gb: per-plan quota; presign refused at 100%.
• chunks per org: MAX_CHUNKS_PER_ORG; chunking stops and the document fails closed over the cap.

Forge64 gives every app its own Postgres database and runs a quality gate that flags unauthenticated writes and secrets in client code before you see the app. It does not replace your own testing, and it holds no third-party security certification today.

This is a reference design for the category, not a promise that every item is generated today. Built-ins such as email, background jobs, file uploads and search are coming soon, so anything here that needs them is something you add yourself for now.

• Sign in as two different users and try to open each other's records by changing the id in the address.
• Submit each form with an empty value, a very long value and a script tag, and check the result.
• Call each write endpoint without signing in and check that it refuses.

What is the biggest security risk in an AI document chatbot?

Conversations are private to their owner (and org admins by policy); collections are private or org-wide; every retrieval query is RLS-scoped and carries an explicit collection_id/org_id predicate, and the document ids in citations are re-authorised before a source link is shown, so no answer can quote a document the viewer cannot open.

Does an AI document chatbot have compliance obligations?

GDPR / data protection, if you have EU or UK users: Conversations and uploaded documents are personal data and often contain third-party data: DPA with the model provider, no training on customer data, retention limits, export and erase across messages, chunks, embeddings and caches, and a documented sub-processor list. This is a starting point, not legal advice.

Will Forge64 certify my app as secure?

No. Forge64 holds no third-party security certification today and does not certify apps. The checklist is a way to test your own app.

• How to build an AI document chatbot
• AI document chatbot architecture
• Example prompt for an AI document chatbot
• AI document chatbot scaling checklist
• Often combined: SaaS dashboard
• Often combined: CRM
• All app types