EXPLAINER

Databases for AI Applications

AI apps still need ordinary databases for users, conversations, documents and evaluation rows. Vectors are an extra index for similarity — not a replacement for relational data.

Short answer: Store users, chats, documents and evals in a normal database; add vector search only for similarity. AI does not remove SQL fundamentals.

What is it?

An AI feature still runs inside an application. That application needs durable records: accounts, roles, conversation threads, uploaded files, feedback scores. Those are classic database rows. Embeddings and vector indexes help with "find similar passages"; they do not replace transactions, unique constraints or joins.

Why it matters

Teams sometimes jump to a vector database and forget sessions, billing entitlements and audit logs. Those gaps cause security and product failures unrelated to model quality.

How to do it

Typical tables:

  • users, orgs, memberships
  • conversations, messages (role, content, tokens, created_at)
  • documents, document_chunks (text, source, embedding id)
  • eval_cases, eval_runs, eval_scores

Practices:

  1. Parameterised queries only — never string-concat SQL from model output.
  2. Store message history for Agent Memory: Short-Term, Long-Term and Beyond; trim what you re-send to the model.
  3. Keep raw uploads in object storage; metadata in SQL.
  4. Index foreign keys you filter on (conversation_id, org_id).
  5. Decide retention and encryption with AI Privacy and Security: What to Know Before You Paste Data in mind.

Example

Flow for a chat product:

  1. Authenticated user opens a conversation row.
  2. Each turn inserts a message row.
  3. Server loads recent messages, calls the model, inserts the assistant row.
  4. If RAG is enabled, retrieve chunk ids, store citations beside the message.
  5. Nightly job writes eval scores for sampled threads (How to Evaluate AI Systems and Outputs).

When to use it

Introduce a real database as soon as you need history across devices or users. In-memory arrays are fine for demos only.

Practical notes

Indexing and lifecycle deserve explicit choices. Soft-delete messages if users may undo; hard-delete when retention policy requires it. Store model name and parameter snapshots on each assistant message so you can reproduce behaviour during debugging. Separate "content for the user" from "content for the model" when tool traces are noisy. For RAG, keep chunk text and embedding identity aligned with document versions so re-indexing does not orphan vectors. Use transactions when inserting a user message and enqueueing a worker job. Apply the same migration discipline you would for any SaaS app — AI features are not exempt from schema reviews. When vectors enter the picture, read Vector Database vs Traditional Database before splitting systems. Work through one real request end to end, write down the failure modes you observed, and add a regression test so the next change does not silently drop validation, auth, or logging. Prefer boring, reviewable code over a clever abstraction until the second or third product needs the same pattern. Work through one real request end to end, write down the failure modes you observed, and add a regression test so the next change does not silently drop validation, auth, or logging. Prefer boring, reviewable code over a clever abstraction until the second or third product needs the same pattern.

Common mistakes

  • Putting embeddings in the only database and losing relational integrity.
  • Letting the model invent SQL.
  • No org_id checks on message fetches (cross-tenant leaks).