Skip to content

MVP Architecture: Agentic RAG System

Ekanatha Reddy Urivakili26 min5,115 words

0. Review Notes

Implementation evidence: the test suites and checks listed in the README; operating procedures: runbooks. Performance targets below remain targets until measured on production-like hardware (§8, Phase 6); the conditional eval CI job requires configured infrastructure and credentials.

Revision 2 — architecture review

#BeforeNowWhy
15 LLM agents (Supervisor, Router, Retriever, Generator, Critic)One LangGraph state machine; only 2–3 nodes call an LLMThe graph is the supervisor. The old design made 5–8 LLM calls per query → slow, costly, hard to debug.
2Dense-only vector searchHybrid (dense + BM25 sparse) + cross-encoder rerankerBiggest retrieval-quality win per unit of effort; cheaper than agent loops.
3Unbounded Critic → Retriever/Generator loopsMax 1 query-rewrite retry; deterministic grading via reranker scoreBounded latency and cost; no infinite loops.
4Vector DB is the only storePostgres = source of truth (docs, chunks, jobs, chats); Qdrant = derived indexRe-embedding, deletes, audit, and recovery become possible without re-parsing files.
5Ingestion as a scriptAsync, idempotent jobs (content hash + deterministic point IDs)Re-uploads don't duplicate; failures retry; API stays responsive.
6"Qdrant or Chroma", "LangGraph or AutoGen", "Streamlit or Next.js"One choice each, behind thin interfacesDecisions made; vendor swaps isolated to adapters.
7No conversation handlingFollow-ups condensed into standalone queries; history in PostgresChat follow-ups ("what about the second one?") otherwise retrieve garbage.
8Evaluation in week 6Golden dataset in week 1; eval runs in CIEvery chunking/prompt/model change is measured, not guessed.
9MissingMulti-tenancy, prompt-injection handling, tracing, retries/timeoutsCheap to add now, expensive to retrofit.

Revision 3 — platform requirements

#RequirementDesignSection
10Podman for localhostcompose.yaml + Containerfile, run with podman compose; Ollama runs on the host for GPU access§7.1
11Mailpit for emailTransactional email via SMTP through a jobs-table outbox; Mailpit locally, real SMTP provider in prod via env only§4.8
12RBACTenant-scoped roles (admin, editor, viewer), permission checks as FastAPI dependencies, audit log§4.7
13Local Ollama + Anthropic + OpenAIPer-node model routing with ordered fallbacks; local qwen3.5:4b for orchestration steps, Claude for generation, OpenAI as fallback and eval judge§4.3

Revision 4 — chat history and Next.js UI

#RequirementDesignSection
14Keep a history of chats and continue any of themconversations.updated_at (last activity) drives ordering; keyset-paginated, searchable history; rename and delete; deep links /c/{id}; posting to an existing conversation continues it with its last turns as context§4.10
15Next.js UI, built with the OWASP Web and API Top 10 in mindNext.js App Router as a backend-for-frontend (BFF): tokens only in an encrypted HttpOnly cookie, allowlisted server-side proxy to the API, Origin-checked mutations, per-request nonce CSP, no raw HTML rendering§6

Revision 5 — Phase 6 hardening (20 October 2026)

#ChangeDesignSection
16Metrics and alertsPrometheus /metrics (token-protected; off in prod without a token) with per-attempt LLM metrics, llm_fallback_total{purpose,from,to}, TTFT, answer outcomes and Postgres-derived job/online-eval gauges; alert rules and a Grafana dashboard in ops/§5.3
17TracingOpenTelemetry over OTLP/HTTP, enabled by OTEL_EXPORTER_OTLP_ENDPOINT: one span per request, graph node, LLM attempt and worker job. Jaeger locally; Langfuse via its OTLP endpoint§5.3
18Online faithfulnessjudge_answer job grades a sampled share of cited answers with the eval_judge route into answer_evals; off by default§4.2, §5.4
19Load, recovery, operationsLocust load test with a TTFT gate; backup/restore script with an automated restore test; runbooks§8
20Browser end-to-end testsPlaywright suite run in CI against API + worker + production web build§6.1
21No LangGraph checkpointerThe graph runs once per turn with history loaded from messages; a checkpointer would add a write per node with nothing to resume. Revisit for human-in-the-loop interrupts§4.10, §9

1. Scope

Goal: Email-verified signup creates a private workspace; invited users join an existing workspace. Users upload documents and ask questions in a chat UI; answers are grounded in those documents, streamed, and cite sources. When the documents don't contain the answer, the system says so. Access is controlled by role.

MVP non-goals (explicitly deferred): fine-tuning, GraphRAG, web search tools, per-document ACLs (tenant-level roles only), custom roles, SSO/OIDC, agents with side-effecting tools, multi-region. OCR of images/scanned PDFs and structured receipt extraction are implemented.

Targets (MVP)

MetricTarget
p95 time-to-first-token< 3 s
Retrieval recall@8 on golden set≥ 0.85
Faithfulness (golden set)≥ 0.90
Ingestion100-page PDF searchable in < 2 min
LLM calls per query≤ 3 (typical: 2, of which 1 is local)

2. Technology Stack

ConcernChoiceNotes
Language / toolingPython 3.12, uv, ruff, mypy, pytest
APIFastAPI + Pydantic v2SSE for streaming (simpler than WebSockets, works through proxies)
OrchestrationLangGraphGraph only; LLM SDKs called directly via thin adapters (no LangChain chat-model layer)
LLM — orchestration stepsOllama qwen3.5:4b (local, host)Query analysis, rewrite, direct replies. Fallback: Claude Haiku 4.5
LLM — generationAnthropic Claude Sonnet 5.5Fallback: OpenAI (model set in config)
LLM — eval judgeOpenAI (model set in config)Different vendor from the generator avoids self-preference bias
LLM SDKsanthropic, openai, ollamaOne adapter each behind an LLM protocol (§4.3)
Embeddings (dense)text-embedding-3-small (1536-d) or an Ollama embedding model (e.g. bge-m3, 1024-d)Fixed per collection; switching = re-index via alias (§4.5)
Sparse / keywordBM25 via fastembed (CPU, local)Stored as Qdrant sparse vectors
Rerankerbge-reranker-v2-m3 (local cross-encoder)Behind a Reranker protocol
Vector DBQdrantNative hybrid search + RRF fusion, payload indexes, multi-tenancy, horizontal scaling
Relational DBPostgreSQL 16 + SQLAlchemy 2 (async) + AlembicSource of truth; job queue; chat history
Job queuePostgres jobs table + SELECT … FOR UPDATE SKIP LOCKED workerIngestion, email, re-index jobs. No extra infra for MVP
File storageS3-compatible (MinIO locally)Raw uploads, immutable
Parsingpymupdf4llm (PDF → Markdown), python-docx, trafilatura (HTML)Add Docling/Unstructured later for complex layouts
AuthEmail + password (argon2id), JWT access + rotating refresh tokens, API keys for service accountspwdlib/argon2-cffi, pyjwt
AuthorizationRBAC, 3 fixed roles per tenant, permissions in code§4.7
Emailaiosmtplib + Jinja2 templates; Mailpit locallyOutbox via jobs table (§4.8)
ObservabilityOpenTelemetry (OTLP) traces, Prometheus metrics, structured JSON logsTrace backend is any OTLP receiver: Jaeger in the local observability profile, Langfuse via its OTLP endpoint. Prometheus + Grafana in the same profile
EvaluationRetrieval metrics + RAGAS (faithfulness, answer relevancy)
UINext.js 16 (App Router) + React 19 + TypeScript, Tailwind CSS 4, TanStack QueryRuns as a BFF in front of the API (§6). Receipts, Members and Settings are implemented in Next.js
ContainersPodman (rootless) + Compose spec (compose.yaml)Same OCI images in prod

3. High-Level Design (HLD)

Two independent paths share storage: Ingestion (write path) and Query (read path). Every request passes AuthN → RBAC before reaching either.

graph LR
    User((Browser)) -->|same-origin only<br/>session cookie| Web[Next.js BFF<br/>UI + allowlisted proxy]
    Web -->|REST + SSE<br/>Bearer token| API[FastAPI<br/>AuthN + RBAC<br/>stateless, N replicas]

    subgraph Query Path
        API --> Graph[LangGraph RAG Graph]
        Graph --> Ret[Hybrid Retriever + Reranker]
        Graph --> MR[Model Router]
    end

    subgraph LLM Providers
        MR --> Ollama((Ollama<br/>local host))
        MR --> Anthropic((Anthropic API))
        MR --> OpenAI((OpenAI API))
    end

    subgraph Background Worker
        Worker[Worker<br/>N replicas] -->|claim job| PG
        Worker --> Obj
        Worker -->|upsert| VDB
        Worker -->|SMTP| Mail[Mailpit local /<br/>SMTP provider prod]
    end

    API -->|upload| Obj[(Object Storage<br/>S3/MinIO)]
    API -->|enqueue jobs| PG
    Ret --> VDB[(Qdrant<br/>dense + sparse)]
    Graph --> PG[(PostgreSQL<br/>users, roles, docs, chunks,<br/>jobs, chats, audit)]

Layering inside the codebase (dependencies point inward):

api (routers, auth deps) ──> use cases (graph, ingestion, auth, email) ──> domain models
                                        │
                                        └──> adapters (LLM, Embedder, VectorStore, Reranker,
                                                       ObjectStorage, Mailer, DB repositories)

Only vendor-swap points get a Protocol: LLM, Embedder, VectorStore, Reranker, Mailer. Nothing else is abstracted.


4. Low-Level Design (LLD)

4.1 Ingestion Pipeline

sequenceDiagram
    participant U as User/UI
    participant A as API
    participant S as Object Storage
    participant P as Postgres
    participant W as Worker
    participant Q as Qdrant

    U->>A: POST /v1/documents (file)
    A->>A: authN, require(document:write), validate type/size, sha256
    alt hash already ingested for tenant
        A-->>U: 200 {document_id, status: "ready"} (no-op)
    else new or changed
        A->>S: put raw file
        A->>P: insert document(status=queued) + job(type=ingest_document)
        A-->>U: 202 {document_id, job_id}
    end
    W->>P: claim job (FOR UPDATE SKIP LOCKED)
    W->>S: get raw file
    W->>W: parse → markdown → chunk
    W->>P: insert chunks (text, metadata)
    W->>W: embed dense (batched) + sparse (BM25)
    W->>Q: delete points where doc_id = X (previous version)
    W->>Q: upsert points (deterministic IDs)
    W->>P: document.status = ready

Steps & rules

  1. Parse to Markdown so headings survive.
  2. Chunk — structure-aware: split on headings first, then recursively by tokens.
    • Start at ~500 tokens, 50 overlap; tune using the eval set.
    • Prepend the heading path to each chunk's embedded text ("Guide > Install > Linux\n\n<chunk>").
  3. Embed in batches (e.g. 64) with retry/backoff on 429/5xx.
  4. Idempotency
    • Document dedupe: (tenant_id, content_hash) unique.
    • Point ID: uuid5(NAMESPACE, f"{document_id}:{version}:{chunk_index}") — re-running a job overwrites, never duplicates.
    • Re-upload of a changed doc: bump version, delete old points by doc_id, upsert new.
  5. Failure handling — attempts incremented with exponential backoff; after 3 → status=failed, error surfaced in UI, and a send_email job notifies the uploader.
  6. Delete — removes Qdrant points by filter, then chunks, then marks document deleted (raw file purged asynchronously). Recorded in audit_log.

4.2 Query Graph (Agentic / Corrective RAG)

graph TD
    Start([question + history]) --> QA[analyze_query<br/>local Ollama, structured output]
    QA -->|needs_retrieval = false| Direct[respond_direct<br/>local Ollama]
    QA -->|needs_retrieval = true| R[retrieve<br/>hybrid + rerank, no LLM]
    R --> G{grade<br/>top rerank score ≥ threshold?}
    G -->|yes| Gen[generate<br/>Claude, streamed, cited]
    G -->|no, attempts < 1| RW[rewrite_query<br/>local Ollama]
    RW --> R
    G -->|no, attempts exhausted| NA[insufficient_context<br/>“I couldn't find this in your documents”]
    Gen --> V[validate_citations<br/>deterministic]
    V --> End([answer + citations])
    Direct --> End
    NA --> End
NodeModel routeResponsibility
analyze_queryorchestrationOne call returns {standalone_query, needs_retrieval, filters}. Condenses follow-ups using last N turns.
respond_directorchestrationGreetings, "what can you do", polite out-of-scope refusal. No document content.
retrieve—Qdrant hybrid query (dense + sparse prefetch, RRF, top 40) → rerank → top 8. Tenant filter always applied.
grade—Threshold on reranker score (tuned on eval set).
rewrite_queryorchestrationReformulates once (synonyms / HyDE-style). Bounded to 1 retry.
generategenerationAnswers only from numbered context blocks; cites [1], [2]. Streams tokens.
validate_citations—Drops citation markers that don't map to a retrieved chunk; attaches source metadata.
insufficient_context—Fixed honest response + suggestion to upload relevant docs.

Graph state

class RAGState(TypedDict):
    tenant_id: str
    user_id: str
    conversation_id: str
    question: str
    history: list[ChatTurn]          # last N turns, loaded from Postgres
    standalone_query: str
    needs_retrieval: bool
    filters: dict[str, str]
    chunks: list[ScoredChunk]
    attempts: int
    answer: str
    citations: list[Citation]
    models_used: list[ModelUsage]    # provider, model, tokens, latency, fallback flag

Why no LLM Critic in the hot path? It doubles latency and cost. Faithfulness is measured offline (eval suite, OpenAI judge) and on a sampled % of production traffic asynchronously: when ONLINE_EVAL_SAMPLE_RATE > 0, a judge_answer job is queued in the same transaction as a cited answer; the worker reloads the exact chunks (by document, version and index, in citation order) and stores the grade in answer_evals. It is skipped if a source was deleted in between, since citation numbers would shift. If eval shows hallucination problems, add an inline groundedness check behind a feature flag.

Extending to multi-agent later: new capabilities (SQL agent, web search, summarization) are added as subgraphs selected by analyze_query's output, each with its own model route. The state contract and API do not change.

4.3 LLM Provider Layer & Model Routing

Three providers, one interface, routing by purpose (not by node), with ordered deployment fallbacks. Interactive chat overrides both orchestration and generation with the user's selected model; OpenAI is selected by default when configured. Personal connections are scoped to user and tenant, with encrypted write-only provider keys. Selection is request-scoped and remembered on the conversation, including its background summaries. See chat model selection for the API, encryption, and deployment requirements.

class LLM(Protocol):
    provider: str
    model: str

    async def complete(
        self,
        messages: list[Message],
        *,
        schema: type[BaseModel] | None = None,   # structured output when set
        max_tokens: int,
        temperature: float = 0.0,
    ) -> Completion: ...                          # text | parsed, usage, latency

    def stream(self, messages: list[Message], *, max_tokens: int) -> AsyncIterator[str]: ...
AdapterSDKStructured outputNotes
OllamaLLMollama (native /api/chat)format=<JSON schema>think=False for Qwen3.x on orchestration calls (latency); keep_alive set so the model stays loaded
AnthropicLLManthropicStructured outputs / tool schemaPrompt caching on static system prompt
OpenAILLMopenaiStructured outputs (response_format JSON schema)

Every structured response is re-validated with Pydantic regardless of provider — small local models occasionally emit invalid JSON.

Routing config (config/models.yaml, overridable by env):

providers:
  ollama:    { base_url: "${OLLAMA_BASE_URL}" }        # http://host.containers.internal:11434 in containers
  anthropic: { api_key_env: ANTHROPIC_API_KEY }
  openai:    { api_key_env: OPENAI_API_KEY }

routes:  # first entry = primary, rest = fallbacks in order
  orchestration:
    - { provider: ollama,    model: "qwen3.5:4b",        timeout_s: 8 }
    - { provider: anthropic, model: "claude-haiku-4-5",  timeout_s: 8 }
  generation:
    - { provider: anthropic, model: "claude-sonnet-5-5", timeout_s: 60 }
    - { provider: openai,    model: "${OPENAI_GENERATION_MODEL}", timeout_s: 60 }
  eval_judge:
    - { provider: openai,    model: "${OPENAI_JUDGE_MODEL}", timeout_s: 60 }

ModelRouter.for_purpose("orchestration") returns an LLM that tries each entry in order and falls through on: timeout, connection error (e.g. Ollama not running), 429/5xx after retries, or a structured output that fails Pydantic validation twice. Fallbacks are logged and counted (llm_fallback_total{purpose,from,to}).

Rules:

  • Streaming fallback happens only before the first token is sent; after that, errors surface as an SSE error event (no mixing two models in one answer).
  • Prompts are provider-neutral (plain system + user messages; no vendor-specific tags required for correctness).
  • Swapping a model is a config change, then an eval run — the eval suite can run the generation route against each provider and report the results side by side.
  • Sensitive tenants can later be pinned to a local-only route (generation: [ollama/...]) — no code change.

Why support local Ollama for orchestration? These calls are short, structured, and on the critical path before generation: a local 4B model on Apple Silicon can keep raw user questions local. Deployment routes retain cloud fallbacks; a chat selection instead pins all chat model calls to the chosen provider without fallback.

4.4 Prompting rules (generation)

  • Retrieved text is wrapped in delimited blocks (<doc id="1" source="...">…</doc>) and the system prompt states it is data, not instructions (prompt-injection mitigation).
  • "Answer only from the documents. If they don't contain the answer, say so."
  • Prompts live in versioned files under graph/prompts/; prompt version is logged on every trace.

4.5 Data Model

PostgreSQL (source of truth)

-- Identity & access
tenants          (id, name, created_at)
users            (id, email UNIQUE, password_hash, is_active, last_login_at, created_at)
memberships      (user_id, tenant_id, role[admin|editor|viewer], created_at)
                  PK (user_id, tenant_id)
email_tokens     (id, type[invite|password_reset], email, tenant_id NULL, role NULL,
                  token_hash, expires_at, used_at, created_by, created_at)
refresh_tokens   (id, user_id, token_hash, expires_at, revoked_at, replaced_by, created_at)
api_keys         (id, tenant_id, name, role, key_prefix, key_hash, last_used_at,
                  revoked_at, created_by, created_at)
audit_log        (id, tenant_id, actor_user_id, action, target_type, target_id,
                  metadata JSONB, ip, created_at)

-- Content
documents        (id, tenant_id, uploaded_by, title, source_uri, mime_type, content_hash,
                  version, status[queued|processing|ready|failed|deleted],
                  error, created_at, updated_at)
                  UNIQUE (tenant_id, content_hash)
chunks           (id, document_id, version, chunk_index, text, heading_path,
                  page_start, page_end, token_count)

-- Background work
jobs             (id, type[ingest_document|send_email|reindex|purge_document|judge_answer|summarize_conversation], payload JSONB,
                  status[queued|running|done|failed], attempts, run_after,
                  locked_at, finished_at, error, created_at)

-- Chat
conversations    (id, tenant_id, user_id, title, created_at, updated_at, summary, summarized_count)
                  INDEX (tenant_id, user_id, updated_at, id)      -- history list + keyset paging
messages         (id, conversation_id, role, content, citations JSONB, trace_id,
                  prompt_version, provider, model, fallback_used, tokens_in,
                  tokens_out, latency_ms, debug JSONB, created_at)
                  INDEX (conversation_id, created_at)             -- latest window + "load earlier"
feedback         (message_id, user_id, rating SMALLINT, comment, created_at)
answer_evals     (message_id PK → messages ON DELETE CASCADE, faithfulness, relevancy,
                  judge_provider, judge_model, created_at)        -- sampled online grading

All tokens (invite, reset, refresh, API key) are stored hashed (SHA-256 of a 32-byte random secret); the plaintext is shown/sent once.

Qdrant (derived index — fully rebuildable from chunks)

  • Collection chunks_v{n} accessed via alias chunks_current → zero-downtime re-embedding (e.g. switching to an Ollama embedding model): build chunks_v2, flip alias, drop v1.
  • Vectors: dense (size from config, cosine), sparse (BM25, IDF modifier).
  • Payload indexes: tenant_id (keyword, is_tenant=true), doc_id, category.
  • Payload:
{
  "tenant_id": "t_123",
  "doc_id": "uuid",
  "doc_version": 2,
  "chunk_index": 14,
  "text": "chunk text…",
  "heading_path": "Guide > Install > Linux",
  "source": "install-guide.pdf",
  "page": 7,
  "category": "technical_spec",
  "embedding_model": "text-embedding-3-small"
}

Single collection with tenant_id partitioning; the filter is enforced inside the VectorStore adapter so no caller can forget it.

4.6 API Contract (v1)

MethodPathPermissionDescription
POST/v1/auth/loginpublicEmail + password → access token + refresh cookie/token
POST/v1/auth/refreshrefresh tokenRotate refresh token, new access token
POST/v1/auth/logoutauthenticatedRevoke refresh token
POST/v1/auth/password/forgotpublicAlways 202 (no account enumeration); emails reset link if user exists
POST/v1/auth/password/resetreset tokenSet new password; revokes all refresh tokens
POST/v1/invitationsmember:manage (user only){email, role} → emails invite link
POST/v1/invitations/acceptinvite tokenCreate user, or attach an existing account after verifying its current password; + membership. 5 attempts per link per hour
DELETE/v1/invitations/{id}member:manage (user only)Revoke a pending invitation (tenant-scoped, audited)
GET/v1/meauthenticatedUser, current tenant, role, permissions (UI uses this to show/hide actions)
PATCH/v1/meauthenticated (user only){name} update profile display name
GET/v1/membersmember:manage (user only)List members
PATCH/v1/members/{user_id}member:manage (user only)Change role (cannot demote the last admin)
DELETE/v1/members/{user_id}member:manage (user only)Remove membership
GET/POST/DELETE/v1/api-keysapikey:manage (user only)Service-account keys with a role (human admin session required to prevent key self-escalation)
GET/v1/audit-logaudit:readPaginated
POST/v1/documentsdocument:writeMultipart upload → 202 {document_id, job_id}
GET/v1/documents, /v1/documents/{id}document:readList / status, error, chunk count
DELETE/v1/documents/{id}document:deleteRemove doc + vectors
POST/v1/receiptsdocument:writeMultipart upload of receipt PDF/image → 202 {document_id, job_id}
GET/v1/receipts, /v1/receipts/{id}document:readPaginated extracted receipts / receipt detail
POST/v1/chatchat:use (user only){conversation_id?, message, filters?} → SSE stream. With conversation_id, continues that conversation
GET/v1/conversations?limit&q&before&before_idchat:use (own only)History, most recently active first; title search; keyset cursor (updated_at, id)
GET/v1/conversations/{id}?limit&beforechat:use (own only)Latest limit messages (≤ 500) oldest-first + has_more; before loads earlier ones
PATCH/v1/conversations/{id}chat:use (own only){title} rename
DELETE/v1/conversations/{id}chat:use (own only)Hard delete; messages and feedback cascade
POST/v1/messages/{id}/feedbackchat:use (own only){rating, comment?}
GET/healthz, /readyzpublicReadiness checks Postgres + Qdrant; reports Ollama reachability (degraded, not failed)
GET/metricsMETRICS_TOKEN bearerPrometheus exposition. Without a token: open in dev/test, 404 in prod. Not proxied by the BFF

SSE events from /v1/chat

event: status     data: {"step": "retrieving"}
event: token      data: {"text": "The install "}
event: citations  data: [{"n":1,"doc_id":"…","source":"install-guide.pdf","page":7,"snippet":"…"}]
event: done       data: {"message_id":"…","conversation_id":"…"}
event: error      data: {"code":"llm_unavailable","message":"…"}

tenant_id and user_id come from the token, never from the request body.

4.7 Authentication & RBAC

Roles and permissions (tenant-scoped)

Permissionviewereditoradmin
chat:use (own conversations only)✓✓✓
document:read (list docs, see sources)✓✓✓
document:write (upload, re-index)✓✓
document:delete✓✓
member:manage (invite, change role, remove)✓
apikey:manage✓
audit:read✓
  • Roles and their permission sets are defined in code (Role and Permission enums + a ROLE_PERMISSIONS map). No roles table, no policy engine — move to DB-defined roles only when customers need custom roles.
  • A user can belong to several tenants with different roles. The access token carries sub (user) and tid (active tenant); switching tenant issues a new token.
  • The role is loaded from memberships on every request (one indexed lookup), not trusted from the JWT — role changes and removals take effect immediately.
  • Enforcement is a FastAPI dependency on each route:
@router.post("/v1/documents", status_code=202)
async def upload(
    file: UploadFile,
    ctx: Annotated[RequestContext, Depends(require(Permission.DOCUMENT_WRITE))],
) -> UploadAccepted: ...
  • RequestContext(user_id, tenant_id, role, api_key_id) is passed into use cases and repositories; every tenant-owned query filters by ctx.tenant_id. Conversations additionally filter by ctx.user_id — admins cannot read other users' chats. Service-account API keys have user_id=None and api_key_id=<uuid>, and are rejected (403) on user_only=True routes (members, invitations, api-keys, chat, conversations, PATCH /v1/me, auth/switch-tenant, auth/password/change, auth/logout).
  • Guardrails: the last admin of a tenant cannot be demoted or removed; users cannot change their own role.
  • Audit log records: login failures (rate-limited), invites, role changes, member removal, API key create/revoke, document delete.

Authentication

  • Passwords: argon2id; min length 12; login rate-limited per IP + email.
  • Access JWT: 15 min, HS256 (secret from env) — switch to RS256 when other services verify tokens.
  • Refresh token: 14 days, opaque, rotated on every use, stored hashed; reuse of a rotated token revokes the whole chain.
  • API keys: rk_<prefix>_<secret>, tenant-bound with a role; same require(...) path.
  • Bootstrap: rag admin create --email … --tenant … CLI creates the first tenant + admin (no open sign-up).

4.8 Email (Mailpit locally)

EmailTriggerLink expiry
InvitationAdmin invites a member72 h, single use
Password reset/v1/auth/password/forgot30 min, single use
Ingestion failedJob reaches terminal failed—
  • Outbox pattern: the API inserts a send_email job in the same transaction as the business change (e.g. the email_tokens row). The worker sends it with retries. No lost emails on SMTP failures; no emails for rolled-back transactions; requests never block on SMTP.
  • Mailer protocol with one SmtpMailer (aiosmtplib). Jinja2 templates under email/templates/ (HTML + plain-text versions).
  • Links are built from PUBLIC_UI_URL; the token travels in the URL fragment or POST body, never logged.
  • Config (env only, no code change between environments):
SMTP_HOST=mailpit   SMTP_PORT=1025   SMTP_TLS=false   SMTP_FROM="Moontig <no-reply@localhost>"   # local
SMTP_HOST=<provider> SMTP_PORT=587   SMTP_TLS=true    SMTP_USER/SMTP_PASSWORD from secrets  # prod
  • Mailpit UI at http://localhost:8025 to view sent mail. Integration tests assert on emails via Mailpit's REST API (/api/v1/messages) — e.g. "invite → read link from Mailpit → accept → login".

4.9 Project Structure

rag-mvp/
├── pyproject.toml
├── compose.yaml                # Podman: api, worker, web, postgres, qdrant, objectstore, mailpit
├── Containerfile               # one image; api / worker / ui chosen by command
├── .env.example
├── config/models.yaml          # model routes + fallbacks
├── ops/                        # prometheus.yml + alerts.yml, Grafana provisioning + dashboard
├── loadtest/locustfile.py      # Locust load test, TTFT p95 gate
├── scripts/backup.py           # backup / restore / restore test (Postgres, Qdrant, object storage)
├── src/rag/
│   ├── api/                    # routers, schemas, SSE, error handlers
│   ├── auth/                   # passwords, jwt, refresh tokens, api keys, rbac (Role, Permission, require)
│   ├── core/                   # config (pydantic-settings), logging, errors, metrics, telemetry (OTel)
│   ├── domain/                 # Document, Chunk, Citation, ChatTurn, RequestContext (pure Pydantic)
│   ├── ingestion/              # parsers, chunker, pipeline
│   ├── retrieval/              # hybrid search + rerank + grading threshold
│   ├── graph/                  # state, nodes, graph builder, prompts/
│   ├── evaluation/             # LLM judge shared by evals and the judge_answer job
│   ├── email/                  # templates/, send_email job handler
│   ├── worker/                 # job loop, handlers (ingest_document, send_email, reindex, purge_document, judge_answer, summarize_conversation)
│   ├── adapters/
│   │   ├── llm/                # base.py (LLM protocol), ollama.py, anthropic.py, openai.py, router.py
│   │   ├── embedder.py, vectorstore.py, reranker.py, storage.py, mailer.py
│   ├── db/                     # SQLAlchemy models, repositories, alembic migrations
│   └── cli.py                  # admin bootstrap, reindex
├── web/                        # Next.js UI + BFF (§6)
│   ├── Containerfile           # standalone build, non-root, read-only root FS
│   └── src/
│       ├── proxy.ts            # per-request CSP nonce, HSTS, token refresh, page auth redirect
│       ├── app/api/auth/[action]/route.ts   # login/signup/reset/invite/switch/logout → session cookie
│       ├── app/api/v1/[...path]/route.ts    # allowlisted proxy to the API (JSON, multipart, SSE)
│       ├── app/(auth)/…        # sign in, verify signup, reset password, accept invite
│       ├── app/(app)/…         # chat, documents, receipts, members, settings
│       ├── components/         # workspace shell, chat, documents, receipts, members, keys/audit
│       └── lib/                # session (JWE cookie), security (CSRF, allowlist, IP), upstream, SSE parser
│   └── e2e/                    # Playwright browser end-to-end tests (Mailpit-backed auth flows)
├── evals/
│   ├── golden.jsonl            # {question, expected_doc_ids, reference_answer}
│   └── run_eval.py             # --route generation --provider anthropic|openai|ollama
└── tests/                      # unit, integration (testcontainers on Podman), RBAC matrix tests

4.10 Chat History & Continuation

Every chat is a conversation owned by one user in one tenant. The history list and "continue where I left off" are built on Postgres alone; the graph stays stateless.

sequenceDiagram
    participant B as Browser
    participant W as Next.js BFF
    participant A as API
    participant P as Postgres
    participant G as RAG graph

    B->>W: open /c/{id}
    W->>A: GET /v1/conversations/{id}?limit=200
    A->>P: own conversation? (tenant_id, user_id) → latest 200 messages
    A-->>B: messages (oldest first) + has_more
    B->>W: POST /api/v1/chat {conversation_id, message}
    W->>A: POST /v1/chat (Bearer, X-Forwarded-For)
    A->>P: own conversation? bump updated_at, load last N turns, insert user message (commit)
    A->>G: question + history
    G-->>B: SSE status / model / token / answer / citations
    A->>P: insert assistant message, bump updated_at
    A-->>B: SSE done {message_id, conversation_id}
ConcernDecision
Orderingupdated_at = last message activity (user turn or assistant answer). Renaming does not reorder. Backfilled from max(messages.created_at) in migration 0004.
Paging the listKeyset cursor (updated_at, id) (before, before_id), never OFFSET: stable while chats are continued or deleted between page loads, O(log n) on the (tenant_id, user_id, updated_at, id) index.
SearchCase-insensitive title match (ILIKE, wildcards escaped), ≤ 100 chars. Full-text search over message bodies is deferred until users ask for it (needs a tsvector index on messages).
Opening a long chatLatest limit messages (default 200, max 500) plus has_more; "Load earlier" pages backwards by created_at. Bounded response size (OWASP API4).
ContinuingPOST /v1/chat with conversation_id. The graph receives the last HISTORY_TURNS messages; analyze_query condenses them into a standalone query, so follow-ups ("and the second one?") retrieve correctly no matter how old the chat is.
Context growthOnly a fixed window of turns is sent to models, so cost and latency don't grow with chat length. Messages that slide out of the window are folded into conversations.summary by the summarize_conversation worker job (one orchestration call per slide, after the answer is saved, so it adds no latency). analyze_query and respond_direct receive the summary as delimited, escaped data.
Rename / deletePATCH {title}; DELETE is a hard delete with ON DELETE CASCADE to messages and feedback (chat content is the user's data, not an audit record).
AccessEvery query filters by tenant_id and user_id; another user's or tenant's ID is a 404, identical to a missing one. Admins cannot read, rename, delete or continue other users' chats. Covered by the RBAC matrix and tests/integration/test_conversations.py.
Workspace switchConversations are tenant-scoped; switching workspace shows that workspace's history only.
Interrupted answersThe user turn is committed before streaming. If the stream fails or the user stops it, any streamed text is saved as a partial assistant turn (debug.partial); consecutive same-role turns are merged before they reach a model, so strict alternating-role APIs never reject the history.

5. Cross-Cutting Concerns

5.1 Reliability

  • Timeouts on every external call (per-route LLM timeouts in models.yaml; embeddings 10 s; Qdrant 2 s; SMTP 10 s).
  • Retries with exponential backoff + jitter (tenacity) on 429/5xx/connection errors only; then model fallback (§4.3).
  • Bounded loops: graph recursion_limit set; max 1 rewrite.
  • Graceful degradation: Ollama down → cloud fallback; reranker failure → RRF results; sparse failure → dense only. All logged as warnings with metrics.
  • Idempotent jobs: ingestion via deterministic IDs; emails via job ID as Message-ID dedupe key; terminal failed state after max attempts.
  • Stateless API: all state in Postgres/Qdrant. The graph is stateless per turn (history is reloaded from messages), so no LangGraph checkpointer is needed.
  • Rate limiting per tenant on /v1/chat and uploads; per IP + email on login and password reset.

5.2 Security

  • AuthN/RBAC as in §4.7; tenant isolation enforced in repositories and the VectorStore adapter.
  • RBAC matrix test: a parametrized test hits every route with every role and asserts allowed/denied — new routes without a require(...) fail CI.
  • Upload validation: allow-listed MIME types (sniffed, not trusted from header), size cap (e.g. 25 MB), filename sanitization.
  • Prompt injection: retrieved content delimited and labelled as data; no side-effecting tools in MVP.
  • Data egress: interactive chat sends questions and relevant document excerpts to the selected provider; the model picker identifies the provider before sending. Background receipt extraction, embeddings, and optional evaluation retain their deployment routes. Selecting Ollama for chat does not change those independent services.
  • Secrets (provider API keys, JWT secret, SMTP creds) via env / secret manager; never returned to the UI; .env git-ignored.
  • No passwords, tokens, prompts or document text in application logs (traces in Langfuse with access control).
  • Browser-facing controls (BFF token custody, CSRF, CSP, XSS-safe rendering) are in §6.3.

5.3 Observability

  • Traces (OpenTelemetry, OTLP/HTTP; on when OTEL_EXPORTER_OTLP_ENDPOINT is set): one trace per request with a span per graph node (rag.node.*) and per LLM attempt (llm.complete/llm.stream with gen_ai.* provider, model and token attributes, outcome and route position), plus one span per worker job. Spans carry identifiers, scores and token counts, never prompts or document text. trace_id is on every JSON log line and stored on each assistant message. Per-message SSE spans are suppressed so a streamed answer is one span.
  • Per message (Postgres): provider, model, fallback_used, prompt version, tokens, latency, and in dev the graph path and rerank scores.
  • Metrics (GET /metrics, Prometheus): http_requests_total / http_request_duration_seconds by route template; llm_requests_total{purpose,provider,model,outcome}, llm_request_duration_seconds, llm_tokens_total, llm_fallback_total{purpose,from,to} (to="none" = route exhausted); rag_time_to_first_token_seconds, rag_answer_duration_seconds, rag_answers_total{outcome} (answered, direct, insufficient_context, error), rag_top_rerank_score, rag_retrieval_degraded_total{kind}, rag_feedback_total{rating}. Job backlog/age/terminal failures and 24 h online faithfulness/relevancy are read from Postgres at scrape time, so every replica reports the same values (aggregate with max). The worker serves its own registry on :8001/metrics (WORKER_METRICS_PORT, 0 disables; internal network only) so llm_* for the receipt_extraction and eval_judge routes and rag_ingestion_documents_total{kind,outcome} / rag_ingestion_duration_seconds{kind} are scraped; Prometheus scrapes worker:8001.
  • Dashboards/alerts (ops/): API down, 5xx ratio (excluding health checks and scrapes), p95 TTFT > 3 s, chat error rate, fallback rate per route (rising = Ollama or a provider is unhealthy), exhausted routes, insufficient_context rate, thumbs-down rate, job failures by completion time and stale backlog, online faithfulness < 0.90. Each alert maps to a runbook. The local profile evaluates rules only; notification delivery requires a separately configured Alertmanager and receivers.

5.4 Evaluation

  • Golden set (50–100 real questions) with expected source documents.
  • Retrieval metrics (deterministic): recall@k, MRR — on every change to chunking, embeddings, or retrieval.
  • Generation metrics: faithfulness + answer relevancy from a small LLM judge (rag.evaluation.judge, RAGAS-style definitions) on the eval_judge route (OpenAI). The same judge grades sampled production answers (§4.2).
  • Provider comparison: same golden set run through each generation candidate (Claude, OpenAI, a local Ollama model) → quality, latency, and cost table that drives the routing config.
  • Orchestration check: accuracy of analyze_query (needs_retrieval + standalone query) for the local model vs. the fallback — confirms the local 4B model is good enough.
  • Runs in CI on PRs touching ingestion/, retrieval/, graph/, config/models.yaml; fails on regression beyond tolerance.

6. UI (Next.js BFF)

The UI is a Next.js 16 App Router app (web/) that also acts as the backend-for-frontend. The browser only ever talks to its own origin; the BFF holds the API tokens and calls FastAPI server-to-server. The API stays the enforcement point for authN, RBAC, tenancy and rate limits; the BFF adds browser-facing defences and never widens what a user can do.

graph LR
    B((Browser)) -->|"pages, /api/auth/*, /api/v1/*<br/>cookie: __Host-rag_session (JWE)"| P[proxy.ts<br/>CSP nonce · HSTS<br/>token refresh · login redirect]
    P --> RSC[Server components<br/>GET /v1/me]
    P --> AUTH[/api/auth/action<br/>zod-validated → set/clear cookie/]
    P --> PX[/api/v1/...path<br/>allowlist · Origin check<br/>stream JSON, multipart, SSE/]
    RSC -->|Bearer| API[FastAPI]
    AUTH -->|tokens never leave the server| API
    PX -->|Bearer + X-Forwarded-For| API

6.1 Pages

The shell loads /v1/me server-side and shows/hides actions by permission (UX only).

RouteWhoContents
/loginpublicTabs: sign in, sign up (email verification), reset password
/verify_signup, /reset_password, /accept_invitepublic (link token)Token read server-side from ?token=, stripped from the address bar on load; set password / workspace
/chat:useNew chat: history sidebar + empty thread
/c/{id}chat:use (own only)Past conversation: latest 200 messages, "Load earlier", composer continues it. Streamed answers, status/model line, [n] citations with Sources, 👍/👎 feedback, Stop button
/documentsdocument:readLive status table (polls only while ingesting); upload/delete with document:write/document:delete
/receiptsdocument:readPaginated extracted receipts, details/line items/totals/payment/warnings; upload via Documents
/membersmember:manageEmail invitations, pending invitations with confirmed revocation, paginated member list, role changes, confirmed removal; own role/removal disabled
/settingsapikey:manage and/or audit:readPaginated keys, creation with one-time secret display/copy/dismiss, confirmed revocation, audit history with keyset pagination

History sidebar: "New chat", debounced title search, groups (Today / Yesterday / Previous 7 days / Older), keyset "Load more", inline rename, two-step delete. The first answer in a new chat moves the URL to /c/{id} without re-fetching.

Workspace navigation shows Chat/Documents/Receipts to viewers and editors; Members and Settings appear when the user has their management/audit permissions. Direct page access also shows a permission message when denied, while the API remains the authorization boundary. Admin lists use 25-row pages; audit history uses an ID cursor. Receipt amounts are displayed as returned decimal strings with currency, and warnings remain visible. API-key secrets are held only in transient component state after creation and can be copied/dismissed; list responses contain only prefixes.

Implementation verification (20 October 2026): 72 frontend unit tests and 170 backend tests pass; the production Podman web image builds and runs locally. Browser end-to-end tests (web/e2e, Playwright) cover signup with email verification, sign-out/sign-in, wrong password, invite → accept → role-limited navigation, API key create/dismiss/revoke, CSP and hardening headers, CSRF and proxy allowlist, and (opt-in, E2E_CHAT=1, needs a model) a streamed chat that moves to /c/{id} and persists. CI runs them against the API, worker and production web build; the chat spec runs locally only. scripts/smoke_workspace.py still exercises the live BFF/API/email/role/key/receipt/audit paths.

6.2 Session and token handling

  • Login (POST /api/auth/login): the BFF calls /v1/auth/login, then stores {access, refresh, access_exp} in one cookie encrypted with JWE dir + A256GCM (key = SHA-256 of SESSION_SECRET). Response body is {ok: true}. Browser JavaScript cannot read the tokens; XSS could still perform actions using the session.
  • Cookie: HttpOnly, SameSite=Strict, Path=/, 14-day max-age; Secure + __Host- prefix whenever WEB_ORIGIN is https (prod requires it).
  • Refresh: proxy.ts refreshes when the access token has < 60 s left, rewrites the cookie on both the forwarded request and the response, so server components and route handlers see the new token. Refresh tokens rotate and reuse revokes the whole family (§4.7), so concurrent requests holding the same expiring cookie share one in-flight refresh, and the result is reused for 60 s. That cache is per process: before scaling out, run web replicas with sticky sessions or set the API's REFRESH_REUSE_GRACE_S (e.g. 10), which lets a just-rotated token refresh again within that window while its successor is still live instead of revoking the family. Logout and family revocation still win. API unreachable ≠ invalid: only a 401 from /v1/auth/refresh clears the cookie.
  • Logout / workspace switch: revoke the refresh family at the API, then clear/replace the cookie. Switching workspace revokes the old workspace's session.

6.3 OWASP mapping (Web 2021 / API 2023)

RiskControl in web/
A01 / API1, API5 Broken access controlNo authorization logic in the UI; the API checks tenant, owner and role on every call. The BFF proxies only an explicit (method, path) allowlist with UUID-shaped IDs (lib/security.ts); member/invitation/key/audit routes are allowlisted for the implemented pages; the API enforces their permissions. Path segments are restricted to [A-Za-z0-9-] (no traversal / encoded slashes).
A01 CSRFSameSite=Strict cookie and every non-GET BFF request must carry Origin == WEB_ORIGIN (fallback Sec-Fetch-Site: same-origin); otherwise 403. GETs are side-effect free.
A02 Cryptographic failuresTokens only inside an AES-256-GCM JWE cookie; HSTS (2 years) and upgrade-insecure-requests when https; Secure cookies; prod refuses a non-https WEB_ORIGIN.
A03 Injection / XSSReact escaping everywhere; ESLint bans dangerouslySetInnerHTML. Model output is rendered with react-markdown with raw HTML skipped, images disallowed, javascript: URLs stripped, links rel="noopener noreferrer nofollow". Per-request nonce CSP with strict-dynamic, object-src 'none', base-uri 'none', frame-ancestors 'none', img-src 'self' data: blob: (blocks prompt-injection exfiltration through remote images), connect-src 'self'.
A04 Insecure designBFF pattern instead of tokens in localStorage; strict zod schemas (strictObject, unknown keys rejected) on every auth input; validation errors never echo values.
A05 / API8 MisconfigurationpoweredByHeader: false, no production source maps, nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer (link tokens never leak via Referer), COOP/CORP, restrictive Permissions-Policy. Container: non-root, read-only root FS, cap_drop: ALL, no-new-privileges, bound to 127.0.0.1. Error pages show no details.
A06 Vulnerable componentspackage-lock.json; npm audit --omit=dev --audit-level=high in CI.
A07 / API2 Auth failuresRotating refresh tokens with reuse detection preserved (single-flight refresh); logout revokes server-side; 401 anywhere sends the user to sign-in and drops cached data.
A08 IntegrityLockfile installs (npm ci --ignore-scripts in the image); no third-party scripts or CDNs.
A09 LoggingBFF logs no bodies, tokens or cookies; the API's request ID is passed through (X-Request-ID).
A10 / API7 SSRFThe upstream origin is fixed server config (API_URL); user input only selects an allowlisted path, never a host.
API4 Resource consumptionAuth and proxied JSON bodies capped at 16 KB; uploads at 30 MB (actual stream bytes, including chunked bodies). Queries over 2 KB return 414. Upstream timeout 30 s (chat: 5 min); browser disconnects cancel upstream work. Forwarded IPs are ignored by default (TRUSTED_PROXY_HOPS=0), so in dev users share the BFF IP bucket; ENV=prod requires TRUSTED_PROXY_HOPS>=1. Password reset and signup verification are also limited per link token. API refresh is limited to 120 attempts/IP/minute by default.

6.4 Not exposed to end users

  • Model/temperature selection (server config).
  • Debug details (graph path, rewritten query, scores) stay in the dev-only API debug event; the Next.js UI doesn't render them.

7. Deployment

7.1 Local (Podman)

Prerequisites (macOS):

podman machine init --cpus 6 --memory 12288 --disk-size 60   # once
podman machine start
ollama serve                                                  # on the host (Metal GPU); not containerized
ollama pull qwen3.5:4b                                        # already installed

Run:

cp .env.example .env              # add ANTHROPIC_API_KEY, OPENAI_API_KEY
podman compose up -d              # core services
podman compose exec api rag admin create --email you@example.com --tenant demo

podman compose delegates to the installed compose provider (docker-compose or podman-compose); compose.yaml uses only the Compose spec, so it is portable.

ServiceImagePort(s)Notes
apiapp image8000uvicorn rag.api.main:app
workerapp image—python -m rag.worker; reranker model cached in a volume
webweb/Containerfile3000Next.js UI + BFF; only service the browser uses; reaches api over the private bff network (fixed IP 172.31.250.2, the only address the API trusts for X-Forwarded-For)
postgrespostgres:165432named volume, healthcheck
qdrantqdrant/qdrant6333, 6334named volume
objectstorerustfs/rustfs9000, 9001named volume, S3-compatible
mailpitaxllent/mailpit1025 (SMTP), 8025 (UI)dev only
prometheus, grafana, jaegerupstream images9090; 3001 (GRAFANA_HOST_PORT); 16686 (UI), 4318 (OTLP)--profile observability; config in ops/
Ollamahost process11434containers use OLLAMA_BASE_URL=http://host.containers.internal:11434

Podman notes:

  • Ollama on the host, not in a container: the Podman VM on macOS has no Metal GPU access, so containerized Ollama would run on CPU only. If containers can't reach it, start Ollama with OLLAMA_HOST=0.0.0.0.
  • Use named volumes (not bind mounts) for data services — avoids rootless UID/permission issues and slow file sharing on macOS. On Linux with SELinux, bind mounts need :Z.
  • depends_on: { condition: service_healthy } starts dependencies in order. The local Compose API runs alembic upgrade head on every start; production should use a coordinated release migration step.
  • Integration tests use running Compose services with a separate rag_test database, rag-test bucket and chunks_test alias/collections. They do not use Testcontainers. Mailpit recipients are unique per test tenant; existing captured email is preserved.

7.2 MVP production

Same OCI image (built with podman build) deployed to a PaaS, or a single VM running Podman Quadlet (systemd-managed containers). Managed Postgres; Qdrant Cloud or self-hosted with snapshots; S3; real SMTP provider instead of Mailpit (env change only). Ollama on a GPU host if orchestration stays local in prod — otherwise point the orchestration route at Claude Haiku via config. Migrations run as a release step.

The web image runs behind the HTTPS ingress; keep the API private (with separately authenticated service-account access if needed). Set ENV=prod, an HTTPS WEB_ORIGIN, and matching PUBLIC_UI_URL. Set TRUSTED_PROXY_HOPS to the number of trusted ingress hops that overwrite/append the actual peer IP, normally 1 for one ingress; web refuses to start with ENV=prod and TRUSTED_PROXY_HOPS=0, because every browser would then share the BFF's IP in the API's per-IP auth rate limits (one client could lock out all logins, signups and resets). Next.js preserves an existing X-Forwarded-For header and does not add a hop. Prevent direct public access to web when enabling forwarded identity. Set the API's FORWARDED_ALLOW_IPS to the web replicas' addresses.

7.3 Scale path (only when a metric demands it)

ComponentMVPEvolve whenTo
API1–2 replicasCPU/latencyMore replicas behind LB (stateless)
Worker1 replicaJob backlogMore replicas (SKIP LOCKED already safe)
QueuePostgres jobsSustained high job throughputRedis (arq) or SQS
Vector DBSingle Qdrant node> ~10M vectors or HA neededQdrant cluster (shards + replicas)
Local LLMOllama on one hostConcurrency (OLLAMA_NUM_PARALLEL saturated)Multiple Ollama hosts behind LB, or vLLM
RerankerLocal CPULatency under loadGPU instance or hosted rerank API
Authz3 fixed rolesCustom roles / per-doc ACLsDB-defined roles; doc ACL payload filter in Qdrant
AuthEmail + passwordEnterprise customersOIDC/SSO (keep RBAC unchanged)
Web BFF1 replicaConcurrency / HAMore replicas with sticky sessions (refresh single-flight is per process), or a shared refresh lock in Redis
AgentsCorrective RAG graphNew use casesAdditional LangGraph subgraphs (SQL, web, tools)

8. Implementation Plan (7 weeks)

Phase 0 — Foundation (Week 1, days 1–3)

  • Repo skeleton, pyproject, ruff/mypy/pytest, CI.
  • compose.yaml + Containerfile with Postgres, Qdrant, MinIO, Mailpit; podman compose up green.
  • Config via pydantic-settings; structured logging; OpenTelemetry wiring.
  • LLM adapters (Ollama, Anthropic, OpenAI) + ModelRouter with fallbacks; smoke test hits all three.
  • Draft golden eval set (≥ 50 Q&A with expected docs).
  • Exit: all services healthy under Podman; a structured-output call succeeds on each provider; fallback fires when Ollama is stopped.

Phase 1 — Identity, RBAC, Email (Weeks 1–2)

  • Tenants, users, memberships, tokens, API keys, audit log; Alembic migrations.
  • Login / refresh / logout; require(...) dependency; RBAC matrix test.
  • jobs table + worker loop; send_email handler; invite + password-reset flows via Mailpit.
  • Admin bootstrap CLI.
  • Exit: invite → email in Mailpit → accept → login works end to end in an integration test; every role/route combination covered by the matrix test.

Phase 2 — Ingestion (Weeks 2–3)

  • Document endpoints (permission-guarded); object storage.
  • ingest_document handler: parse, chunk, dense + sparse embed, Qdrant upsert with deterministic IDs; failure email.
  • Exit: 100 docs ingested; re-upload is a no-op; deletion removes vectors; recall@k measured on golden set.

Phase 3 — Retrieval + Baseline RAG (Weeks 3–4)

  • Hybrid search with RRF, reranker, tenant filter enforcement.
  • Single-pass generate (Claude) with citations; /v1/chat SSE; citation validation.
  • Exit: baseline eval numbers recorded (recall@8, faithfulness, p95 TTFT); Claude vs. OpenAI comparison run.

Phase 4 — Agentic Graph + Conversations (Weeks 4–5)

  • LangGraph nodes: analyze_query, respond_direct, grade, rewrite_query, insufficient_context on the local orchestration route.
  • Conversations/messages persistence (own-only access), history condensation. (A Postgres checkpointer was planned and dropped: see Revision 5, #21.)
  • Tune grading threshold; measure local-model analyze_query accuracy vs. Haiku.
  • Exit: eval ≥ Phase 3 baseline; follow-ups resolve correctly; ≤ 3 LLM calls/query; fallback rate < 5% in the eval run.

Phase 5 — UI (Week 6)

  • Streamlit pages: login/reset/accept-invite, chat, documents, members, settings.
  • Permission-aware rendering via /v1/me; debug toggle.
  • Exit: end-to-end demo with admin, editor, and viewer accounts.

Phase 5b — Chat history + Next.js UI (Revision 4)

  • Done in this step: migration 0004 (conversations.updated_at, indexes); history list ordering, keyset paging, title search, rename, delete, bounded message window with "load earlier"; tests for ordering, paging, search, rename/delete cascade, continuation context, and owner-only access.
  • Done in this step: web/ BFF (session cookie, refresh single-flight, allowlisted proxy, CSRF, CSP nonce, security headers); auth flows; chat with history sidebar and continuation; documents. CI job: lint, typecheck, format, unit tests, build, npm audit.
  • Implemented: Receipts, Members (invitations/roles/removal), Settings (API keys/audit history), permission-aware responsive navigation, and the corresponding BFF allowlist. The Streamlit UI has been removed; Next.js serves all workspace features.
  • Implemented (Revision 5): Playwright browser end-to-end tests in CI (web/e2e).
  • Done: legacy Streamlit UI removed; rolling conversation summary (§4.10).
  • Exit: all workspace pages have Next.js equivalents and browser CI coverage, and the legacy Streamlit code is removed (done).

Phase 6 — Hardening & Launch (Week 7)

  • Rate limits (chat, upload, login, reset); upload limits.
  • Eval in CI with regression gate; dashboards and alerts (incl. fallback rate).
  • Load test (e.g. Locust) against §1 targets, including Ollama concurrency.
  • Build images with Podman; deploy; backup/snapshot for Postgres and Qdrant verified by a restore test.
  • Exit: §1 targets met; runbooks for re-indexing, restore, and provider outage.
  • Status (20 October 2026): implemented — rate limits (chat, upload, login, signup, reset, refresh, invite); upload limits; eval CI gate (conditional on infrastructure secrets); metrics, alerts, dashboard and tracing (§5.3); Locust load test (loadtest/) with a TTFT p95 gate; backup/restore with an automated restore test (scripts/backup.py), verified locally including a full restore round trip on the test database; runbooks. Not yet met: the §1 targets have not been measured on production hardware. A local smoke run (Apple Silicon, local Ollama orchestration only, 2 users, workspace without documents) gave chat TTFT p95 ≈ 5.2 s, above the 3 s target; re-measure with cloud generation and a seeded corpus before launch.

9. Key Decisions (ADR summary)

DecisionChosenRejectedReason
OrchestrationLangGraph state machineAutoGen; free-form multi-agent chatDeterministic, testable control flow; checkpointing; extend with subgraphs
Agent count2–3 LLM nodes5 LLM agentsLatency, cost, debuggability; quality comes from retrieval
LLM accessOwn thin adapters + ModelRouter (3 small files)LiteLLM; LangChain chat modelsOnly 3 providers; need provider-specific structured output, prompt caching, and explicit fallback semantics. Revisit LiteLLM if providers grow beyond ~4.
Orchestration modelLocal Ollama qwen3.5:4b, cloud fallbackCloud-onlyFree, low-latency on Apple Silicon, keeps questions local; fallback covers outages and quality misses
Generation modelClaude Sonnet 5.5, OpenAI fallbackSingle providerProvider outage doesn't take the product down; choice re-validated by eval
RetrievalHybrid + rerankDense onlyExact terms (IDs, error codes, names) fail with dense-only
Vector DBQdrantChroma, pgvectorHybrid + multi-tenancy + clustering built in (pgvector viable if corpus stays small)
Source of truthPostgresVector DB payloadsRebuildable index, re-embedding, deletes, audit
Authorization3 fixed roles, permissions in code, role read from DB per requestDB-defined roles; Casbin/OPAEnough for MVP, trivially testable; immediate revocation
AuthEmail/password + JWT + rotating refreshOIDC nowNo external IdP needed for MVP; OIDC slots in later without touching RBAC
EmailSMTP via jobs-table outbox; Mailpit locallySending inline; vendor SDKNo lost/phantom emails; provider swap is env-only
QueuePostgres SKIP LOCKEDCelery/RedisNo extra infra at MVP volume
ContainersPodman (rootless) + Compose specDocker DesktopDaemonless/rootless, no licensing concerns; Compose spec keeps it portable; Quadlet for single-VM prod
StreamingSSEWebSocketsOne-way stream is enough; simpler infra
UINext.js App Router as a BFFStreamlit; SPA holding tokens in localStorageProduct-grade UI; tokens never reachable from JS; one same-origin surface for CSP/CSRF. Streamlit got the MVP out; the API-first design made the swap a UI-only change
Chat history pagingKeyset (updated_at, id)OFFSETStable while chats move and are deleted; constant cost per page
Long-chat contextFixed window of last N turns + query condensationWhole transcript; rolling summary nowBounded cost/latency; add a rolling summary only if evals show degradation
Graph stateStateless graph per turn, history from messagesLangGraph Postgres checkpointerNothing to resume between turns; avoids a write per node. Add a checkpointer with human-in-the-loop interrupts
TracingOpenTelemetry over OTLP (Jaeger locally, Langfuse optional)Langfuse SDK directlyVendor-neutral; the same spans go to any OTLP backend, and Langfuse ingests OTLP
MetricsPrometheus pull, DB-derived gauges at scrapePush gateway; per-process job countersJob and eval health come from Postgres, so they are correct with any number of API/worker replicas
Online qualitySampled async judge (judge_answer job)Inline groundedness checkOff the hot path; zero latency cost; sample rate is config