1. Context and scale
A production RAG system is a search product with an LLM attached—not a prompt template with a vector database behind it. Its quality depends on whether the right source material is ingested, represented, retrieved, ranked, authorized, and kept current. The model cannot compensate for a chunking strategy that loses context or a retrieval path that misses the evidence.
This blueprint serves an authenticated knowledge assistant over internal policies, product documentation, and support material. It returns answers with source citations and declines when retrieved evidence is insufficient. AWS is the chosen cloud; the application is a Python modular monolith deployed on ECS Fargate, with a separate worker process from the same codebase.
The scale is intentionally medium. A small team can operate the design without adopting a fleet of independently deployed services, while the queue, managed search, and multi-AZ database provide clear seams for later growth.
| Assumption | Planning value | Why it matters |
|---|---|---|
| Daily active users | 35,000 | Within the requested 10k–100k scale |
| Query volume | 140,000/day; 25 requests/s peak | Assumes four queries per active user and uneven traffic |
| Corpus | 500 GB source files; about 2 million chunks | Drives index size, ingestion duration, and refresh policy |
| Change rate | Up to 20,000 changed documents/day | Requires incremental ingestion and deletion handling |
| Team size | 5 engineers across 1–2 teams | Supports a modular monolith plus shared platform ownership |
| Monthly cloud budget | $8,000–$15,000 planning envelope | Includes model usage; actual cost depends on token and traffic mix |
These are capacity-planning assumptions, not measured results or vendor quotes. The first production release should measure document-size distribution, query concurrency, token consumption, and index growth, then revise the assumptions.
The request path is latency-sensitive, but ingestion is not. We therefore separate them with a queue: ingestion can lag temporarily without making every user request wait for parsing and indexing. The query path has a strict tenant and document-access boundary; retrieval must apply that boundary before any candidate can reach a reranker or model.
2. Requirements
Functional requirements
| Requirement | Acceptance condition |
|---|---|
| Ingest and update sources | Accept supported files from S3; detect updates and deletions; preserve source version and ingestion status |
| Create searchable chunks | Store chunk text, stable identifiers, source metadata, access attributes, and embeddings |
| Retrieve evidence | Combine lexical and vector candidate lists, apply authorization filters, and rerank a bounded candidate set |
| Generate cited answers | Return answer text, cited source references, and an explicit “insufficient evidence” outcome |
| Support evaluation | Run a fixed, versioned query set against candidate index, chunking, and prompt changes |
| Operate safely | Retry transient work, quarantine poison documents, and provide an operator-visible replay path |
Non-functional requirements
| Concern | Initial target |
|---|---|
| Query latency | p95 under 4 seconds for the complete answer path, measured separately from streaming completion |
| Availability | 99.9% monthly availability for the query API, excluding planned maintenance |
| Retrieval quality | Release gate based on judged retrieval and citation metrics; no universal threshold assumed |
| Freshness | 95% of accepted source changes searchable within 15 minutes |
| Isolation | Every retrieval is scoped to the authenticated tenant and current authorization policy |
| Resilience | Multi-AZ application and database deployment; durable queue with dead-letter handling |
| Data handling | Encrypt data in transit and at rest; apply retention and deletion policy to source, metadata, and index |
| Operability | Trace a request across retrieval, reranking, and model calls without logging source text or prompts by default |
The p95 target is an end-to-end objective, not a promise that each component has a four-second budget. Establish per-stage latency budgets from production traces. In particular, model generation can dominate the request, while a large candidate set can make reranking the avoidable tail-latency contributor.
3. High-level design
The API and ingestion worker share domain code but run as separate ECS tasks. This keeps deployment and scaling straightforward for a small team while isolating ingestion CPU and memory from interactive traffic. PostgreSQL is the source of truth for document and job state; OpenSearch holds the derived retrieval representation.
flowchart LR
User["Authenticated user"] --> Api["ECS Fargate query API"]
Api --> Auth["Amazon Cognito or enterprise OIDC"]
Api --> Cache["ElastiCache Redis"]
Api --> Search["Amazon OpenSearch Service"]
Api --> Rank["ECS Fargate reranker"]
Api --> Bedrock["Amazon Bedrock"]
Api --> Db["Amazon RDS PostgreSQL"]
Source["S3 source bucket"] --> Queue["Amazon SQS"]
Queue --> Worker["ECS Fargate ingestion worker"]
Worker --> Parse["Document parsing and chunking"]
Parse --> Bedrock
Parse --> Search
Worker --> Db
Search --> Api
Db --> Api
Worker --> Dlq["SQS dead-letter queue"]Caption: Query and ingestion workloads share application code but use separate ECS task services and scaling policies.
The query API authenticates the caller, derives tenant and access claims from trusted identity context, and computes a cache key that includes those claims. It embeds the query, retrieves lexical and vector candidates, fuses their ranks, and reranks the bounded result set. Only authorized evidence is passed to the generation step. Responses include references that the UI can resolve to a permitted source location.
The ingestion worker consumes S3 object events through SQS. It parses and normalizes content, applies versioned chunking rules, calls the embedding model, upserts chunks into OpenSearch, and updates relational job state. A source update is not complete merely because embedding succeeded: the worker records the index generation and reports completion only after the searchable representation is committed. Deletions are explicit tombstones that remove old chunks.
Redis is an optimization, not a system of record. Cache only results whose authorization context, index generation, and prompt version are part of the cache key. Do not cache unfiltered search results and rely on a later layer to remove unauthorized passages.
4. Cloud service mapping
AWS is selected for this blueprint. The equivalents below are architectural counterparts, not feature-for-feature substitutes; validate networking, ranking, identity, and model behavior before a cross-cloud migration.
| Component | AWS choice | Azure counterpart | GCP counterpart |
|---|---|---|---|
| Containers and API | Amazon ECS on AWS Fargate | Azure Container Apps | Cloud Run |
| Object storage | Amazon S3 | Azure Blob Storage | Cloud Storage |
| Work queue | Amazon SQS | Azure Service Bus queues | Cloud Tasks or Pub/Sub |
| Relational metadata | Amazon RDS for PostgreSQL, Multi-AZ | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Hybrid retrieval index | Amazon OpenSearch Service | Azure AI Search | Vertex AI Search, or a managed vector/search design |
| Embedding and generation | Amazon Bedrock | Azure AI Foundry | Vertex AI |
| Reranking | ECS service running an approved cross-encoder | Container Apps or Azure ML endpoint | Cloud Run or Vertex AI endpoint |
| Cache | Amazon ElastiCache for Redis | Azure Managed Redis | Memorystore |
| Secrets and keys | AWS Secrets Manager and AWS KMS | Key Vault | Secret Manager and Cloud KMS |
| Metrics and logs | Amazon CloudWatch | Azure Monitor | Cloud Monitoring and Cloud Logging |
| Tracing | OpenTelemetry collector/export to an approved backend | Azure Monitor or another OTLP backend | Cloud Trace or another OTLP backend |
A managed search product can reduce operational work, but it does not remove the need to inspect ranking behavior, filters, refresh semantics, and relevance evaluation. Keep the application-level fusion and evaluation contracts stable enough that changing search backends does not require rewriting the product’s quality gates.
5. Data design
Use PostgreSQL for source identity, access metadata, ingestion state, and version history. Store original files in versioned S3 buckets. OpenSearch is a derived index: each indexed chunk carries a stable chunk ID, tenant ID, document ID, source version, ACL attributes, text, and embedding. Rebuild it from S3 and PostgreSQL rather than treating it as the only copy.
Chunk IDs should be deterministic for a given document version and chunking configuration. That makes retries idempotent and prevents a failed re-ingestion from producing duplicate passages. Keep the chunker version and embedding model version in metadata; otherwise, teams cannot explain why two documents were indexed differently or safely roll out a new strategy.
erDiagram
TENANT ||--o{ DOCUMENT : owns
DOCUMENT ||--o{ DOCUMENT_VERSION : has
DOCUMENT_VERSION ||--o{ INGESTION_JOB : processed_by
DOCUMENT_VERSION ||--o{ CHUNK_REFERENCE : produces
TENANT ||--o{ EVALUATION_RUN : scopes
EVALUATION_RUN ||--o{ EVALUATION_RESULT : containsCaption: PostgreSQL records lifecycle and provenance; the chunk reference identifies derived records stored in OpenSearch.
CREATE TABLE tenant (
tenant_id uuid PRIMARY KEY,
name text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE document (
document_id uuid PRIMARY KEY,
tenant_id uuid NOT NULL REFERENCES tenant(tenant_id),
source_key text NOT NULL,
title text NOT NULL,
access_labels jsonb NOT NULL DEFAULT '[]'::jsonb,
active_version text,
deleted_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, source_key)
);
CREATE TABLE document_version (
tenant_id uuid NOT NULL,
document_id uuid NOT NULL REFERENCES document(document_id),
source_version text NOT NULL,
content_sha256 char(64) NOT NULL,
chunker_version text NOT NULL,
embedding_model text NOT NULL,
status text NOT NULL CHECK
(status IN ('queued','processing','indexed','failed','deleted')),
chunk_count integer NOT NULL DEFAULT 0 CHECK (chunk_count >= 0),
indexed_at timestamptz,
PRIMARY KEY (document_id, source_version)
);
CREATE TABLE ingestion_job (
job_id uuid PRIMARY KEY,
tenant_id uuid NOT NULL,
document_id uuid NOT NULL REFERENCES document(document_id),
source_version text NOT NULL,
attempt integer NOT NULL DEFAULT 0,
status text NOT NULL CHECK
(status IN ('queued','running','succeeded','failed','dead_letter')),
error_code text,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (document_id, source_version)
);
CREATE INDEX ingestion_job_status_created_idx
ON ingestion_job (status, created_at);
CREATE TABLE chunk_reference (
chunk_id text PRIMARY KEY,
tenant_id uuid NOT NULL REFERENCES tenant(tenant_id),
document_id uuid NOT NULL REFERENCES document(document_id),
source_version text NOT NULL,
ordinal integer NOT NULL,
token_count integer NOT NULL,
index_generation text NOT NULL,
UNIQUE (document_id, source_version, ordinal)
);
CREATE TABLE evaluation_run (
run_id uuid PRIMARY KEY,
dataset_version text NOT NULL,
pipeline_version text NOT NULL,
started_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz,
status text NOT NULL
);
CREATE TABLE evaluation_result (
run_id uuid NOT NULL REFERENCES evaluation_run(run_id),
case_id text NOT NULL,
metric_name text NOT NULL,
metric_value double precision NOT NULL,
details jsonb NOT NULL DEFAULT '{}'::jsonb,
PRIMARY KEY (run_id, case_id, metric_name)
);
The index should use separate fields for searchable text, exact filters, and vector values. Map IDs and access attributes as exact-match fields; do not make tenant or ACL fields analyzed text. Keep source text out of operational logs. For deletion requests, remove index records and cached answers, then retain only the minimum audit record allowed by policy.
6. Low-level design
Keep the application organized around a small number of modules: ingestion, retrieval, generation, identity, and evaluation. The interfaces matter more than the folder layout. In particular, keep chunking configuration, retrieval candidate limits, fusion weights, and prompt version explicit so that evaluation can attribute a quality change to a specific pipeline revision.
classDiagram
class IngestionWorker {
+handle(message)
+process_document(source)
}
class Chunker {
+chunk(text, config)
}
class Embedder {
+embed(texts)
}
class SearchRepository {
+index(chunks)
+hybrid_search(query, filters)
}
class Ranker {
+rerank(query, candidates)
}
class AnswerService {
+answer(query, evidence)
}
IngestionWorker --> Chunker
IngestionWorker --> Embedder
IngestionWorker --> SearchRepository
SearchRepository --> Ranker
Ranker --> AnswerServiceCaption: Domain interfaces keep pipeline behavior testable without binding evaluation logic to one provider SDK.
Minimal runnable example: chunk boundaries
This example uses character windows with overlap. It is deliberately small enough to run directly, but production chunking should use tokenizer-aware limits and preserve structural boundaries such as headings, paragraphs, and table rows.
def chunk_text(text: str, size: int = 500, overlap: int = 80) -> list[str]:
if size <= 0 or overlap < 0 or overlap >= size:
raise ValueError("Require size > overlap >= 0")
text = " ".join(text.split())
chunks = []
start = 0
while start < len(text):
end = min(start + size, len(text))
chunks.append(text[start:end])
if end == len(text):
break
start = end - overlap
return chunks
if __name__ == "__main__":
for i, chunk in enumerate(chunk_text("A short example document. " * 40)):
print(i, len(chunk), chunk[:48])
What to notice: overlap preserves local continuity, but character limits are not model-token limits and can split a sentence or table.
Realistic runnable example: hybrid retrieval and reranking
The following script queries an OpenSearch domain twice—once with BM25 and once with k-NN—then applies reciprocal-rank fusion and a cross-encoder reranker. It assumes the index has text, embedding, tenant_id, and document_id fields, and that the caller has already validated the tenant identity. Install opensearch-py, boto3, sentence-transformers, and requests-aws4auth; set the environment variables and AWS credentials before running. The selected cross-encoder must be approved and available to the service.
import json
import os
import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
from sentence_transformers import CrossEncoder
region = os.environ["AWS_REGION"]
host = os.environ["OPENSEARCH_HOST"]
index = os.getenv("OPENSEARCH_INDEX", "rag-chunks")
model_id = os.getenv("EMBED_MODEL", "amazon.titan-embed-text-v2:0")
credentials = boto3.Session().get_credentials().get_frozen_credentials()
auth = AWS4Auth(
credentials.access_key, credentials.secret_key, region, "es",
session_token=credentials.token,
)
search = OpenSearch(
hosts=[{"host": host, "port": 443}],
http_auth=auth, use_ssl=True, verify_certs=True,
connection_class=RequestsHttpConnection, timeout=3,
)
bedrock = boto3.client("bedrock-runtime", region_name=region)
reranker = CrossEncoder(os.getenv(
"RERANK_MODEL", "cross-encoder/ms-marco-MiniLM-L-6-v2"
))
def embed(text: str) -> list[float]:
response = bedrock.invoke_model(
modelId=model_id,
body=json.dumps({"inputText": text, "dimensions": 1024, "normalize": True}),
)
return json.loads(response["body"].read())["embedding"]
def retrieve(query: str, tenant_id: str, limit: int = 5) -> list[dict]:
vector = embed(query)
lexical = search.search(index=index, body={
"size": 40,
"query": {"bool": {
"must": [{"match": {"text": query}}],
"filter": [{"term": {"tenant_id": tenant_id}}],
}},
"_source": ["text", "document_id"],
})["hits"]["hits"]
dense = search.search(index=index, body={
"size": 40,
"query": {"knn": {"embedding": {
"vector": vector, "k": 40,
"filter": {"term": {"tenant_id": tenant_id}},
}}},
"_source": ["text", "document_id"],
})["hits"]["hits"]
fused = {}
for ranked_list in (lexical, dense):
for rank, hit in enumerate(ranked_list, start=1):
item = fused.setdefault(hit["_id"], {
"id": hit["_id"], **hit["_source"], "rrf": 0.0
})
item["rrf"] += 1.0 / (60 + rank)
candidates = sorted(fused.values(), key=lambda x: x["rrf"], reverse=True)[:40]
scores = reranker.predict([(query, item["text"]) for item in candidates])
for item, score in zip(candidates, scores):
item["rerank_score"] = float(score)
return sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)[:limit]
if __name__ == "__main__":
results = retrieve(os.environ["QUERY"], os.environ["TENANT_ID"])
for result in results:
print(result["document_id"], round(result["rerank_score"], 4), result["text"][:160])
What to notice: authorization filtering is applied in both retrieval requests, fusion uses rank rather than incomparable raw scores, and reranking happens only after candidate generation.
In production, validate index mappings and model dimensions at startup, reuse clients and the loaded reranker across requests, and bound all candidate counts and timeouts. If the reranker cannot respond within its budget, return the fused ranking with a degraded-response signal rather than blocking indefinitely.
Chunking is a quality decision, not just a storage decision. Start with structure-aware chunks and a modest overlap; evaluate alternatives against the same corpus and query set. Larger chunks may preserve context but dilute relevance and increase prompt cost. Smaller chunks may improve precision but separate a fact from its qualification. Track chunker version, token count, section path, and source offsets so answers can cite useful locations.
Hybrid retrieval handles complementary failure modes. Lexical search is strong for identifiers, error codes, names, and exact phrases. Dense retrieval can find semantically related wording. Do not add their raw scores unless calibration has been demonstrated; rank fusion is a safer first baseline. Reranking then spends more compute on a much smaller candidate set to improve ordering. It cannot recover a relevant passage that neither retriever returned.
7. Key flows
The ingestion flow is asynchronous and idempotent. S3 events are notifications, not a complete source-of-truth log; the worker verifies the object version and content hash before doing expensive work. A retry may repeat parsing or embedding, so writes use deterministic chunk IDs and upserts.
sequenceDiagram
participant S3 as S3
participant Q as SQS
participant W as Ingestion worker
participant B as Bedrock embeddings
participant O as OpenSearch
participant P as PostgreSQL
S3->>Q: Object event
Q->>W: Deliver message
W->>S3: Read object version
W->>P: Claim idempotent job
W->>W: Parse and chunk
W->>B: Embed batches
B-->>W: Vectors
W->>O: Upsert versioned chunks
W->>P: Mark indexed
W-->>Q: AcknowledgeCaption: A job is acknowledged only after indexing and state updates succeed; failures remain retryable or move to the dead-letter queue.
The query flow treats retrieval as a sequence of bounded stages. Every stage records latency and outcome, but sensitive content is not copied into telemetry. The generator receives only the evidence necessary to answer the question, not the full source corpus.
sequenceDiagram
participant U as User
participant A as Query API
participant I as Identity
participant O as OpenSearch
participant R as Reranker
participant B as Bedrock generation
U->>A: Ask question
A->>I: Validate identity and claims
I-->>A: Tenant and access scope
A->>A: Embed query
A->>O: Lexical and vector search
O-->>A: Candidate passages
A->>R: Rank bounded candidates
R-->>A: Ordered evidence
A->>B: Question and authorized evidence
B-->>A: Answer with citations
A-->>U: Answer and source referencesCaption: The API establishes access scope before search and passes only authorized, reranked evidence to generation.
8. Build and release
Infrastructure changes should be reviewed with application changes. Keep environment-specific values out of Terraform code, use remote state with locking and encryption, and create separate staging and production accounts or equivalent isolated environments. The snippet below establishes an encrypted SQS queue and dead-letter queue; production infrastructure also needs least-privilege policies, private networking, encryption keys, and alarms.
resource "aws_sqs_queue" "ingestion_dlq" {
name = "rag-ingestion-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sqs_queue" "ingestion" {
name = "rag-ingestion"
visibility_timeout_seconds = 300
message_retention_seconds = 345600
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.ingestion_dlq.arn
maxReceiveCount = 5
})
}
This GitHub Actions example runs quality checks, builds an image, and deploys to staging after tests pass. Configure AWS federation through GitHub OIDC rather than long-lived access keys; production promotion should be a separate protected environment with approval and a rollback path.
name: rag-ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
id-token: write
jobs:
test-and-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: pytest -q
- run: ruff check .
- run: docker build -t rag-api:${{ github.sha }} .
- if: github.ref == 'refs/heads/main'
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.STAGING_DEPLOY_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- if: github.ref == 'refs/heads/main'
run: |
./scripts/push-image.sh rag-api:${{ github.sha }}
./scripts/deploy-staging.sh ${{ github.sha }}
Test strategy:
- Unit: chunk boundaries, stable IDs, ACL filter construction, fusion ordering, citation formatting, and retry classification.
- Integration: parsing fixtures, PostgreSQL state transitions, OpenSearch mappings and filters, and model-client timeout behavior.
- Contract: validate source event schemas, index document shape, model request/response parsing, and API response compatibility.
- Load: replay representative query and ingestion mixes against staging; measure p95 by stage, index pressure, queue age, connection pools, and model throttling. Use realistic document lengths and query distributions, not only synthetic short prompts.
- Evaluation: run a fixed dataset before and after changes to chunker, embedding model, index settings, reranker, prompt, or generation model. Block releases on agreed regressions, not on a single aggregate score that can hide a harmed user segment.
Promote the same immutable image through staging and production. Index and prompt changes should be independently versioned so a rollback does not require rebuilding all source documents. Run a small canary query set against a new index generation before switching production traffic.
9. Observability
Instrument the request as a trace with spans for authentication, embedding, lexical retrieval, vector retrieval, fusion, reranking, generation, and citation validation. Record model and pipeline versions as attributes, but do not attach raw prompts, answers, or retrieved text by default. For debugging sensitive content, use an explicit, audited, time-limited diagnostic mechanism with redaction and access controls.
A structured log event should support correlation without turning logs into a secondary content store:
{
"timestamp": "2026-11-10T12:00:00Z",
"level": "INFO",
"service": "rag-api",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"request_id": "req_8d2c",
"tenant_hash": "sha256:9a7f...",
"pipeline_version": "retrieval-17",
"index_generation": "chunks-2026-11-10-a",
"candidate_counts": {"lexical": 40, "dense": 40, "fused": 40, "reranked": 5},
"stage_ms": {"embed": 84, "search": 112, "rerank": 96, "generation": 1290},
"outcome": "answered"
}
For RED, monitor request rate, error rate, and duration for the API and each model/search dependency. For USE, monitor utilization, saturation, and errors for ECS CPU/memory, OpenSearch nodes and thread pools, RDS connections/storage, Redis memory/evictions, and SQS age and in-flight messages. Separate user-visible errors from expected “no evidence” outcomes.
OpenTelemetry instrumentation should propagate context across HTTP and provider calls. This Python example creates a span around retrieval; initialize the provider and exporter at process startup and use an OTLP-compatible backend.
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer("rag.retrieval")
def traced_retrieve(query: str, tenant_id: str):
with tracer.start_as_current_span("rag.retrieve") as span:
span.set_attribute("rag.pipeline.version", "retrieval-17")
span.set_attribute("rag.tenant.hash", hash_tenant(tenant_id))
try:
result = retrieve(query, tenant_id)
span.set_attribute("rag.result.count", len(result))
return result
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR))
raise
| SLI | Initial SLO | Measurement |
|---|---|---|
| Query API availability | 99.9% monthly | Successful non-5xx requests divided by eligible requests |
| Query latency | 95% under 4 seconds | End-to-end request duration; track streaming separately |
| Searchable freshness | 95% within 15 minutes | Source accepted-to-indexed duration |
| Ingestion completion | 99% of valid jobs within 30 minutes | Successful jobs, excluding documented source-side delays |
| Citation integrity | 99% of returned citations resolve to an authorized source | Sampled automated check plus evaluation set |
Dashboard panels should include API RED, dependency latency/error breakdown, cache hit ratio, candidate counts, retrieval-empty rate, generation token volume, SQS oldest-message age, dead-letter count, and index freshness. Add a quality dashboard for evaluation trends by query category and tenant segment. Aggregate quality scores must not conceal a regression on exact-match or access-control cases.
Page on sustained API SLO burn, a high-severity authorization or data-isolation failure, search or model dependency failure that breaks the user path, growing queue age beyond the freshness objective, and exhausted database or search capacity. Alert on trends and actionable conditions; route evaluation-score changes to release review rather than paging an on-call engineer at night.
10. Security and compliance
Identity and authorization are part of retrieval correctness. Validate issuer, audience, signature, expiry, and required claims at the API boundary. Derive tenant and ACL filters from trusted identity and server-side policy—not from user-provided request fields. Apply the filter to both lexical and vector search, then validate candidate access again before generation. Treat cache entries as access-controlled data.
Use private subnets for ECS tasks, RDS, and OpenSearch where supported by the chosen deployment. Restrict egress to required AWS endpoints and approved model services. Use task roles scoped to specific buckets, queues, indexes, and secrets; avoid credentials in images or environment files. Encrypt S3, database, search, queue, and backups with managed keys according to the organization’s key policy.
Additional controls:
- Define allowed file types and enforce size, decompression, and parser time limits; treat uploaded documents as untrusted input.
- Protect against prompt injection in retrieved content. Retrieved passages are evidence, not instructions; delimit them and instruct the model not to follow embedded commands.
- Apply malware scanning and content classification if required by the data policy.
- Establish retention and deletion behavior for source versions, index records, caches, logs, backups, and evaluation fixtures.
- Record model/provider, prompt version, source references, and policy version for audit without logging the full private content.
- Review provider data handling, regional processing, model access, and contractual terms before sending regulated or sensitive content.
- Keep evaluation datasets permissioned and scrubbed; test that unauthorized documents never appear in candidate results or citations.
Compliance is not a property that can be inferred from using a managed service. Map the actual data classes and processing locations to the organization’s controls, then verify the configuration and operational evidence.
11. Cost
The following is an assumption-based monthly planning estimate, not a current AWS quote. It assumes one production region, multi-AZ database and application deployment, a modest OpenSearch cluster sized for the stated corpus, 140,000 daily queries, and regular document updates. Model spend is especially sensitive to prompt length, output length, caching, and model choice.
| Cost area | Monthly planning assumption |
|---|---|
| ECS Fargate API and ingestion tasks | $1,100 |
| Amazon OpenSearch Service | $2,200 |
| RDS PostgreSQL Multi-AZ | $800 |
| Bedrock embeddings and generation | $3,200 |
| S3 storage and requests | $150 |
| ElastiCache Redis | $350 |
| SQS, CloudWatch, and telemetry | $500 |
| Data transfer and miscellaneous networking | $400 |
| Planning total | $8,700 |
Validate line items with the AWS Pricing Calculator for the chosen region and actual instance types. Before launch, measure tokens per request and cache hit rate; generation and embedding costs can change materially with model choice and usage. Keep an explicit per-request token budget and report spend by environment, model, and workload. If the budget is exceeded, first reduce unnecessary context and repeated embedding—not silently lower answer quality or remove authorization checks.
12. Failure modes, resilience and disaster recovery
| Failure mode | User or system impact | Response |
|---|---|---|
| SQS delivery repeats | Duplicate ingestion work | Idempotent job key and deterministic chunk IDs; safe upsert |
| Parser fails on a document | That source is stale or unavailable | Retry transient failures; quarantine invalid inputs; expose job status |
| Bedrock throttles or times out | Slow ingestion or incomplete answers | Bounded retries with jitter; concurrency limits; queue backpressure |
| OpenSearch is degraded | Retrieval errors or latency | Fail fast within a budget; return a clear unavailable response rather than unsupported answers |
| Reranker is unavailable | Lower ordering quality | Use fused ranking as a documented degraded mode |
| Stale index after an update | Old content may be retrieved | Track source version; suppress superseded versions; alert on freshness lag |
| Redis contains outdated answer | Stale response | Version cache keys by index and prompt; bounded TTL and targeted invalidation |
| Tenant filter regression | Potential data exposure | Deny by default; integration tests; policy-level alarm and incident procedure |
Use exponential backoff with jitter for transient dependency failures, but cap retries so a single request cannot occupy resources indefinitely. Set queue visibility timeout above normal job duration and extend it for long jobs where appropriate. Use a dead-letter queue for repeated failures, with an operator workflow that can inspect a redacted error, correct the cause, and replay safely.
Deploy the API and workers across Availability Zones. PostgreSQL backups and point-in-time recovery should be configured and restore-tested. Treat OpenSearch as rebuildable derived state, but recognize that a full rebuild can take long enough to affect freshness objectives; retain the source and indexing metadata needed to rebuild it. Define recovery time and recovery point objectives with product owners. A second region is not justified by this medium-scale baseline unless the business recovery requirement demands it.
During a regional or provider outage, fail closed for authentication and authorization. Do not fall back to an unfiltered cache or a stale index whose access policy cannot be trusted. If generation fails after retrieval, an optional degraded response can return authorized citations or a clear retry message, but it must not present unsupported prose as an answer.
13. Evolving to the next scale
Do not split the modular monolith just because it contains several modules. Split only when operational or ownership boundaries justify the cost of another deployable service.
| Signal observed in production | Likely next change |
|---|---|
| API and ingestion contend for CPU or memory | Keep separate ECS services and tune task scaling independently; move heavy parsing to a dedicated worker image |
| Queue freshness SLO misses during ordinary change volume | Increase worker concurrency with provider-aware throttles; partition workloads by source or tenant |
| Search latency or capacity is the dominant bottleneck | Tune mappings, shard strategy, filters, and candidate limits; load-test a larger or separated search deployment |
| Reranking dominates tail latency or cost | Reduce candidate count using evaluation evidence, batch inference, or operate a separately scaled inference service |
| One index cannot meet isolation or lifecycle needs | Partition indexes by access domain or retention class, with explicit routing and migration tooling |
| Multiple teams need independent release ownership | Extract ingestion or evaluation only after contracts, on-call ownership, and deployment boundaries are clear |
| Recovery requirements demand regional continuity | Build and test a second-region restore or replication plan with defined consistency and failover behavior |
A change in architecture should follow measured pressure. For example, a larger candidate set may improve recall but increase reranker cost; a more specialized index may improve filtering while adding operational work. Every such change should enter the evaluation harness first, then a staging load test, then a controlled production rollout.
14. Phased implementation plan
- Establish the quality baseline. Gather representative questions, expected evidence, authorization cases, and “should abstain” examples. Version the dataset and agree on retrieval, citation, and latency measures before selecting a final chunking strategy.
- Build ingestion and provenance. Implement S3-to-SQS processing, parsing, deterministic chunk IDs, PostgreSQL job state, retries, deletion handling, and a small searchable corpus. Confirm replays and partial failures are safe.
- Add hybrid retrieval. Implement lexical and vector candidate retrieval with tenant filtering in both paths. Compare each retriever independently, then test rank fusion. Record retrieval results for evaluation without retaining unnecessary private content.
- Add reranking and grounded generation. Bound the candidate set, introduce a cross-encoder, and pass only authorized evidence to the model. Require citations and an abstention path. Test prompt injection and citation resolution.
- Harden the operational path. Add timeouts, rate limits, queue alarms, tracing, dashboards, cost attribution, secrets management, and restore procedures. Run integration and load tests in staging using representative traffic shapes.
- Release behind controlled exposure. Promote an immutable build, run a canary query set, compare quality and latency to the baseline, then gradually expand traffic. Keep the previous index generation and application version available for rollback.
- Review after real usage. Inspect failure classes, freshness, retrieval misses, cache behavior, latency, and cost by workload. Change one major quality variable at a time—chunking, embedding, fusion, reranker, or prompt—and require evaluation evidence before rollout.
The production bar is not “the model can answer a demo question.” It is that the system can find authorized evidence, explain where that evidence came from, decline when it has none, and remain diagnosable when any stage fails. Chunking, hybrid retrieval, reranking, and evaluation are one pipeline; operate and release them as such.
This article was generated with AI. #AIGenerated