Skip to content

Production RAG Pipeline: From Chunk Boundaries to Release Gates

A medium-scale AWS blueprint for building a production RAG pipeline, from document ingestion and hybrid retrieval to reranking, evaluation, and operational readiness.

Ekanatha Reddy Urivakili19 min3,793 words

1. Context and scale

A production RAG system is a search product with an LLM attached—not a prompt template with a vector database behind it. Its quality depends on whether the right source material is ingested, represented, retrieved, ranked, authorized, and kept current. The model cannot compensate for a chunking strategy that loses context or a retrieval path that misses the evidence.

This blueprint serves an authenticated knowledge assistant over internal policies, product documentation, and support material. It returns answers with source citations and declines when retrieved evidence is insufficient. AWS is the chosen cloud; the application is a Python modular monolith deployed on ECS Fargate, with a separate worker process from the same codebase.

The scale is intentionally medium. A small team can operate the design without adopting a fleet of independently deployed services, while the queue, managed search, and multi-AZ database provide clear seams for later growth.

AssumptionPlanning valueWhy it matters
Daily active users35,000Within the requested 10k–100k scale
Query volume140,000/day; 25 requests/s peakAssumes four queries per active user and uneven traffic
Corpus500 GB source files; about 2 million chunksDrives index size, ingestion duration, and refresh policy
Change rateUp to 20,000 changed documents/dayRequires incremental ingestion and deletion handling
Team size5 engineers across 1–2 teamsSupports a modular monolith plus shared platform ownership
Monthly cloud budget$8,000–$15,000 planning envelopeIncludes model usage; actual cost depends on token and traffic mix

These are capacity-planning assumptions, not measured results or vendor quotes. The first production release should measure document-size distribution, query concurrency, token consumption, and index growth, then revise the assumptions.

The request path is latency-sensitive, but ingestion is not. We therefore separate them with a queue: ingestion can lag temporarily without making every user request wait for parsing and indexing. The query path has a strict tenant and document-access boundary; retrieval must apply that boundary before any candidate can reach a reranker or model.

2. Requirements

Functional requirements

RequirementAcceptance condition
Ingest and update sourcesAccept supported files from S3; detect updates and deletions; preserve source version and ingestion status
Create searchable chunksStore chunk text, stable identifiers, source metadata, access attributes, and embeddings
Retrieve evidenceCombine lexical and vector candidate lists, apply authorization filters, and rerank a bounded candidate set
Generate cited answersReturn answer text, cited source references, and an explicit “insufficient evidence” outcome
Support evaluationRun a fixed, versioned query set against candidate index, chunking, and prompt changes
Operate safelyRetry transient work, quarantine poison documents, and provide an operator-visible replay path

Non-functional requirements

ConcernInitial target
Query latencyp95 under 4 seconds for the complete answer path, measured separately from streaming completion
Availability99.9% monthly availability for the query API, excluding planned maintenance
Retrieval qualityRelease gate based on judged retrieval and citation metrics; no universal threshold assumed
Freshness95% of accepted source changes searchable within 15 minutes
IsolationEvery retrieval is scoped to the authenticated tenant and current authorization policy
ResilienceMulti-AZ application and database deployment; durable queue with dead-letter handling
Data handlingEncrypt data in transit and at rest; apply retention and deletion policy to source, metadata, and index
OperabilityTrace a request across retrieval, reranking, and model calls without logging source text or prompts by default

The p95 target is an end-to-end objective, not a promise that each component has a four-second budget. Establish per-stage latency budgets from production traces. In particular, model generation can dominate the request, while a large candidate set can make reranking the avoidable tail-latency contributor.

3. High-level design

The API and ingestion worker share domain code but run as separate ECS tasks. This keeps deployment and scaling straightforward for a small team while isolating ingestion CPU and memory from interactive traffic. PostgreSQL is the source of truth for document and job state; OpenSearch holds the derived retrieval representation.

flowchart LR
    User["Authenticated user"] --> Api["ECS Fargate query API"]
    Api --> Auth["Amazon Cognito or enterprise OIDC"]
    Api --> Cache["ElastiCache Redis"]
    Api --> Search["Amazon OpenSearch Service"]
    Api --> Rank["ECS Fargate reranker"]
    Api --> Bedrock["Amazon Bedrock"]
    Api --> Db["Amazon RDS PostgreSQL"]
    Source["S3 source bucket"] --> Queue["Amazon SQS"]
    Queue --> Worker["ECS Fargate ingestion worker"]
    Worker --> Parse["Document parsing and chunking"]
    Parse --> Bedrock
    Parse --> Search
    Worker --> Db
    Search --> Api
    Db --> Api
    Worker --> Dlq["SQS dead-letter queue"]

Caption: Query and ingestion workloads share application code but use separate ECS task services and scaling policies.

The query API authenticates the caller, derives tenant and access claims from trusted identity context, and computes a cache key that includes those claims. It embeds the query, retrieves lexical and vector candidates, fuses their ranks, and reranks the bounded result set. Only authorized evidence is passed to the generation step. Responses include references that the UI can resolve to a permitted source location.

The ingestion worker consumes S3 object events through SQS. It parses and normalizes content, applies versioned chunking rules, calls the embedding model, upserts chunks into OpenSearch, and updates relational job state. A source update is not complete merely because embedding succeeded: the worker records the index generation and reports completion only after the searchable representation is committed. Deletions are explicit tombstones that remove old chunks.

Redis is an optimization, not a system of record. Cache only results whose authorization context, index generation, and prompt version are part of the cache key. Do not cache unfiltered search results and rely on a later layer to remove unauthorized passages.

4. Cloud service mapping

AWS is selected for this blueprint. The equivalents below are architectural counterparts, not feature-for-feature substitutes; validate networking, ranking, identity, and model behavior before a cross-cloud migration.

ComponentAWS choiceAzure counterpartGCP counterpart
Containers and APIAmazon ECS on AWS FargateAzure Container AppsCloud Run
Object storageAmazon S3Azure Blob StorageCloud Storage
Work queueAmazon SQSAzure Service Bus queuesCloud Tasks or Pub/Sub
Relational metadataAmazon RDS for PostgreSQL, Multi-AZAzure Database for PostgreSQLCloud SQL for PostgreSQL
Hybrid retrieval indexAmazon OpenSearch ServiceAzure AI SearchVertex AI Search, or a managed vector/search design
Embedding and generationAmazon BedrockAzure AI FoundryVertex AI
RerankingECS service running an approved cross-encoderContainer Apps or Azure ML endpointCloud Run or Vertex AI endpoint
CacheAmazon ElastiCache for RedisAzure Managed RedisMemorystore
Secrets and keysAWS Secrets Manager and AWS KMSKey VaultSecret Manager and Cloud KMS
Metrics and logsAmazon CloudWatchAzure MonitorCloud Monitoring and Cloud Logging
TracingOpenTelemetry collector/export to an approved backendAzure Monitor or another OTLP backendCloud Trace or another OTLP backend

A managed search product can reduce operational work, but it does not remove the need to inspect ranking behavior, filters, refresh semantics, and relevance evaluation. Keep the application-level fusion and evaluation contracts stable enough that changing search backends does not require rewriting the product’s quality gates.

5. Data design

Use PostgreSQL for source identity, access metadata, ingestion state, and version history. Store original files in versioned S3 buckets. OpenSearch is a derived index: each indexed chunk carries a stable chunk ID, tenant ID, document ID, source version, ACL attributes, text, and embedding. Rebuild it from S3 and PostgreSQL rather than treating it as the only copy.

Chunk IDs should be deterministic for a given document version and chunking configuration. That makes retries idempotent and prevents a failed re-ingestion from producing duplicate passages. Keep the chunker version and embedding model version in metadata; otherwise, teams cannot explain why two documents were indexed differently or safely roll out a new strategy.

erDiagram
    TENANT ||--o{ DOCUMENT : owns
    DOCUMENT ||--o{ DOCUMENT_VERSION : has
    DOCUMENT_VERSION ||--o{ INGESTION_JOB : processed_by
    DOCUMENT_VERSION ||--o{ CHUNK_REFERENCE : produces
    TENANT ||--o{ EVALUATION_RUN : scopes
    EVALUATION_RUN ||--o{ EVALUATION_RESULT : contains

Caption: PostgreSQL records lifecycle and provenance; the chunk reference identifies derived records stored in OpenSearch.

CREATE TABLE tenant (
    tenant_id       uuid PRIMARY KEY,
    name            text NOT NULL,
    created_at      timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE document (
    document_id     uuid PRIMARY KEY,
    tenant_id       uuid NOT NULL REFERENCES tenant(tenant_id),
    source_key      text NOT NULL,
    title           text NOT NULL,
    access_labels   jsonb NOT NULL DEFAULT '[]'::jsonb,
    active_version  text,
    deleted_at      timestamptz,
    created_at      timestamptz NOT NULL DEFAULT now(),
    UNIQUE (tenant_id, source_key)
);

CREATE TABLE document_version (
    tenant_id          uuid NOT NULL,
    document_id        uuid NOT NULL REFERENCES document(document_id),
    source_version      text NOT NULL,
    content_sha256      char(64) NOT NULL,
    chunker_version     text NOT NULL,
    embedding_model     text NOT NULL,
    status              text NOT NULL CHECK
                        (status IN ('queued','processing','indexed','failed','deleted')),
    chunk_count         integer NOT NULL DEFAULT 0 CHECK (chunk_count >= 0),
    indexed_at          timestamptz,
    PRIMARY KEY (document_id, source_version)
);

CREATE TABLE ingestion_job (
    job_id          uuid PRIMARY KEY,
    tenant_id       uuid NOT NULL,
    document_id     uuid NOT NULL REFERENCES document(document_id),
    source_version  text NOT NULL,
    attempt         integer NOT NULL DEFAULT 0,
    status          text NOT NULL CHECK
                    (status IN ('queued','running','succeeded','failed','dead_letter')),
    error_code      text,
    created_at      timestamptz NOT NULL DEFAULT now(),
    updated_at      timestamptz NOT NULL DEFAULT now(),
    UNIQUE (document_id, source_version)
);

CREATE INDEX ingestion_job_status_created_idx
    ON ingestion_job (status, created_at);

CREATE TABLE chunk_reference (
    chunk_id           text PRIMARY KEY,
    tenant_id          uuid NOT NULL REFERENCES tenant(tenant_id),
    document_id        uuid NOT NULL REFERENCES document(document_id),
    source_version     text NOT NULL,
    ordinal            integer NOT NULL,
    token_count        integer NOT NULL,
    index_generation   text NOT NULL,
    UNIQUE (document_id, source_version, ordinal)
);

CREATE TABLE evaluation_run (
    run_id          uuid PRIMARY KEY,
    dataset_version text NOT NULL,
    pipeline_version text NOT NULL,
    started_at      timestamptz NOT NULL DEFAULT now(),
    completed_at    timestamptz,
    status          text NOT NULL
);

CREATE TABLE evaluation_result (
    run_id          uuid NOT NULL REFERENCES evaluation_run(run_id),
    case_id         text NOT NULL,
    metric_name     text NOT NULL,
    metric_value    double precision NOT NULL,
    details         jsonb NOT NULL DEFAULT '{}'::jsonb,
    PRIMARY KEY (run_id, case_id, metric_name)
);

The index should use separate fields for searchable text, exact filters, and vector values. Map IDs and access attributes as exact-match fields; do not make tenant or ACL fields analyzed text. Keep source text out of operational logs. For deletion requests, remove index records and cached answers, then retain only the minimum audit record allowed by policy.

6. Low-level design

Keep the application organized around a small number of modules: ingestion, retrieval, generation, identity, and evaluation. The interfaces matter more than the folder layout. In particular, keep chunking configuration, retrieval candidate limits, fusion weights, and prompt version explicit so that evaluation can attribute a quality change to a specific pipeline revision.

classDiagram
    class IngestionWorker {
      +handle(message)
      +process_document(source)
    }
    class Chunker {
      +chunk(text, config)
    }
    class Embedder {
      +embed(texts)
    }
    class SearchRepository {
      +index(chunks)
      +hybrid_search(query, filters)
    }
    class Ranker {
      +rerank(query, candidates)
    }
    class AnswerService {
      +answer(query, evidence)
    }
    IngestionWorker --> Chunker
    IngestionWorker --> Embedder
    IngestionWorker --> SearchRepository
    SearchRepository --> Ranker
    Ranker --> AnswerService

Caption: Domain interfaces keep pipeline behavior testable without binding evaluation logic to one provider SDK.

Minimal runnable example: chunk boundaries

This example uses character windows with overlap. It is deliberately small enough to run directly, but production chunking should use tokenizer-aware limits and preserve structural boundaries such as headings, paragraphs, and table rows.

def chunk_text(text: str, size: int = 500, overlap: int = 80) -> list[str]:
    if size <= 0 or overlap < 0 or overlap >= size:
        raise ValueError("Require size > overlap >= 0")
    text = " ".join(text.split())
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(text[start:end])
        if end == len(text):
            break
        start = end - overlap
    return chunks


if __name__ == "__main__":
    for i, chunk in enumerate(chunk_text("A short example document. " * 40)):
        print(i, len(chunk), chunk[:48])

What to notice: overlap preserves local continuity, but character limits are not model-token limits and can split a sentence or table.

Realistic runnable example: hybrid retrieval and reranking

The following script queries an OpenSearch domain twice—once with BM25 and once with k-NN—then applies reciprocal-rank fusion and a cross-encoder reranker. It assumes the index has text, embedding, tenant_id, and document_id fields, and that the caller has already validated the tenant identity. Install opensearch-py, boto3, sentence-transformers, and requests-aws4auth; set the environment variables and AWS credentials before running. The selected cross-encoder must be approved and available to the service.

import json
import os

import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
from sentence_transformers import CrossEncoder

region = os.environ["AWS_REGION"]
host = os.environ["OPENSEARCH_HOST"]
index = os.getenv("OPENSEARCH_INDEX", "rag-chunks")
model_id = os.getenv("EMBED_MODEL", "amazon.titan-embed-text-v2:0")

credentials = boto3.Session().get_credentials().get_frozen_credentials()
auth = AWS4Auth(
    credentials.access_key, credentials.secret_key, region, "es",
    session_token=credentials.token,
)
search = OpenSearch(
    hosts=[{"host": host, "port": 443}],
    http_auth=auth, use_ssl=True, verify_certs=True,
    connection_class=RequestsHttpConnection, timeout=3,
)
bedrock = boto3.client("bedrock-runtime", region_name=region)
reranker = CrossEncoder(os.getenv(
    "RERANK_MODEL", "cross-encoder/ms-marco-MiniLM-L-6-v2"
))


def embed(text: str) -> list[float]:
    response = bedrock.invoke_model(
        modelId=model_id,
        body=json.dumps({"inputText": text, "dimensions": 1024, "normalize": True}),
    )
    return json.loads(response["body"].read())["embedding"]


def retrieve(query: str, tenant_id: str, limit: int = 5) -> list[dict]:
    vector = embed(query)
    lexical = search.search(index=index, body={
        "size": 40,
        "query": {"bool": {
            "must": [{"match": {"text": query}}],
            "filter": [{"term": {"tenant_id": tenant_id}}],
        }},
        "_source": ["text", "document_id"],
    })["hits"]["hits"]

    dense = search.search(index=index, body={
        "size": 40,
        "query": {"knn": {"embedding": {
            "vector": vector, "k": 40,
            "filter": {"term": {"tenant_id": tenant_id}},
        }}},
        "_source": ["text", "document_id"],
    })["hits"]["hits"]

    fused = {}
    for ranked_list in (lexical, dense):
        for rank, hit in enumerate(ranked_list, start=1):
            item = fused.setdefault(hit["_id"], {
                "id": hit["_id"], **hit["_source"], "rrf": 0.0
            })
            item["rrf"] += 1.0 / (60 + rank)

    candidates = sorted(fused.values(), key=lambda x: x["rrf"], reverse=True)[:40]
    scores = reranker.predict([(query, item["text"]) for item in candidates])
    for item, score in zip(candidates, scores):
        item["rerank_score"] = float(score)
    return sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)[:limit]


if __name__ == "__main__":
    results = retrieve(os.environ["QUERY"], os.environ["TENANT_ID"])
    for result in results:
        print(result["document_id"], round(result["rerank_score"], 4), result["text"][:160])

What to notice: authorization filtering is applied in both retrieval requests, fusion uses rank rather than incomparable raw scores, and reranking happens only after candidate generation.

In production, validate index mappings and model dimensions at startup, reuse clients and the loaded reranker across requests, and bound all candidate counts and timeouts. If the reranker cannot respond within its budget, return the fused ranking with a degraded-response signal rather than blocking indefinitely.

Chunking is a quality decision, not just a storage decision. Start with structure-aware chunks and a modest overlap; evaluate alternatives against the same corpus and query set. Larger chunks may preserve context but dilute relevance and increase prompt cost. Smaller chunks may improve precision but separate a fact from its qualification. Track chunker version, token count, section path, and source offsets so answers can cite useful locations.

Hybrid retrieval handles complementary failure modes. Lexical search is strong for identifiers, error codes, names, and exact phrases. Dense retrieval can find semantically related wording. Do not add their raw scores unless calibration has been demonstrated; rank fusion is a safer first baseline. Reranking then spends more compute on a much smaller candidate set to improve ordering. It cannot recover a relevant passage that neither retriever returned.

7. Key flows

The ingestion flow is asynchronous and idempotent. S3 events are notifications, not a complete source-of-truth log; the worker verifies the object version and content hash before doing expensive work. A retry may repeat parsing or embedding, so writes use deterministic chunk IDs and upserts.

sequenceDiagram
    participant S3 as S3
    participant Q as SQS
    participant W as Ingestion worker
    participant B as Bedrock embeddings
    participant O as OpenSearch
    participant P as PostgreSQL
    S3->>Q: Object event
    Q->>W: Deliver message
    W->>S3: Read object version
    W->>P: Claim idempotent job
    W->>W: Parse and chunk
    W->>B: Embed batches
    B-->>W: Vectors
    W->>O: Upsert versioned chunks
    W->>P: Mark indexed
    W-->>Q: Acknowledge

Caption: A job is acknowledged only after indexing and state updates succeed; failures remain retryable or move to the dead-letter queue.

The query flow treats retrieval as a sequence of bounded stages. Every stage records latency and outcome, but sensitive content is not copied into telemetry. The generator receives only the evidence necessary to answer the question, not the full source corpus.

sequenceDiagram
    participant U as User
    participant A as Query API
    participant I as Identity
    participant O as OpenSearch
    participant R as Reranker
    participant B as Bedrock generation
    U->>A: Ask question
    A->>I: Validate identity and claims
    I-->>A: Tenant and access scope
    A->>A: Embed query
    A->>O: Lexical and vector search
    O-->>A: Candidate passages
    A->>R: Rank bounded candidates
    R-->>A: Ordered evidence
    A->>B: Question and authorized evidence
    B-->>A: Answer with citations
    A-->>U: Answer and source references

Caption: The API establishes access scope before search and passes only authorized, reranked evidence to generation.

8. Build and release

Infrastructure changes should be reviewed with application changes. Keep environment-specific values out of Terraform code, use remote state with locking and encryption, and create separate staging and production accounts or equivalent isolated environments. The snippet below establishes an encrypted SQS queue and dead-letter queue; production infrastructure also needs least-privilege policies, private networking, encryption keys, and alarms.

resource "aws_sqs_queue" "ingestion_dlq" {
  name                      = "rag-ingestion-dlq"
  message_retention_seconds = 1209600
  sqs_managed_sse_enabled   = true
}

resource "aws_sqs_queue" "ingestion" {
  name                       = "rag-ingestion"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 345600
  sqs_managed_sse_enabled    = true

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.ingestion_dlq.arn
    maxReceiveCount     = 5
  })
}

This GitHub Actions example runs quality checks, builds an image, and deploys to staging after tests pass. Configure AWS federation through GitHub OIDC rather than long-lived access keys; production promotion should be a separate protected environment with approval and a rollback path.

name: rag-ci
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  test-and-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -q
      - run: ruff check .
      - run: docker build -t rag-api:${{ github.sha }} .
      - if: github.ref == 'refs/heads/main'
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.STAGING_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}
      - if: github.ref == 'refs/heads/main'
        run: |
          ./scripts/push-image.sh rag-api:${{ github.sha }}
          ./scripts/deploy-staging.sh ${{ github.sha }}

Test strategy:

  • Unit: chunk boundaries, stable IDs, ACL filter construction, fusion ordering, citation formatting, and retry classification.
  • Integration: parsing fixtures, PostgreSQL state transitions, OpenSearch mappings and filters, and model-client timeout behavior.
  • Contract: validate source event schemas, index document shape, model request/response parsing, and API response compatibility.
  • Load: replay representative query and ingestion mixes against staging; measure p95 by stage, index pressure, queue age, connection pools, and model throttling. Use realistic document lengths and query distributions, not only synthetic short prompts.
  • Evaluation: run a fixed dataset before and after changes to chunker, embedding model, index settings, reranker, prompt, or generation model. Block releases on agreed regressions, not on a single aggregate score that can hide a harmed user segment.

Promote the same immutable image through staging and production. Index and prompt changes should be independently versioned so a rollback does not require rebuilding all source documents. Run a small canary query set against a new index generation before switching production traffic.

9. Observability

Instrument the request as a trace with spans for authentication, embedding, lexical retrieval, vector retrieval, fusion, reranking, generation, and citation validation. Record model and pipeline versions as attributes, but do not attach raw prompts, answers, or retrieved text by default. For debugging sensitive content, use an explicit, audited, time-limited diagnostic mechanism with redaction and access controls.

A structured log event should support correlation without turning logs into a secondary content store:

{
  "timestamp": "2026-11-10T12:00:00Z",
  "level": "INFO",
  "service": "rag-api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "request_id": "req_8d2c",
  "tenant_hash": "sha256:9a7f...",
  "pipeline_version": "retrieval-17",
  "index_generation": "chunks-2026-11-10-a",
  "candidate_counts": {"lexical": 40, "dense": 40, "fused": 40, "reranked": 5},
  "stage_ms": {"embed": 84, "search": 112, "rerank": 96, "generation": 1290},
  "outcome": "answered"
}

For RED, monitor request rate, error rate, and duration for the API and each model/search dependency. For USE, monitor utilization, saturation, and errors for ECS CPU/memory, OpenSearch nodes and thread pools, RDS connections/storage, Redis memory/evictions, and SQS age and in-flight messages. Separate user-visible errors from expected “no evidence” outcomes.

OpenTelemetry instrumentation should propagate context across HTTP and provider calls. This Python example creates a span around retrieval; initialize the provider and exporter at process startup and use an OTLP-compatible backend.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("rag.retrieval")

def traced_retrieve(query: str, tenant_id: str):
    with tracer.start_as_current_span("rag.retrieve") as span:
        span.set_attribute("rag.pipeline.version", "retrieval-17")
        span.set_attribute("rag.tenant.hash", hash_tenant(tenant_id))
        try:
            result = retrieve(query, tenant_id)
            span.set_attribute("rag.result.count", len(result))
            return result
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise
SLIInitial SLOMeasurement
Query API availability99.9% monthlySuccessful non-5xx requests divided by eligible requests
Query latency95% under 4 secondsEnd-to-end request duration; track streaming separately
Searchable freshness95% within 15 minutesSource accepted-to-indexed duration
Ingestion completion99% of valid jobs within 30 minutesSuccessful jobs, excluding documented source-side delays
Citation integrity99% of returned citations resolve to an authorized sourceSampled automated check plus evaluation set

Dashboard panels should include API RED, dependency latency/error breakdown, cache hit ratio, candidate counts, retrieval-empty rate, generation token volume, SQS oldest-message age, dead-letter count, and index freshness. Add a quality dashboard for evaluation trends by query category and tenant segment. Aggregate quality scores must not conceal a regression on exact-match or access-control cases.

Page on sustained API SLO burn, a high-severity authorization or data-isolation failure, search or model dependency failure that breaks the user path, growing queue age beyond the freshness objective, and exhausted database or search capacity. Alert on trends and actionable conditions; route evaluation-score changes to release review rather than paging an on-call engineer at night.

10. Security and compliance

Identity and authorization are part of retrieval correctness. Validate issuer, audience, signature, expiry, and required claims at the API boundary. Derive tenant and ACL filters from trusted identity and server-side policy—not from user-provided request fields. Apply the filter to both lexical and vector search, then validate candidate access again before generation. Treat cache entries as access-controlled data.

Use private subnets for ECS tasks, RDS, and OpenSearch where supported by the chosen deployment. Restrict egress to required AWS endpoints and approved model services. Use task roles scoped to specific buckets, queues, indexes, and secrets; avoid credentials in images or environment files. Encrypt S3, database, search, queue, and backups with managed keys according to the organization’s key policy.

Additional controls:

  • Define allowed file types and enforce size, decompression, and parser time limits; treat uploaded documents as untrusted input.
  • Protect against prompt injection in retrieved content. Retrieved passages are evidence, not instructions; delimit them and instruct the model not to follow embedded commands.
  • Apply malware scanning and content classification if required by the data policy.
  • Establish retention and deletion behavior for source versions, index records, caches, logs, backups, and evaluation fixtures.
  • Record model/provider, prompt version, source references, and policy version for audit without logging the full private content.
  • Review provider data handling, regional processing, model access, and contractual terms before sending regulated or sensitive content.
  • Keep evaluation datasets permissioned and scrubbed; test that unauthorized documents never appear in candidate results or citations.

Compliance is not a property that can be inferred from using a managed service. Map the actual data classes and processing locations to the organization’s controls, then verify the configuration and operational evidence.

11. Cost

The following is an assumption-based monthly planning estimate, not a current AWS quote. It assumes one production region, multi-AZ database and application deployment, a modest OpenSearch cluster sized for the stated corpus, 140,000 daily queries, and regular document updates. Model spend is especially sensitive to prompt length, output length, caching, and model choice.

Cost areaMonthly planning assumption
ECS Fargate API and ingestion tasks$1,100
Amazon OpenSearch Service$2,200
RDS PostgreSQL Multi-AZ$800
Bedrock embeddings and generation$3,200
S3 storage and requests$150
ElastiCache Redis$350
SQS, CloudWatch, and telemetry$500
Data transfer and miscellaneous networking$400
Planning total$8,700

Validate line items with the AWS Pricing Calculator for the chosen region and actual instance types. Before launch, measure tokens per request and cache hit rate; generation and embedding costs can change materially with model choice and usage. Keep an explicit per-request token budget and report spend by environment, model, and workload. If the budget is exceeded, first reduce unnecessary context and repeated embedding—not silently lower answer quality or remove authorization checks.

12. Failure modes, resilience and disaster recovery

Failure modeUser or system impactResponse
SQS delivery repeatsDuplicate ingestion workIdempotent job key and deterministic chunk IDs; safe upsert
Parser fails on a documentThat source is stale or unavailableRetry transient failures; quarantine invalid inputs; expose job status
Bedrock throttles or times outSlow ingestion or incomplete answersBounded retries with jitter; concurrency limits; queue backpressure
OpenSearch is degradedRetrieval errors or latencyFail fast within a budget; return a clear unavailable response rather than unsupported answers
Reranker is unavailableLower ordering qualityUse fused ranking as a documented degraded mode
Stale index after an updateOld content may be retrievedTrack source version; suppress superseded versions; alert on freshness lag
Redis contains outdated answerStale responseVersion cache keys by index and prompt; bounded TTL and targeted invalidation
Tenant filter regressionPotential data exposureDeny by default; integration tests; policy-level alarm and incident procedure

Use exponential backoff with jitter for transient dependency failures, but cap retries so a single request cannot occupy resources indefinitely. Set queue visibility timeout above normal job duration and extend it for long jobs where appropriate. Use a dead-letter queue for repeated failures, with an operator workflow that can inspect a redacted error, correct the cause, and replay safely.

Deploy the API and workers across Availability Zones. PostgreSQL backups and point-in-time recovery should be configured and restore-tested. Treat OpenSearch as rebuildable derived state, but recognize that a full rebuild can take long enough to affect freshness objectives; retain the source and indexing metadata needed to rebuild it. Define recovery time and recovery point objectives with product owners. A second region is not justified by this medium-scale baseline unless the business recovery requirement demands it.

During a regional or provider outage, fail closed for authentication and authorization. Do not fall back to an unfiltered cache or a stale index whose access policy cannot be trusted. If generation fails after retrieval, an optional degraded response can return authorized citations or a clear retry message, but it must not present unsupported prose as an answer.

13. Evolving to the next scale

Do not split the modular monolith just because it contains several modules. Split only when operational or ownership boundaries justify the cost of another deployable service.

Signal observed in productionLikely next change
API and ingestion contend for CPU or memoryKeep separate ECS services and tune task scaling independently; move heavy parsing to a dedicated worker image
Queue freshness SLO misses during ordinary change volumeIncrease worker concurrency with provider-aware throttles; partition workloads by source or tenant
Search latency or capacity is the dominant bottleneckTune mappings, shard strategy, filters, and candidate limits; load-test a larger or separated search deployment
Reranking dominates tail latency or costReduce candidate count using evaluation evidence, batch inference, or operate a separately scaled inference service
One index cannot meet isolation or lifecycle needsPartition indexes by access domain or retention class, with explicit routing and migration tooling
Multiple teams need independent release ownershipExtract ingestion or evaluation only after contracts, on-call ownership, and deployment boundaries are clear
Recovery requirements demand regional continuityBuild and test a second-region restore or replication plan with defined consistency and failover behavior

A change in architecture should follow measured pressure. For example, a larger candidate set may improve recall but increase reranker cost; a more specialized index may improve filtering while adding operational work. Every such change should enter the evaluation harness first, then a staging load test, then a controlled production rollout.

14. Phased implementation plan

  1. Establish the quality baseline. Gather representative questions, expected evidence, authorization cases, and “should abstain” examples. Version the dataset and agree on retrieval, citation, and latency measures before selecting a final chunking strategy.
  2. Build ingestion and provenance. Implement S3-to-SQS processing, parsing, deterministic chunk IDs, PostgreSQL job state, retries, deletion handling, and a small searchable corpus. Confirm replays and partial failures are safe.
  3. Add hybrid retrieval. Implement lexical and vector candidate retrieval with tenant filtering in both paths. Compare each retriever independently, then test rank fusion. Record retrieval results for evaluation without retaining unnecessary private content.
  4. Add reranking and grounded generation. Bound the candidate set, introduce a cross-encoder, and pass only authorized evidence to the model. Require citations and an abstention path. Test prompt injection and citation resolution.
  5. Harden the operational path. Add timeouts, rate limits, queue alarms, tracing, dashboards, cost attribution, secrets management, and restore procedures. Run integration and load tests in staging using representative traffic shapes.
  6. Release behind controlled exposure. Promote an immutable build, run a canary query set, compare quality and latency to the baseline, then gradually expand traffic. Keep the previous index generation and application version available for rollback.
  7. Review after real usage. Inspect failure classes, freshness, retrieval misses, cache behavior, latency, and cost by workload. Change one major quality variable at a time—chunking, embedding, fusion, reranker, or prompt—and require evaluation evidence before rollout.

The production bar is not “the model can answer a demo question.” It is that the system can find authorized evidence, explain where that evidence came from, decline when it has none, and remain diagnosable when any stage fails. Chunking, hybrid retrieval, reranking, and evaluation are one pipeline; operate and release them as such.


This article was generated with AI. #AIGenerated