1. Context and scale
A research assistant is not just a chat endpoint with a search tool. It must turn an open-ended question into bounded work, gather evidence from multiple sources, detect gaps or contradictions, and return an answer whose claims can be inspected. This blueprint uses LangGraph to coordinate a planner, parallel research workers, a critic, and a synthesizer. The workflow is asynchronous: the API accepts a request, places a job on a queue, and lets the user poll for progress and the final report.
The design targets a medium-sized product: a handful of services, one primary cloud, one or two engineering teams, and enough traffic to justify a queue and a multi-AZ database—but not a fleet of independently deployed agents. The graph is application logic inside a worker service, not a collection of autonomous services.
Assumptions — planning inputs, not forecasts:
| Dimension | Assumption |
|---|---|
| Users | 20,000 daily active users; within a 10,000–100,000 daily-user product |
| API traffic | 1 request/second average, 8 requests/second peak; research jobs are a subset |
| Research jobs | 1,500 jobs/day; up to 4 parallel worker tasks per job |
| Data volume | 500 GB of source documents and extracted text; growth is mostly append-only |
| Team | 5–7 engineers across product and platform responsibilities |
| Monthly budget | $4,500–$14,000, including an assumption-based model and search allowance |
Treat these values as capacity-planning inputs. Actual model, search, and storage costs depend on prompt size, model selection, retrieval volume, cache hit rate, and provider pricing. Put hard limits on those variables before optimizing infrastructure.
The application accepts a question, optionally scopes it to an internal corpus, and returns a cited report. The internal corpus is curated and indexed ahead of time; workers can also query an approved external search provider. This is a research assistant, not an autonomous browser with permission to act on arbitrary websites or make changes to business systems.
2. Requirements
Functional requirements describe user-visible behavior and boundaries. Non-functional requirements turn those boundaries into measurable operating goals.
| Functional requirement | Design implication |
|---|---|
| Accept a question and return a job identifier | API validates, persists, and enqueues; it does not hold a request open for the entire research run |
| Plan a bounded set of research tasks | Planner emits structured tasks with a maximum count and explicit query scope |
| Research tasks concurrently | Workers use bounded fan-out, per-provider timeouts, and a shared job deadline |
| Retrieve internal and approved external sources | Retrieval adapters return normalized evidence with canonical URLs and timestamps |
| Critique evidence before synthesis | Critic checks coverage, source quality, contradictions, and unsupported claims |
| Return citations and job progress | Persist source metadata and job state; expose polling endpoints |
| Support user cancellation and retention | Check cancellation between graph steps; apply tenant-aware retention policies |
| Non-functional requirement | Initial target / constraint |
|---|---|
| Availability | 99.9% monthly availability for job submission and status APIs |
| Latency | API acknowledgement under 500 ms at p95; report completion is asynchronous and separately measured |
| Reliability | At-least-once queue delivery with idempotent job processing |
| Safety | No arbitrary tool execution; worker tools are allowlisted, read-only, and time-bounded |
| Quality | Every factual answer claim should have one or more evidence references, or be identified as uncertain |
| Isolation | Tenant-scoped authorization and data access; no cross-tenant retrieval |
| Cost control | Limits on task count, search calls, tokens, retries, and total job duration |
| Operability | Trace graph steps and provider calls without logging prompts, answers, or document contents by default |
A critic can identify weak evidence, but it cannot prove that a claim is true. The system’s quality contract is therefore narrower: make evidence provenance visible, reject unsupported statements where possible, and return uncertainty when available sources do not support a conclusion.
3. High-level design
AWS is the chosen cloud. The application has two deployable workloads: an API and a queue-driven worker. Both are containerized on Amazon ECS with AWS Fargate. The API handles authentication, validation, job creation, and status reads. The worker runs LangGraph, calls Amazon Bedrock for language-model inference, retrieves internal evidence, and uses an approved external search adapter.
flowchart LR
User["User"] --> Cognito["Amazon Cognito"]
Cognito --> ALB["Application Load Balancer"]
ALB --> API["ECS Fargate API"]
API --> Queue["Amazon SQS"]
API --> DB["Amazon Aurora PostgreSQL"]
Queue --> Worker["ECS Fargate LangGraph worker"]
Worker --> DB
Worker --> Bedrock["Amazon Bedrock"]
Worker --> Search["Approved search provider"]
Worker --> Index["Amazon OpenSearch Service"]
Index --> S3["Amazon S3 corpus"]
Worker --> Cache["Amazon ElastiCache Redis"]
Worker --> Telemetry["CloudWatch and OpenTelemetry"]The API creates durable work; the worker owns graph execution and evidence gathering.
The index contains searchable representations of approved internal material, with the original objects in S3. Aurora PostgreSQL stores jobs, source metadata, graph checkpoints, and results. OpenSearch provides lexical and vector retrieval; do not introduce a separate vector database until retrieval measurements show a need. Redis is an optional short-lived cache for repeated retrieval or provider responses, not the source of truth.
A job progresses through explicit states such as queued, running, completed, failed, and cancelled. A queue message contains an opaque job identifier, not the full question or sensitive evidence. The worker loads authorized job state from the database, executes the graph, and writes durable progress and results. API clients poll the status endpoint; server-sent events can be added later if product requirements justify maintaining connections.
4. Cloud service mapping
The AWS choices favor managed services and a small operational surface. Equivalents are approximate: feature sets, networking, identity models, and regional availability differ, so a migration is not a name-for-name replacement.
| Component | AWS choice | Azure equivalent | GCP equivalent |
|---|---|---|---|
| Containerized API and worker | Amazon ECS on AWS Fargate | Azure Container Apps | Cloud Run |
| Ingress and TLS | Application Load Balancer, ACM | Application Gateway, Key Vault certificates | Cloud Load Balancing, Certificate Manager |
| User identity | Amazon Cognito | Microsoft Entra External ID | Identity Platform |
| Job queue and dead-letter queue | Amazon SQS | Azure Service Bus queues | Pub/Sub |
| Transactional data and checkpoints | Amazon Aurora PostgreSQL, Multi-AZ | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Internal search index | Amazon OpenSearch Service | Azure AI Search | Vertex AI Search or managed Elasticsearch-compatible service |
| Source objects | Amazon S3 | Azure Blob Storage | Cloud Storage |
| Model inference | Amazon Bedrock | Azure AI Foundry models | Vertex AI |
| Optional cache | Amazon ElastiCache for Redis | Azure Managed Redis | Memorystore |
| Secrets and encryption keys | AWS Secrets Manager, AWS KMS | Key Vault | Secret Manager, Cloud KMS |
| Logs and service metrics | Amazon CloudWatch | Azure Monitor | Cloud Monitoring and Cloud Logging |
| Distributed tracing | OpenTelemetry Collector and an OTLP backend | Azure Monitor OpenTelemetry | Google Cloud Trace |
Prefer the managed PostgreSQL service that your team can operate well. LangGraph checkpoint persistence should use the same operational discipline as application data: backups, connection limits, schema migration controls, and restore tests. The AWS service choice does not remove those responsibilities.
5. Data design
Keep job lifecycle, graph execution, and evidence provenance separate enough to reason about, but avoid turning every graph node into a database table. The graph checkpoint is execution state; the application tables are the durable product contract. Do not expose checkpoint internals as an API response.
erDiagram
TENANT ||--o{ RESEARCH_JOB : owns
RESEARCH_JOB ||--o{ RESEARCH_RUN : has
RESEARCH_RUN ||--o{ RESEARCH_FINDING : records
RESEARCH_FINDING ||--o{ CITATION : cites
RESEARCH_JOB ||--o{ JOB_EVENT : emits
RESEARCH_JOB {
uuid id PK
uuid tenant_id FK
text status
text question
timestamptz created_at
}
RESEARCH_RUN {
uuid id PK
uuid job_id FK
int attempt
text graph_version
timestamptz started_at
}
RESEARCH_FINDING {
uuid id PK
uuid run_id FK
text task_id
text summary
}
CITATION {
uuid id PK
uuid finding_id FK
text canonical_url
text title
}
JOB_EVENT {
bigserial id PK
uuid job_id FK
text event_type
timestamptz created_at
}The application data model records ownership, attempts, findings, citations, and auditable lifecycle events.
An initial schema can use PostgreSQL JSONB for evolving report structures while keeping query-critical fields relational. Apply tenant filters in the data-access layer and reinforce them with database row-level security where the team can test and maintain the policies. Checkpoints are stored by the LangGraph Postgres saver; they are intentionally not duplicated in this schema.
CREATE TABLE tenant (
id uuid PRIMARY KEY,
name text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE research_job (
id uuid PRIMARY KEY,
tenant_id uuid NOT NULL REFERENCES tenant(id),
status text NOT NULL CHECK (
status IN ('queued', 'running', 'completed', 'failed', 'cancelled')
),
question text NOT NULL,
result jsonb,
error_code text,
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz
);
CREATE INDEX research_job_tenant_created
ON research_job (tenant_id, created_at DESC);
CREATE TABLE research_run (
id uuid PRIMARY KEY,
job_id uuid NOT NULL REFERENCES research_job(id),
attempt integer NOT NULL CHECK (attempt > 0),
graph_version text NOT NULL,
started_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz,
UNIQUE (job_id, attempt)
);
CREATE TABLE research_finding (
id uuid PRIMARY KEY,
run_id uuid NOT NULL REFERENCES research_run(id),
task_id text NOT NULL,
summary text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE citation (
id uuid PRIMARY KEY,
finding_id uuid NOT NULL REFERENCES research_finding(id),
canonical_url text NOT NULL,
title text,
published_at timestamptz,
retrieved_at timestamptz NOT NULL DEFAULT now(),
content_hash text
);
CREATE TABLE job_event (
id bigserial PRIMARY KEY,
job_id uuid NOT NULL REFERENCES research_job(id),
event_type text NOT NULL,
details jsonb NOT NULL DEFAULT '{}'::jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
Partitioning is not an initial requirement at this scale. Start with indexed relational tables and a retention policy. If job_event grows enough to affect maintenance or query latency, partition it by creation time and archive old events. Keep document chunks and embeddings in the search index; keep source object versions and ingestion manifests in S3 so the index can be rebuilt.
6. Low-level design
Model the graph as a bounded state machine. Each node should have one responsibility and a typed input/output contract. The planner proposes no more than a small configured number of tasks; workers return evidence-backed findings; the critic decides whether coverage is sufficient; and the synthesizer produces the final response. A revision returns to planning only while the run budget permits.
classDiagram
class ResearchState {
+question: str
+tasks: list
+findings: list
+critique: str
+iteration: int
+report: dict
}
class Planner {
+create_tasks(state) dict
}
class Worker {
+research(task) dict
}
class Critic {
+review(state) dict
}
class Synthesizer {
+compose(state) dict
}
class SearchPort {
+search(query) list
}
class RetrievalPort {
+retrieve(query, tenant_id) list
}
Planner --> ResearchState
Worker --> SearchPort
Worker --> RetrievalPort
Critic --> ResearchState
Synthesizer --> ResearchStateGraph nodes depend on explicit retrieval and search ports rather than owning provider-specific behavior.
A minimal graph illustrates the LangGraph state and execution model without requiring a model account. It is runnable with pip install langgraph; the deterministic nodes make it useful as a local orchestration smoke test, not as a research implementation.
from typing import TypedDict
from langgraph.graph import END, START, StateGraph
class State(TypedDict):
question: str
plan: str
answer: str
def planner(state: State) -> dict:
return {"plan": f"Find evidence relevant to: {state['question']}"}
def worker(state: State) -> dict:
return {"answer": f"Research task completed: {state['plan']}"}
builder = StateGraph(State)
builder.add_node("planner", planner)
builder.add_node("worker", worker)
builder.add_edge(START, "planner")
builder.add_edge("planner", "worker")
builder.add_edge("worker", END)
app = builder.compile()
print(app.invoke({"question": "How does a queue improve resilience?"}))
Notice that graph state is explicit and the compiled graph can be tested without cloud services.
The realistic pattern below uses parallel Send work, an injected search adapter, a bounded critic loop, and Bedrock-backed chat models. Install langgraph, langchain-aws, and boto3; configure AWS credentials and BEDROCK_MODEL_ID. Supply a real, approved asynchronous search adapter in production. The placeholder adapter returns no web results, so this example runs the graph but does not perform external research.
import operator
import os
from typing import Annotated, TypedDict
from langchain_aws import ChatBedrockConverse
from langgraph.graph import END, START, StateGraph
from langgraph.types import Send
class State(TypedDict, total=False):
question: str
tasks: list[str]
findings: Annotated[list[str], operator.add]
feedback: str
iteration: int
report: str
model = ChatBedrockConverse(
model=os.environ["BEDROCK_MODEL_ID"],
temperature=0,
)
async def search(query: str) -> list[str]:
# Replace with an authenticated, rate-limited search-provider adapter.
return []
async def plan(state: State) -> dict:
prompt = (
"Create at most three short, distinct research queries. "
"Return one query per line. Question: "
f"{state['question']}\nFeedback: {state.get('feedback', 'none')}"
)
response = await model.ainvoke(prompt)
tasks = [line.strip("- ").strip() for line in response.content.splitlines()
if line.strip()][:3]
return {"tasks": tasks, "iteration": state.get("iteration", 0) + 1}
def dispatch(state: State):
return [Send("worker", {"question": state["question"], "task": task})
for task in state.get("tasks", [])]
async def worker(state: State) -> dict:
evidence = await search(state["task"])
prompt = (
"Summarize only what this evidence supports. Identify uncertainty. "
f"Question: {state['question']}\nTask: {state['task']}\n"
f"Evidence: {evidence}"
)
response = await model.ainvoke(prompt)
return {"findings": [response.content]}
async def critic(state: State) -> dict:
prompt = (
"Assess whether findings answer the question with sufficient evidence. "
"Reply PASS or REVISE followed by a short reason.\n"
f"Question: {state['question']}\nFindings: {state.get('findings', [])}"
)
response = await model.ainvoke(prompt)
return {"feedback": response.content}
def route_after_critic(state: State) -> str:
feedback = state.get("feedback", "")
if feedback.startswith("PASS") or state.get("iteration", 0) >= 2:
return "synthesize"
return "plan"
async def synthesize(state: State) -> dict:
prompt = (
"Answer the question using only the findings. Mark gaps and uncertainty. "
f"Question: {state['question']}\nFindings: {state.get('findings', [])}"
)
response = await model.ainvoke(prompt)
return {"report": response.content}
builder = StateGraph(State)
builder.add_node("plan", plan)
builder.add_node("worker", worker)
builder.add_node("critic", critic)
builder.add_node("synthesize", synthesize)
builder.add_edge(START, "plan")
builder.add_conditional_edges("plan", dispatch, ["worker"])
builder.add_edge("worker", "critic")
builder.add_conditional_edges(
"critic", route_after_critic, {"plan": "plan", "synthesize": "synthesize"}
)
builder.add_edge("synthesize", END)
graph = builder.compile()
Notice that fan-out is explicit, the critic loop has a hard iteration limit, and provider access is behind an adapter.
In production, use structured model outputs and schema validation instead of parsing free-form lines or prefixes. Persist source identifiers alongside each finding; do not expect the model to reliably reproduce URLs from raw text. Add per-node deadlines, a maximum evidence payload, a maximum report size, and cancellation checks. Use a LangGraph Postgres checkpointer with a stable thread_id derived from the job ID, and pin graph versions on each run so a deployment cannot silently reinterpret an in-flight checkpoint.
7. Key flows
The submission path is deliberately short. It authenticates the caller, checks tenant-level limits, creates a durable job record, and enqueues an identifier. If the database write succeeds but queue publication fails, a transactional outbox or a reconciler must ensure that the job is eventually queued.
sequenceDiagram
actor User
participant API as ECS API
participant DB as Aurora PostgreSQL
participant SQS as Amazon SQS
participant Worker as ECS worker
User->>API: Submit question
API->>DB: Authorize and create job
API->>SQS: Enqueue job identifier
API-->>User: Return job identifier
SQS->>Worker: Deliver message
Worker->>DB: Claim job and load state
Worker->>Worker: Run LangGraph research
Worker->>DB: Persist report and statusJob submission and graph execution are separate operations, so API latency is not model latency.
The second flow covers a single research run. Individual workers can call both internal retrieval and the approved search service, but each adapter must enforce its own timeout, rate limit, and result-size cap. Citation records should be persisted from retrieved evidence, not inferred from the final prose.
sequenceDiagram
participant Worker as LangGraph worker
participant Planner
participant Search as Search adapter
participant Index as OpenSearch
participant Model as Amazon Bedrock
participant Critic
Worker->>Planner: Create bounded tasks
Planner-->>Worker: Task list
par Research task
Worker->>Index: Retrieve tenant-scoped evidence
Index-->>Worker: Internal passages
and Research task
Worker->>Search: Search approved sources
Search-->>Worker: Normalized results
end
Worker->>Model: Summarize evidence
Model-->>Worker: Findings with source references
Worker->>Critic: Review coverage and support
Critic-->>Worker: Pass or bounded revision
Worker->>Model: Synthesize cited report
Model-->>Worker: Final reportParallel retrieval reduces serial dependency while bounded execution keeps work and cost predictable.
8. Build and release
Keep infrastructure-as-code in the application repository and review it like application code. This Terraform fragment defines a queue and dead-letter queue; a complete deployment also needs encryption, IAM policies, private networking, alarms, and service integrations. Queue visibility timeout must exceed the worker’s expected processing window, or the worker must extend visibility while it is active. The timeout is an explicit capacity assumption, not a universal setting.
resource "aws_sqs_queue" "research_dlq" {
name = "research-jobs-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sqs_queue" "research_jobs" {
name = "research-jobs"
visibility_timeout_seconds = 900
message_retention_seconds = 345600
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.research_dlq.arn
maxReceiveCount = 4
})
}
Notice that poison messages are isolated after a bounded number of deliveries rather than retried forever.
A staging environment should mirror production’s identity flow, database topology, queue semantics, and model access as closely as practical. The following GitHub Actions outline uses placeholders for the AWS role and ECS deployment details; configure short-lived credentials through GitHub OIDC rather than storing long-lived access keys.
name: build-test-deploy
on:
pull_request:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements-dev.txt
- run: ruff check .
- run: mypy app
- run: pytest -q
- run: python -m compileall app
deploy-staging:
if: github.event_name == 'push'
needs: test
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.STAGING_DEPLOY_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- run: terraform -chdir=infra init
- run: terraform -chdir=infra apply -auto-approve
- run: ./scripts/build-push-deploy.sh staging
- run: ./scripts/smoke-test.sh staging
Notice that tests gate staging deployment and that deployment assumes an environment-scoped role.
The test strategy should make graph behavior testable without a live model call. Use fakes for search and model adapters in unit tests, then reserve limited provider-backed tests for staging.
| Test layer | What to verify |
|---|---|
| Unit | Task limits, state transitions, citation validation, cancellation, retry classification, prompt and output-schema handling |
| Integration | PostgreSQL migrations and checkpoint persistence; SQS redelivery; tenant-scoped retrieval; S3/index ingestion |
| Contract | Search and model adapter schemas, timeout behavior, error mapping, provider response changes |
| Load | API submission and polling at peak assumptions; concurrent jobs; database pool saturation; queue age; model-provider throttling |
| Evaluation | Curated question set, citation coverage, evidence relevance, unsupported-claim checks, regressions by graph version |
Do not use a single aggregate “answer quality” score as the release gate. Inspect representative outputs, track measurable properties such as citation coverage, and make release decisions against a versioned evaluation set. A model change, prompt change, retrieval change, and graph change can each alter behavior; record those versions on each run.
9. Observability
Instrument the API, queue consumer, graph nodes, database operations, retrieval, and provider calls. Use OpenTelemetry trace context across the API-to-queue boundary by placing trace propagation data in message attributes. Do not put prompts, retrieved text, user identifiers, or access tokens into span attributes. Logs should carry stable correlation identifiers and bounded categorical fields.
A structured log event might look like this:
{
"timestamp": "2026-11-09T12:00:00Z",
"level": "INFO",
"service": "research-worker",
"event": "graph_node_completed",
"job_id": "opaque-job-id",
"tenant_id_hash": "tenant-hash",
"node": "critic",
"graph_version": "2026-11-09.3",
"duration_ms": 842,
"outcome": "revise",
"iteration": 1,
"trace_id": "otel-trace-id"
}
Log metadata that supports diagnosis; keep sensitive content out of normal application logs.
For RED, monitor rate, errors, and duration for API requests, queue processing, retrieval requests, model calls, and graph runs. For USE, monitor utilization, saturation, and errors on Fargate tasks, database connections and storage, queue backlog, and OpenSearch capacity. Break down metrics by operation and outcome, not by high-cardinality values such as job ID or user ID.
This Python example creates a manual OpenTelemetry span around a graph node. Configure an OTLP exporter and resource attributes at process startup; propagate the active context through SQS using the OpenTelemetry propagator rather than starting an unrelated trace in the worker.
from opentelemetry import trace
tracer = trace.get_tracer("research-assistant.graph")
async def traced_node(node_name: str, operation):
with tracer.start_as_current_span("langgraph.node") as span:
span.set_attribute("graph.node", node_name)
try:
result = await operation()
span.set_attribute("graph.outcome", "ok")
return result
except Exception as exc:
span.set_attribute("graph.outcome", "error")
span.record_exception(exc)
raise
Notice that the span records node identity and failure without recording question or evidence contents.
Use these initial objectives as service-level targets and revisit them after measuring real traffic. Completion time is a product-level experience with a long tail; define it against a specific job class and exclude user cancellation.
| SLI | Initial SLO | Measurement |
|---|---|---|
| Submission availability | 99.9% monthly | Successful authenticated submissions / valid submission attempts |
| Status API latency | 99% under 500 ms | Server-side latency for authorized status reads |
| Queue start delay | 95% under 60 seconds | Enqueue timestamp to worker claim |
| Job completion | 90% within 5 minutes | Completed eligible jobs within target window |
| Citation integrity | 99% of returned citations resolve to stored evidence | Valid citation references / returned references |
Dashboard panels should include API RED, queue age and dead-letter count, worker concurrency and task restarts, graph duration by node, model and search errors/latency, token use by model and job class, database connections and query latency, retrieval hit rate, and SLO burn rates. Alert on user impact or imminent loss of capacity: submission error-rate burn, sustained queue age, dead-letter growth, worker inability to claim messages, database connection exhaustion, and provider throttling that pushes completion SLOs out of budget. Do not page on an isolated model timeout if retries and SLOs remain healthy.
10. Security and compliance
Treat retrieved documents and search results as untrusted input. Prompt injection can appear in ordinary web pages or internal documents; instructions found in evidence must never grant new tools, broaden access, or override system policy. Workers should have read-only capabilities and receive only the minimum tenant-scoped evidence needed for a task. A search result is evidence to assess, not code or instructions to execute.
Use Cognito or the organization’s identity provider for user authentication, then enforce tenant authorization in every API operation and retrieval query. Give API and worker tasks separate IAM roles. The API needs permission to write jobs and enqueue messages; the worker needs permission to consume jobs, read authorized data, call approved model endpoints, and write results. Avoid shared administrator credentials and use Secrets Manager for provider secrets that cannot use workload identity.
Protect data in transit and at rest with managed TLS and encryption controls. Establish retention periods for questions, results, events, checkpoints, and source content separately. Redact or hash tenant identifiers in broad operational logs, and define access controls for the traces that may contain operational metadata. Do not send sensitive customer data to an external search or model provider unless the organization’s data processing terms and classification policy permit it.
Define a source allowlist, block private and link-local network ranges in URL fetchers, validate redirects, and cap response sizes. These controls reduce SSRF and malicious-content risk but do not make arbitrary web fetching safe. Document the applicable privacy, contractual, and regulatory obligations before production; the use of managed cloud services is not, by itself, a compliance determination.
11. Cost
The following is an assumption-based monthly estimate, not a quoted AWS price. It assumes one primary region, moderate multi-AZ database sizing, 1,500 research jobs/day, bounded worker fan-out, cached repeated retrieval, and a modest search-provider allowance. It excludes taxes, enterprise support, and significant egress. Recalculate with the current regional price calculator and measured model-token usage before committing a budget.
| Cost area | Monthly planning range | Main cost driver |
|---|---|---|
| ECS Fargate API and workers | $500–$1,800 | Always-on API capacity and worker runtime |
| Aurora PostgreSQL, backups, and storage | $700–$2,000 | Multi-AZ capacity, I/O, retention, and checkpoint writes |
| OpenSearch and S3 | $500–$2,000 | Search capacity, index size, replicas, and document retention |
| Amazon Bedrock inference | $1,500–$6,000 | Model choice, input/output tokens, critic iterations, and retries |
| External search provider | $300–$1,500 | Search calls per job and provider plan |
| SQS, Redis, networking, and telemetry | $500–$1,500 | NAT/data transfer, cache size, logs, metrics, and traces |
| Planning total | $4,000–$14,800 | Highly sensitive to inference and search usage |
The most effective cost controls are usually application-level: cap worker tasks, constrain evidence bytes, cache stable retrieval results, select a suitable model per graph step, and stop revision loops when additional evidence is unlikely to improve the answer. Set a per-job budget and fail with a useful status when it is exhausted. Measure actual tokens and calls per completed job before rightsizing compute.
12. Failure modes, resilience, and disaster recovery
Assume every external dependency can fail independently. Queue delivery is at least once, so the worker must make processing idempotent: claim a job using a transactional status update, persist attempt numbers, and avoid duplicating externally visible results. A duplicate delivery should observe a completed or actively claimed job and exit safely. Use a transactional outbox or reconciliation process to close the gap between job creation and queue publication.
| Failure | Behavior and mitigation |
|---|---|
| Search provider timeout or quota exhaustion | Retry transient failures with jitter; continue with internal evidence when policy allows; report reduced coverage |
| Model throttling or outage | Apply bounded retries and concurrency limits; defer jobs or mark them retryable rather than looping |
| Worker crash | Allow SQS redelivery; recover graph state from checkpoints; use job-level idempotency |
| Poison message | Move to DLQ after bounded receives; alarm and provide a controlled replay procedure |
| Database unavailable | Stop claiming new jobs; keep API degradation explicit; restore from managed backups and verify checkpoint consistency |
| Search index unavailable | Return a degraded result only if the product contract permits it; do not silently imply a corpus search occurred |
| Critic loop or oversized output | Enforce iteration, token, evidence, and wall-clock limits; terminate with an honest partial status |
Use multi-AZ managed services for availability within a region, but do not confuse that with disaster recovery. Define an RPO and RTO that the product can support, enable point-in-time recovery for PostgreSQL, version important S3 objects, and document how to rebuild the search index from manifests and source objects. Test database restores and index rebuilds. A cross-region recovery design introduces replication, identity, routing, and data-residency complexity; add it only when an agreed business objective requires it.
13. Evolving to the next scale
Keep the graph, provider adapters, and API contracts modular while the deployment remains small. A worker service can scale horizontally on queue depth; the API can scale independently on request load. The next scaling step is usually not “one microservice per agent.” First determine which resource is constrained: database connections, search capacity, model quotas, worker memory, queue delay, or provider rate limits.
| Signal | Evolution |
|---|---|
| Queue delay rises while workers are saturated | Increase worker capacity within provider and database concurrency budgets; separate interactive and batch queues if their SLOs differ |
| Search traffic dominates latency or cost | Add measured retrieval caching, improve indexing and filters, or separate ingestion from query capacity |
| Checkpoints or event tables affect database operations | Tune retention and indexing; isolate checkpoint storage only if measured contention justifies the operational cost |
| Independent ownership emerges across teams | Split a service at a stable capability boundary, such as ingestion or search, with an explicit API and SLO |
| Regional recovery becomes a business requirement | Design and test a regional recovery path, including data replication and model/search availability |
An evaluation service, model gateway, or dedicated ingestion service can be useful later, but each adds deployment and operational overhead. Keep the critic as a graph node while its lifecycle and scaling needs are coupled to the research run. Extract it only for a concrete reason such as separate model governance or independently measured workload demand.
14. Phased implementation plan
Deliver this system in phases that make failure visible early. Do not begin with broad web access, multi-region replication, or a large autonomous agent framework. Start with a narrow corpus and an explicit quality contract; then add bounded concurrency and production controls.
| Phase | Deliverables | Exit criteria |
|---|---|---|
| 1. Evidence contract | Define source policy, citation format, question classes, and a small evaluation set | Product and engineering agree what counts as supported output |
| 2. Local graph | Implement planner, one worker, critic, and synthesizer with fakes | Unit tests prove task and iteration bounds; results retain source references |
| 3. AWS vertical slice | Deploy API, SQS, worker, Aurora, and staging model access | A job can be submitted, resumed safely, and polled to completion |
| 4. Retrieval and safety | Add tenant-scoped OpenSearch retrieval and approved search adapter | SSRF, injection, access-control, timeout, and payload-limit tests pass |
| 5. Operability | Add tracing, dashboards, SLOs, alarms, DLQ replay, and restore runbooks | On-call can diagnose a failed job without inspecting sensitive prompt content |
| 6. Controlled launch | Limit concurrency and traffic; review cost and quality weekly | SLOs and evaluation gates hold under representative load |
A successful launch is not the moment the graph produces a convincing answer. It is the point at which the team can explain why an answer was produced, identify the evidence behind it, stop an unsafe or over-budget run, and recover from partial failure without losing the user’s job. LangGraph provides the orchestration model; production quality comes from bounded tools, durable state, evidence contracts, and disciplined operations.
This article was generated with AI. #AIGenerated