1) Context and scale
This blueprint is for a small SaaS team building an order-intake and fulfillment API. Customers submit orders through a web application or API; the system validates and persists each order, then processes fulfillment asynchronously. Traffic is uneven: quiet overnight, with short spikes from promotions, partner integrations, and customer imports.
The primary design choice is whether to run the API and background work on containers or functions. Here, AWS Lambda is a good fit because the handlers are short-lived, requests arrive in bursts, and the team wants to spend its operational attention on product behavior rather than cluster capacity. This is not a claim that functions are universally cheaper or simpler. A continuously busy service, a long-running workload, or a workload requiring specialized runtime control may be a better container deployment.
The estimates below are planning assumptions, not measured production figures.
| Dimension | Assumption |
|---|---|
| Daily active users | 20,000 |
| API traffic | 300,000 requests/day; about 3.5 requests/s average |
| Peak API traffic | 100 requests/s for short bursts |
| Order submissions | 25,000/day, with a temporary 5× peak |
| Data volume | 250 GB initially, including indexes and retained order history |
| Team | 5 engineers across 1 product team; shared platform support is limited |
| Monthly cloud budget | $3,000 target, excluding engineering labor |
| Availability target | 99.9% for order submission, with asynchronous fulfillment allowed to lag |
The functions in this design have bounded work: validate, read or write a small number of database rows, and return or enqueue. The fulfillment worker processes one message at a time per invocation, with a configured concurrency limit. There is no persistent in-process state to preserve between requests.
A useful decision rule is: choose functions when workload shape and team size make per-invocation scaling and managed execution more valuable than long-lived process control. Choose containers when the service is busy and steady enough that reserved capacity is efficient, or when it needs long-lived connections, custom operating-system behavior, or work that exceeds function execution limits.
2) Requirements
Functional requirements
| ID | Requirement |
|---|---|
| F1 | Authenticate customers and accept an order through a versioned API. |
| F2 | Validate the request and persist the order and its items atomically. |
| F3 | Return an order identifier without waiting for downstream fulfillment. |
| F4 | Process fulfillment asynchronously and safely handle duplicate delivery. |
| F5 | Expose order status and support a bounded customer query. |
| F6 | Record failed work for investigation and controlled redrive. |
Non-functional requirements
| Area | Requirement |
|---|---|
| Availability | Order submission target of 99.9% monthly; asynchronous processing may lag during recovery. |
| Latency | API p95 under 500 ms at the assumed peak, excluding client network time. |
| Durability | An accepted order is committed before the API returns success. |
| Scaling | Absorb short submission spikes without requiring pre-provisioned application capacity. |
| Consistency | Order creation and its outbox record commit in one database transaction. |
| Security | Authenticate at the edge; use least-privilege workload identities and encrypted data. |
| Operations | One team can deploy, observe, and recover the system without running a Kubernetes control plane. |
| Cost | Keep the base-case cloud estimate comfortably below the stated monthly budget. |
The API should not promise that fulfillment has completed when it only knows that the order has been accepted. That distinction is part of the contract, not an implementation detail: clients receive an order ID and query its status, while internal retries remain invisible unless they delay the status transition.
3) High-level design
The web application is served from S3 through CloudFront. Amazon Cognito issues customer tokens. API Gateway validates the token and routes requests to a small set of Lambda handlers. The order handler writes the order, line items, and an outbox row in one Aurora PostgreSQL transaction. A scheduled publisher Lambda transfers unpublished outbox records to SQS. A worker Lambda consumes the queue and updates fulfillment state.
flowchart LR
Browser["Customer browser"] --> CDN["CloudFront"]
CDN --> Web["S3 static web app"]
Browser --> Auth["Amazon Cognito"]
Browser --> API["API Gateway HTTP API"]
API --> OrderFn["Order API Lambda"]
OrderFn --> Proxy["RDS Proxy"]
Proxy --> DB["Aurora PostgreSQL"]
Scheduler["EventBridge Scheduler"] --> Publisher["Outbox publisher Lambda"]
Publisher --> Proxy
Publisher --> Queue["Amazon SQS"]
Queue --> Worker["Fulfillment worker Lambda"]
Worker --> Proxy
Queue --> DLQ["SQS dead-letter queue"]
API --> Logs["CloudWatch and X-Ray"]
OrderFn --> Logs
Publisher --> Logs
Worker --> LogsThe synchronous path commits an order; the queue-backed path performs fulfillment independently.
This is intentionally a small number of deployable units, not one Lambda per business operation. Keep related API handlers in a modular application package until independent scaling, ownership, or release cadence justifies splitting them. A single function can host several related routes; separate functions only where permissions, scaling, runtime, or failure isolation differ.
The API Lambda does not call the fulfillment worker directly. SQS provides buffering and retry semantics, while the outbox closes the gap between committing an order and publishing its event. RDS Proxy reduces connection churn from concurrent Lambda invocations; it does not make arbitrary database connection patterns safe. Keep transactions short and set concurrency with database capacity in mind.
4) Cloud service mapping
AWS is the selected cloud. The equivalents below describe comparable services, not perfectly interchangeable behavior or pricing.
| Component | AWS choice | Azure equivalent | Google Cloud equivalent |
|---|---|---|---|
| Static application delivery | S3 and CloudFront | Blob Storage and Azure Front Door | Cloud Storage and Cloud CDN |
| Customer identity | Amazon Cognito | Microsoft Entra External ID | Identity Platform |
| API entry point | API Gateway HTTP API | Azure API Management | API Gateway |
| Request handlers | AWS Lambda | Azure Functions | Cloud Run functions |
| Relational database | Aurora PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Database connection management | RDS Proxy | PgBouncer or managed connection pooling | Cloud SQL connection pooling options |
| Work queue and dead-letter handling | Amazon SQS | Azure Service Bus queues | Cloud Tasks or Pub/Sub, depending on semantics |
| Scheduled outbox polling | EventBridge Scheduler | Azure Functions timer trigger | Cloud Scheduler |
| Secrets and keys | Secrets Manager and KMS | Key Vault | Secret Manager and Cloud KMS |
| Logs, metrics, traces | CloudWatch and X-Ray | Azure Monitor and Application Insights | Cloud Monitoring and Cloud Trace |
| Edge filtering | AWS WAF | Azure WAF | Cloud Armor |
Use the queue service that matches the needed delivery and ordering semantics. This design needs durable at-least-once delivery and a dead-letter queue; it does not require global message ordering. If strict ordering becomes a business requirement, model its scope explicitly before switching to a FIFO or partitioned queue.
5) Data design
Aurora PostgreSQL is the system of record because orders, items, idempotency, and publication state benefit from relational constraints and transactions. The outbox row is committed with the order. The publisher may send a message more than once if it crashes after sending but before marking the row published, so the consumer must be idempotent.
erDiagram
ORDERS ||--|{ ORDER_ITEMS : contains
ORDERS ||--o{ OUTBOX_EVENTS : emits
ORDERS ||--o{ PROCESSED_MESSAGES : tracks
ORDERS {
uuid id PK
text customer_id
text status
timestamptz created_at
}
ORDER_ITEMS {
uuid id PK
uuid order_id FK
text sku
integer quantity
numeric unit_price
}
OUTBOX_EVENTS {
uuid id PK
uuid order_id FK
text event_type
timestamptz published_at
}
PROCESSED_MESSAGES {
text message_id PK
timestamptz processed_at
}The order transaction and outbox record establish durable intent; processed-message keys make delivery retries safe.
A compact initial schema can use ordinary PostgreSQL tables and indexes. Partitioning is not a day-one requirement at this volume. Add time-based partitioning only when retention, vacuum behavior, or measured query performance warrants the operational complexity.
CREATE TABLE orders (
id uuid PRIMARY KEY,
customer_id text NOT NULL,
status text NOT NULL CHECK (status IN ('accepted', 'processing', 'fulfilled', 'failed')),
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX orders_customer_created_idx
ON orders (customer_id, created_at DESC);
CREATE TABLE order_items (
id uuid PRIMARY KEY,
order_id uuid NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
sku text NOT NULL,
quantity integer NOT NULL CHECK (quantity > 0),
unit_price numeric(12, 2) NOT NULL CHECK (unit_price >= 0)
);
CREATE TABLE outbox_events (
id uuid PRIMARY KEY,
order_id uuid NOT NULL REFERENCES orders(id),
event_type text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz
);
CREATE INDEX outbox_unpublished_idx
ON outbox_events (created_at)
WHERE published_at IS NULL;
CREATE TABLE processed_messages (
message_id text PRIMARY KEY,
processed_at timestamptz NOT NULL DEFAULT now()
);
Do not use the outbox payload as an unbounded copy of the entire order. Include the event version, order ID, and the fields a consumer needs. Keep customer-facing status queries scoped by authenticated customer ID and indexed accordingly. Apply a documented retention policy to old orders, outbox records, and queue messages; backups are not a substitute for retention design.
6) Low-level design
The application has three logical responsibilities: the API accepts and validates commands, the outbox publisher moves committed events to the queue, and the worker applies fulfillment transitions. They can share one repository and domain package while deploying as separate Lambda entry points.
classDiagram
class OrderHandler {
+handle(request) response
-validate(command)
}
class OrderService {
+create(command) orderId
}
class OrderRepository {
+createWithOutbox(command) orderId
+markFulfilled(orderId)
}
class OutboxPublisher {
+publishBatch()
}
class FulfillmentWorker {
+handleBatch(records)
}
class MessageQueue {
+send(event)
}
OrderHandler --> OrderService
OrderService --> OrderRepository
OutboxPublisher --> OrderRepository
OutboxPublisher --> MessageQueue
FulfillmentWorker --> OrderRepositoryKeep business rules in application services and isolate cloud adapters at the edges.
Minimal function example
This handler illustrates the shape of a bounded API function. It does not replace authentication, request validation, or persistence; those concerns belong in the production handler and its API contract.
type ApiEvent = { body?: string | null };
export async function handler(event: ApiEvent) {
if (!event.body) {
return { statusCode: 400, body: JSON.stringify({ error: "body_required" }) };
}
let input: unknown;
try {
input = JSON.parse(event.body);
} catch {
return { statusCode: 400, body: JSON.stringify({ error: "invalid_json" }) };
}
// Replace this example response with the validated application command.
return {
statusCode: 202,
headers: { "content-type": "application/json" },
body: JSON.stringify({ accepted: true, input }),
};
}
Notice that the handler returns an explicit response for malformed input and does not wait for downstream work.
Realistic SQS worker example
This worker uses PostgreSQL transactions and the SQS partial batch response contract. The database uniqueness constraint makes duplicate delivery harmless: the message is recorded once, and the order transition is applied in the same transaction. The SQS event source must be configured to report partial batch failures.
import { Pool } from "pg";
import type { SQSEvent, SQSBatchResponse } from "aws-lambda";
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
max: 4,
idleTimeoutMillis: 10_000,
connectionTimeoutMillis: 3_000,
});
type FulfillmentEvent = {
eventId: string;
orderId: string;
eventType: "OrderAccepted";
version: 1;
};
async function processMessage(messageId: string, body: string): Promise<void> {
const event = JSON.parse(body) as FulfillmentEvent;
if (event.eventType !== "OrderAccepted" || event.version !== 1) {
throw new Error("unsupported_event");
}
const client = await pool.connect();
try {
await client.query("BEGIN");
const inserted = await client.query(
`INSERT INTO processed_messages (message_id)
VALUES ($1)
ON CONFLICT DO NOTHING
RETURNING message_id`,
[event.eventId],
);
if (inserted.rowCount === 1) {
await client.query(
`UPDATE orders
SET status = 'fulfilled'
WHERE id = $1 AND status IN ('accepted', 'processing')`,
[event.orderId],
);
}
await client.query("COMMIT");
} catch (error) {
await client.query("ROLLBACK");
throw error;
} finally {
client.release();
}
}
export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
const failures: SQSBatchResponse["batchItemFailures"] = [];
for (const record of event.Records) {
try {
await processMessage(record.messageId, record.body);
} catch (error) {
console.error(JSON.stringify({
level: "error",
event: "queue_message_failed",
messageId: record.messageId,
errorType: error instanceof Error ? error.name : "UnknownError",
}));
failures.push({ itemIdentifier: record.messageId });
}
}
return { batchItemFailures: failures };
}
Notice that retries are expected, failures are returned per record, and the database transaction is the idempotency boundary.
In a real fulfillment integration, do not mark an order fulfilled before an external side effect succeeds. Persist a state transition or work record, use the external provider’s idempotency key where available, and define how an operator reconciles ambiguous outcomes. Also size the PostgreSQL pool and Lambda reserved concurrency together: the maximum possible client connections are a product of concurrent environments and connections per environment.
7) Key flows
Order submission
The response means “durably accepted,” not “fulfillment finished.” The transaction ensures that an order cannot be accepted without a corresponding outbox event.
sequenceDiagram
participant C as "Customer client"
participant G as "API Gateway"
participant L as "Order API Lambda"
participant P as "RDS Proxy"
participant D as "Aurora PostgreSQL"
C->>G: "POST order with token"
G->>L: "Validated request"
L->>P: "Begin transaction"
P->>D: "Insert order, items, and outbox event"
D-->>L: "Commit"
L-->>C: "202 Accepted with order ID"The synchronous contract ends after the database confirms the commit.
Outbox publication and fulfillment
SQS delivery is at least once. The publisher and worker are designed for retries rather than assuming exactly-once execution. Configure the queue visibility timeout to exceed the worker’s expected processing duration, and use a redrive policy to move repeatedly failing messages to the DLQ.
sequenceDiagram
participant E as "EventBridge Scheduler"
participant P as "Outbox publisher Lambda"
participant D as "Aurora PostgreSQL"
participant Q as "Amazon SQS"
participant W as "Fulfillment worker Lambda"
E->>P: "Invoke publisher"
P->>D: "Claim unpublished rows"
D-->>P: "Outbox events"
P->>Q: "Send event"
P->>D: "Mark event published"
Q->>W: "Deliver event"
W->>D: "Record message and update order transactionally"
D-->>W: "Commit"If publication is repeated after a timeout, worker-side idempotency prevents a duplicate state transition.
8) Build and release
Keep infrastructure and application code versioned together where practical, but separate deployment permissions and production approvals from ordinary pull-request checks. A staging environment should use the same function configuration, queue behavior, and database migration process as production, scaled down where safe. Avoid using a developer’s personal credentials in CI; use GitHub Actions OIDC to assume a narrowly scoped AWS role.
The following CDK fragment shows the queue and worker wiring. Production code should also define the API, database, alarms, encryption, and IAM policies; those are omitted here to keep the example focused on the function-versus-container boundary.
import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as nodejs from "aws-cdk-lib/aws-lambda-nodejs";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as sources from "aws-cdk-lib/aws-lambda-event-sources";
export class OrderWorkerStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const deadLetterQueue = new sqs.Queue(this, "OrderDeadLetterQueue", {
encryption: sqs.QueueEncryption.SQS_MANAGED,
retentionPeriod: cdkDurationDays(14),
});
const queue = new sqs.Queue(this, "OrderQueue", {
encryption: sqs.QueueEncryption.SQS_MANAGED,
visibilityTimeout: cdkDurationSeconds(180),
deadLetterQueue: { queue: deadLetterQueue, maxReceiveCount: 5 },
});
const worker = new nodejs.NodejsFunction(this, "FulfillmentWorker", {
runtime: lambda.Runtime.NODEJS_22_X,
entry: "src/worker.ts",
handler: "handler",
timeout: cdkDurationSeconds(60),
memorySize: 512,
reservedConcurrentExecutions: 20,
environment: { QUEUE_URL: queue.queueUrl },
bundling: { minify: true, sourceMap: true },
});
worker.addEventSource(new sources.SqsEventSource(queue, {
batchSize: 5,
reportBatchItemFailures: true,
}));
queue.grantConsumeMessages(worker);
}
}
// Aliases keep the fragment short; import Duration from aws-cdk-lib in a full app.
function cdkDurationSeconds(value: number) {
return require("aws-cdk-lib").Duration.seconds(value);
}
function cdkDurationDays(value: number) {
return require("aws-cdk-lib").Duration.days(value);
}
Notice that concurrency and queue retry behavior are explicit; they are part of capacity and failure control.
A production CDK app should prefer direct Duration imports rather than the compact helper above, and should synthesize and test its stack as part of CI. Reserve worker concurrency based on the database’s connection budget, not on the maximum Lambda quota. Apply database migrations as a controlled release step, using backward-compatible expand-and-contract changes.
A compact GitHub Actions pipeline can gate staging and production promotion. Store the production role ARN as a protected repository or environment variable, and configure the role trust policy for the repository and branch or environment.
name: order-service
on:
pull_request:
push:
branches: [main]
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm run test:integration
- run: npm run cdk:synth
deploy-staging:
if: github.event_name == 'push'
needs: verify
runs-on: ubuntu-latest
environment: staging
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.AWS_STAGING_ROLE_ARN }}
aws-region: us-east-1
- run: npx cdk deploy --all --require-approval never
deploy-production:
needs: deploy-staging
runs-on: ubuntu-latest
environment: production
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.AWS_PRODUCTION_ROLE_ARN }}
aws-region: us-east-1
- run: npx cdk deploy --all --require-approval never
Test strategy
- Unit: Validate request schemas, business rules, status transitions, and idempotency decisions without cloud dependencies.
- Integration: Run repository and migration tests against PostgreSQL, and exercise queue handling against a local or isolated AWS test environment.
- Contract: Verify API request and response schemas, event version compatibility, and partial batch response behavior. Treat queue messages as versioned contracts.
- Load: Test realistic bursts and sustained traffic; record API latency, Lambda concurrency, database connections, proxy behavior, and queue age. Confirm that concurrency caps protect the database.
- Release: Deploy backward-compatible schema changes before code that requires them. Verify staging smoke tests and a rollback path before promoting.
9) Observability
Instrument the user-visible operation and the asynchronous continuation separately. A healthy API does not imply a healthy fulfillment pipeline. Carry a trace or correlation identifier from the API request into the outbox event and SQS message attributes, then use it to connect the order’s lifecycle across invocations.
Structured logging
Log stable fields rather than free-form strings. Do not log access tokens, full payment details, or unnecessary customer data. Use a request ID for one invocation and a correlation ID for the end-to-end order operation.
{
"timestamp": "2026-11-13T12:00:00Z",
"level": "INFO",
"service": "order-api",
"environment": "production",
"event": "order_accepted",
"requestId": "lambda-request-id",
"correlationId": "order-correlation-id",
"orderId": "7f6b6b0b-88d1-4f37-9aa8-5cc148c32890",
"durationMs": 84,
"outcome": "success"
}
RED and USE signals
| Signal set | What to measure |
|---|---|
| RED — API | Request rate by route, error rate by status class, and latency percentiles. |
| RED — worker | Message processing rate, failures and retries, and processing duration. |
| USE — Lambda | Concurrent executions, throttles, duration and memory utilization. |
| USE — database | CPU, connections, freeable memory, storage, and read/write pressure. |
| USE — queue | Visible and in-flight messages, oldest message age, and DLQ depth. |
| USE — proxy | Client connections, database connections, and connection borrow latency. |
For OpenTelemetry, attach the AWS Distro for OpenTelemetry Lambda layer appropriate to the selected region and runtime, configure the function to emit traces, and propagate context in message attributes. The following application span can be used inside a handler; the layer supplies the SDK and exporter configuration.
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("order-service");
export async function createOrder(command: unknown) {
return tracer.startActiveSpan("orders.create", async (span) => {
try {
span.setAttribute("app.operation", "create_order");
// Call validation and the transactional repository here.
const orderId = await persistOrder(command);
span.setAttribute("order.id", orderId);
return orderId;
} catch (error) {
span.recordException(error as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
});
}
async function persistOrder(command: unknown): Promise<string> {
// Application repository implementation omitted from this tracing example.
return "generated-order-id";
}
Notice that tracing adds operation context, while customer identifiers and other sensitive values should not be added as span attributes.
SLI and SLOs
| SLI | SLO | Measurement window |
|---|---|---|
| Successful authenticated order submissions | 99.9% of valid requests | Rolling 30 days |
| API latency | 95% under 500 ms | Rolling 30 days |
| Accepted orders with durable database commit | 99.99% of successful responses | Rolling 30 days |
| Queue processing delay | 99% processed within 5 minutes | Rolling 30 days |
| DLQ recovery | Alerted and triaged within 15 minutes during staffed hours | Operational target |
Build dashboards for API RED signals, worker and queue health, database and proxy USE signals, and deployment markers. Page on sustained order API availability or latency SLO burn, database unavailability or connection exhaustion, a rapidly increasing oldest-message age that threatens the five-minute target, and any DLQ growth that indicates persistent failures. Route isolated transient errors and low-volume DLQ events to a ticket or team channel unless they threaten customer impact.
10) Security and compliance
The security boundary starts at API Gateway, but authorization still belongs in the application. Validate token issuer, audience, and claims; derive the customer identity from the verified token rather than trusting a client-supplied customer ID. Apply request size limits, schema validation, throttling, and WAF rules appropriate to the public API.
Use a dedicated IAM role per function. The API handler should be able to access only the required database secret and relevant encryption keys; the worker should consume only its queue and access only the database operations it needs. Avoid broad wildcard permissions and shared execution roles. Use short-lived CI credentials through OIDC, and keep production deployment approval separate from pull-request verification.
Place database-connected functions in private subnets and restrict database ingress to the required security group. Avoid adding a NAT gateway unless the function needs outbound internet access; access to AWS services can use appropriate endpoints where available. Encrypt database storage, backups, queues, and secrets. Rotate database credentials and avoid embedding secrets in environment variables or build artifacts.
For compliance, define a data classification, retention schedule, backup retention, and access-review process before production. Enable audit trails for administrative changes and database access where required by the organization. The blueprint does not itself establish compliance: the applicable controls depend on jurisdiction, customer commitments, and the data being processed.
11) Cost
The following is an assumption-based monthly estimate, not a quote. It assumes a single AWS region, the traffic and data volumes in section 1, no NAT gateway, moderate logging, and a multi-AZ Aurora PostgreSQL deployment with a writer and a reader. Actual cost depends heavily on database capacity, request duration, data transfer, retention, and regional prices. Validate the design with the AWS pricing calculator before committing a budget.
| Cost area | Planning estimate per month |
|---|---|
| API Gateway and Lambda invocations/compute | $40–$150 |
| Aurora PostgreSQL compute, multi-AZ reader, and storage | $450–$850 |
| RDS Proxy | $40–$100 |
| SQS, EventBridge Scheduler, and DLQ | $5–$30 |
| S3, CloudFront, and data transfer | $30–$150 |
| Cognito, Secrets Manager, and KMS | $20–$80 |
| CloudWatch logs, metrics, alarms, and X-Ray | $60–$200 |
| WAF and backup storage | $40–$150 |
| Estimated total | $685–$1,710 |
Database compute and observability are the largest uncertainties, not Lambda request charges alone. Set log retention explicitly, sample traces where appropriate, and monitor database utilization before tuning capacity. A NAT gateway or substantial outbound transfer can change the estimate materially; this design avoids assuming either is free.
12) Failure modes, resilience and disaster recovery
| Failure mode | Behavior and mitigation |
|---|---|
| Lambda throttling | API requests may fail or queue consumption may slow. Set reserved concurrency deliberately, alarm on throttles, and return retryable status codes only where client retries are safe. |
| Database connection pressure | RDS Proxy and small per-environment pools reduce connection churn. Cap function concurrency and alarm on connection use and borrow latency. |
| Database failover | Aurora can fail over within the cluster, but in-flight transactions may fail. Use bounded retries with jitter for safe operations; never report success before commit. |
| Duplicate queue delivery | SQS is at-least-once. Enforce idempotency in the database transaction and make external side effects idempotent where supported. |
| Poison message | Repeated failure moves the message to the DLQ. Alert on DLQ depth, preserve the message for diagnosis, and redrive only after the cause is understood. |
| Publisher crash after send | The event may be sent again because the outbox row was not marked published. The consumer’s idempotency key handles replay. |
| Queue backlog | SQS absorbs bursts, but delay increases. Alarm on age, limit concurrency to protect PostgreSQL, and communicate delayed status to clients if needed. |
| Region outage | This blueprint is single-region. Restore from backups into a second region and validate DNS, secrets, queue provisioning, and deployment procedures through a documented recovery exercise. |
Set queue retention and visibility timeout intentionally. A timeout that is too short can cause concurrent processing of the same message; a timeout that is too long delays recovery. For the database, define backup retention and point-in-time recovery objectives, and test restoration rather than treating successful backup jobs as proof of recoverability.
For medium scale, a tested single-region recovery plan is usually a better first investment than active-active multi-region operation. Document an RTO and RPO with product owners, keep infrastructure reproducible, and rehearse restoring the database and rebuilding queues. Queue messages that are not retained across a regional event must be recoverable from the outbox or an agreed replay mechanism.
13) Evolving to the next scale
Do not migrate to containers because a graph looks busy or because “serverless” sounds unsuitable at scale. Migrate a workload when measurements show that its execution model is the limiting factor or a concrete operational requirement is unmet.
| Signal | Likely next change |
|---|---|
| Sustained high utilization makes per-invocation compute inefficient | Compare a continuously running ECS service with Lambda using measured workload and regional prices. |
| Work routinely approaches function timeout or needs a long-lived process | Move that worker to ECS/Fargate or another managed container platform; keep the queue contract. |
| Database connections or write capacity are the bottleneck | Tune queries and indexes, review connection budgets, then consider database scaling or a different data partitioning strategy. |
| Queue delay grows despite safe concurrency increases | Increase worker efficiency, split independent workloads, or provision a dedicated worker service if continuous consumption is appropriate. |
| Teams need independent ownership and release boundaries | Split deployables along stable domain and operational boundaries, not simply by handler count. |
| Cross-region recovery requirements become stricter | Design and test regional data replication, queue replay, identity, and failover behavior as one system. |
Functions still make sense for bursty edges even if one heavy worker moves to containers. A mixed model is often the practical endpoint: API and scheduled tasks remain functions, while a sustained CPU-heavy consumer runs as a container. Keep the event contract and idempotency design stable so this change is an execution-model swap, not a domain redesign.
14) Phased implementation plan
- Confirm workload and service objectives. Measure request shape, payload sizes, burst duration, expected database reads and writes, data retention, and the actual acceptable delay to fulfillment. Agree on the order-submission SLO and recovery objectives.
- Build the smallest vertical slice. Create the API, authentication, PostgreSQL schema, order transaction, and status query. Verify the API never returns success before commit.
- Add the asynchronous boundary. Introduce the outbox, scheduled publisher, SQS queue, worker, idempotency table, and DLQ. Test duplicate delivery and publisher crash scenarios deliberately.
- Automate environments and releases. Provision infrastructure through CDK, establish staging parity, run migrations safely, and configure OIDC-based CI/CD with production approval.
- Establish operational evidence. Add structured logs, traces, RED/USE dashboards, SLO alerts, queue-age alarms, and a database restore exercise before general availability.
- Load-test and tune. Test the expected burst, observe connection counts and queue delay, then adjust Lambda concurrency, batch size, database capacity, and timeouts based on measurements.
- Launch with a recovery playbook. Document DLQ investigation and redrive, database failover behavior, deployment rollback, and regional restore. Assign ownership for each alert.
- Review after real traffic. Compare actual invocation duration, database utilization, queue age, and cost with assumptions. Keep functions while they fit the workload; move only the component whose measured needs justify a container.
The architectural win is not “no servers.” It is a bounded operational surface for a small team: managed request execution, an explicit queue boundary, a transactional record of work, and clear conditions for changing the execution model.
This article was generated with AI. #AIGenerated