1. Context and scale
A product analytics pipeline is a data product, not a collection of track() calls. Product teams need consistent event names and properties; analysts need stable definitions and replayable data; engineering needs a reliable path from application behavior to a warehouse without letting analytics degrade the user-facing service.
This blueprint uses AWS for the application and data platform, RudderStack Cloud for event collection and routing, and Amazon Redshift Serverless as the warehouse. It suits a SaaS product with browser and server events, a small data team, and a moderate event rate. The architecture favors explicit contracts and replay over a large streaming platform.
The application sends browser events directly to RudderStack. For important server-side events, the application writes an outbox row in the same transaction as its business change. A relay publishes those events to SQS, and a publisher forwards them to RudderStack. RudderStack routes accepted events to an S3 raw archive. A separate loader consumes S3 notifications through SQS and loads data into Redshift for modeling.
| Measure | Assumption |
|---|---|
| Daily active users | 50,000 |
| Event volume | 2 million events per day |
| Sustained ingest rate | 25 events/s |
| Short peak ingest rate | 250 events/s |
| Compressed raw data | 20–40 GB/month |
| Application team | 5 engineers |
| Analytics ownership | 1 analytics engineer, part-time product analyst |
| Monthly platform budget | $2,000–$6,000, excluding labor |
These are sizing assumptions, not product limits or measured costs. Validate event size, burstiness, RudderStack plan limits, and AWS pricing against actual usage before committing to a budget.
The key architectural decision is to keep the product’s write path independent of analytics availability. A browser event can be lost when a user closes a tab; a business-critical event should not be. The outbox and queue make that distinction explicit.
2. Requirements
The tracking plan is the interface between product behavior and downstream analysis. It should live in version control, have an owner, and be reviewed like an API. It specifies event names, required properties, allowed values, identity rules, and data classification.
| Functional requirement | Design response |
|---|---|
| Capture browser and server behavior | Use RudderStack’s browser SDK for client events and a server-side HTTP source for backend events. |
| Define consistent event semantics | Store a versioned tracking plan in the application repository; validate it in CI and at event boundaries. |
| Preserve important events through outages | Write business events to a transactional outbox, relay to SQS, then publish to RudderStack. |
| Keep a replayable source of truth | Route accepted events to an append-only S3 raw archive before warehouse modeling. |
| Support analysis and dashboards | Load curated event data into Redshift and build analytics models there. |
| Correct historical logic | Version transformations and models; rebuild derived tables from raw data when definitions change. |
| Respect consent and deletion requests | Check consent before client tracking, propagate deletion requests, and define retention for raw and modeled data. |
| Non-functional requirement | Target or constraint |
|---|---|
| Availability | The product’s business transaction must succeed even if the analytics pipeline is unavailable. |
| Data freshness | Common product events should be queryable within 60 minutes under normal load. |
| Durability | Queue-backed server events remain recoverable during short downstream outages; raw files are retained for replay. |
| Data quality | Required properties, event names, and schema versions are checked before modeled data is published. |
| Privacy | Do not send direct identifiers or sensitive free text in analytics properties by default. |
| Operability | Each stage exposes queue depth, delivery failures, freshness, and event-volume metrics. |
| Change safety | Deploy tracking-plan changes, infrastructure, and warehouse models through staging and CI/CD. |
At this scale, do not promise exactly-once delivery across the browser, HTTP, queue, and warehouse. Retries can create duplicates. Give events stable IDs, retain those IDs through the pipeline, and make warehouse models idempotent.
3. High-level design
The application uses a multi-AZ Amazon RDS for PostgreSQL database. An outbox row is committed with the business change, then a relay transfers it to Amazon SQS. ElastiCache for Redis serves application caching needs; it is not an event buffer and must not be treated as one.
RudderStack Cloud is the collection and routing layer. Browser SDKs send client events to a RudderStack source. The server publisher sends queued events to a separate server source. RudderStack routes accepted events to S3. S3 notifications enqueue load work; a small loader batches files and loads them into Redshift. The raw bucket is the recovery point, while Redshift contains query-ready data.
flowchart LR
Web["Web app"]
Api["Application API on ECS Fargate"]
Db["Amazon RDS PostgreSQL multi-AZ"]
Cache["Amazon ElastiCache Redis"]
Outbox["Transactional outbox relay"]
Queue["Amazon SQS server event queue"]
Publisher["Publisher on ECS Fargate"]
Rudder["RudderStack Cloud"]
Bucket["Amazon S3 raw event bucket"]
LoadQueue["Amazon SQS warehouse load queue"]
Loader["Warehouse loader on ECS Fargate"]
Warehouse["Amazon Redshift Serverless"]
Models["dbt models and BI"]
Ops["Amazon CloudWatch and ADOT"]
Web --> Rudder
Api --> Db
Api --> Cache
Db --> Outbox
Outbox --> Queue
Queue --> Publisher
Publisher --> Rudder
Rudder --> Bucket
Bucket --> LoadQueue
LoadQueue --> Loader
Loader --> Warehouse
Warehouse --> Models
Publisher --> Ops
Loader --> OpsThe design separates application transactions, event delivery, raw retention, and warehouse modeling.
Deploy the API, relay, publisher, and loader as a small number of ECS Fargate services or scheduled tasks rather than creating a service per event type. Use separate SQS queues for server-event delivery and warehouse loading so one consumer’s backlog does not block the other. Configure S3 notifications with a queue policy that permits the bucket to publish only to the intended queue.
4. Cloud service mapping
RudderStack is a managed, cloud-neutral service in this design; it does not have a direct equivalent in each cloud provider. The equivalents below map the surrounding responsibilities, not identical products or integrations.
| Component | Chosen service | AWS equivalent | Azure equivalent | GCP equivalent |
|---|---|---|---|---|
| Event collection and routing | RudderStack Cloud | Third-party managed service | Third-party managed service | Third-party managed service |
| Application compute | Amazon ECS on Fargate | ECS on Fargate | Azure Container Apps | Cloud Run |
| Transactional database | Amazon RDS for PostgreSQL, Multi-AZ | RDS for PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Application cache | Amazon ElastiCache for Redis | ElastiCache | Azure Managed Redis | Memorystore |
| Durable work queue | Amazon SQS | SQS | Azure Service Bus queues | Cloud Tasks or Pub/Sub, depending on delivery pattern |
| Raw event archive | Amazon S3 | S3 | Azure Blob Storage | Cloud Storage |
| Warehouse | Amazon Redshift Serverless | Redshift | Microsoft Fabric Warehouse or Azure Synapse Analytics | BigQuery |
| Scheduled model and load jobs | EventBridge Scheduler and ECS tasks | EventBridge Scheduler | Azure Functions or Container Apps jobs | Cloud Scheduler and Cloud Run jobs |
| Logs and metrics | Amazon CloudWatch | CloudWatch | Azure Monitor | Cloud Monitoring |
| Trace collection | AWS Distro for OpenTelemetry | ADOT with AWS observability backends | Azure Monitor OpenTelemetry | Google Cloud Observability |
Choose one warehouse and one raw archive for this scale. Adding a second warehouse for speculative portability creates duplicate governance, modeling, and cost work without making the tracking plan more portable.
5. Data design
The tracking plan is a contract, not merely a naming guide. Each event definition should specify:
- A stable, lowercase event name such as
checkout_completed. - Required and optional properties, types, and enumerated values.
- Whether the event is browser-originated, server-originated, or both.
- Identity behavior:
userId,anonymousId, and when a user is considered identified. - Data classification and consent requirements.
- An owner, change history, and compatibility policy.
Use a common event envelope: messageId, event name, user or anonymous identity, event timestamp, received timestamp, source, schema version, and properties. Treat client timestamps as untrusted input; retain both client event time and server receipt time. Never put emails, payment data, authentication tokens, or arbitrary form contents into event properties. A pseudonymous user ID is still linkable data and requires appropriate access and retention controls.
The warehouse model separates event facts from small dimensions. Keep the raw event envelope in S3 so that you can rebuild curated tables when the contract or transformation logic changes. Partition raw files by ingestion date, not solely by client event date; late events and clock skew otherwise make operational loading harder. Use message_id as the deduplication key in warehouse transformations.
erDiagram
DIM_USER ||--o{ FACT_PRODUCT_EVENT : identifies
DIM_PRODUCT ||--o{ FACT_PRODUCT_EVENT : describes
DIM_EVENT ||--o{ FACT_PRODUCT_EVENT : classifies
DIM_USER {
varchar user_key PK
varchar anonymous_key
timestamp first_seen_at
timestamp last_seen_at
}
DIM_PRODUCT {
varchar product_key PK
varchar product_id
varchar product_name
}
DIM_EVENT {
varchar event_key PK
varchar event_name
int schema_version
}
FACT_PRODUCT_EVENT {
varchar message_id PK
varchar user_key FK
varchar product_key FK
varchar event_key FK
timestamp occurred_at
timestamp received_at
date received_date
varchar source
varchar properties_json
}The event fact keeps event identity and both event-time and ingestion-time semantics.
A practical first warehouse layer can use one append-oriented staging table plus curated facts. Do not prematurely normalize every property into a column: promote properties into typed columns when they become part of a governed metric or commonly used analysis.
CREATE TABLE raw_product_events (
message_id VARCHAR(128) NOT NULL,
event_name VARCHAR(128) NOT NULL,
user_id VARCHAR(128),
anonymous_id VARCHAR(128),
occurred_at TIMESTAMP NOT NULL,
received_at TIMESTAMP NOT NULL,
schema_version SMALLINT NOT NULL,
source VARCHAR(32) NOT NULL,
properties_json VARCHAR(MAX)
)
DISTSTYLE AUTO
SORTKEY (received_at, event_name);
CREATE TABLE dim_event (
event_key INTEGER IDENTITY(1,1),
event_name VARCHAR(128) NOT NULL,
schema_version SMALLINT NOT NULL,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
PRIMARY KEY (event_key)
)
DISTSTYLE AUTO
SORTKEY (event_name);
CREATE TABLE fact_product_event (
message_id VARCHAR(128) NOT NULL,
event_key INTEGER NOT NULL,
user_key VARCHAR(128),
occurred_at TIMESTAMP NOT NULL,
received_at TIMESTAMP NOT NULL,
received_date DATE NOT NULL,
source VARCHAR(32) NOT NULL,
properties_json VARCHAR(MAX)
)
DISTSTYLE AUTO
SORTKEY (received_date, event_key, occurred_at);
Redshift does not enforce primary-key uniqueness as an application database would. Enforce deduplication in the loading or transformation process, for example by selecting one record per message_id with a deterministic preference for the latest received_at. Keep a data-quality test for duplicate IDs and unexpected schema versions. For larger history, use date-based incremental models and a bounded lookback for late arrivals, with a separate backfill path.
6. Low-level design
Keep event construction behind a small application API. Product code should not invent property names or call the transport directly. A typed event catalog catches accidental contract drift in review; runtime validation catches bad inputs that static typing cannot prevent.
classDiagram
class TrackingPlan {
+validate(event)
+version
}
class EventFactory {
+checkoutCompleted(order)
+productViewed(product)
}
class EventPublisher {
+publish(event)
}
class OutboxRelay {
+claimBatch()
+enqueue(event)
}
class WarehouseLoader {
+load(objects)
+recordCheckpoint()
}
TrackingPlan --> EventFactory
EventFactory --> EventPublisher
EventPublisher --> OutboxRelay
WarehouseLoader --> TrackingPlanApplication code creates contract-valid events; transport and warehouse concerns remain separate.
A minimal Node.js example sends a server-side event to a RudderStack HTTP source. Set RUDDER_DATA_PLANE_URL to the data-plane URL shown for the source and RUDDER_WRITE_KEY to that source’s write key. Run with Node.js 20 or later.
const baseUrl = process.env.RUDDER_DATA_PLANE_URL;
const writeKey = process.env.RUDDER_WRITE_KEY;
if (!baseUrl || !writeKey) {
throw new Error("Set RUDDER_DATA_PLANE_URL and RUDDER_WRITE_KEY");
}
const event = {
messageId: crypto.randomUUID(),
userId: "usr_8f31",
event: "checkout_completed",
timestamp: new Date().toISOString(),
properties: {
order_id: "ord_742",
currency: "USD",
item_count: 2
},
context: {
app: { name: "store-api", version: "1.12.0" }
}
};
const authorization = Buffer.from(`${writeKey}:`).toString("base64");
const response = await fetch(`${baseUrl.replace(/\/$/, "")}/v1/track`, {
method: "POST",
headers: {
authorization: `Basic ${authorization}`,
"content-type": "application/json"
},
body: JSON.stringify(event)
});
if (!response.ok) {
throw new Error(`RudderStack returned HTTP ${response.status}`);
}
Notice that the example assigns a stable message ID before sending; retries should reuse the same event identity.
The minimal example is suitable for local verification, not a business-critical write path: an HTTP failure throws, and the caller must decide what to do. For production events, create the event in the business transaction’s outbox, send it asynchronously, and mark the outbox record complete only after the queue accepts it. The publisher should retain the same messageId across retries.
The following publisher outline shows the transport boundary. It receives an event already persisted to the queue; it should acknowledge the queue message only after RudderStack accepts it. Configure visibility timeouts and retry limits to match the expected request duration, and route exhausted messages to a dead-letter queue. The code deliberately does not interpret every non-2xx response as a permanent failure: operational policy should distinguish retryable transport and service errors from malformed events.
type ProductEvent = {
messageId: string;
userId?: string;
anonymousId?: string;
event: string;
timestamp: string;
properties: Record<string, string | number | boolean | null>;
context?: Record<string, unknown>;
};
export async function sendToRudderStack(
event: ProductEvent,
config: { baseUrl: string; writeKey: string }
): Promise<void> {
const url = `${config.baseUrl.replace(/\/$/, "")}/v1/track`;
const authorization = Buffer
.from(`${config.writeKey}:`)
.toString("base64");
const response = await fetch(url, {
method: "POST",
headers: {
authorization: `Basic ${authorization}`,
"content-type": "application/json"
},
body: JSON.stringify(event),
signal: AbortSignal.timeout(5_000)
});
if (!response.ok) {
throw new Error(`event_delivery_failed status=${response.status}`);
}
}
Notice that transport errors become retryable failures for the queue consumer, while event construction and identity stay upstream.
Do not log the authorization header or full event body. If an event is rejected, log the message ID, event name, source, and error class, then retain enough protected context to diagnose it. Avoid making the RudderStack write key available to browser code beyond the intended public-source configuration; use separate sources and credentials for browser and server traffic.
7. Key flows
The business event flow uses an outbox so that a successful database transaction does not depend on SQS or RudderStack being reachable.
sequenceDiagram
participant Client
participant Api
participant Db
participant Relay
participant Queue
participant Publisher
participant Rudder
Client->>Api: Submit checkout
Api->>Db: Commit order and outbox event
Db-->>Api: Transaction committed
Api-->>Client: Checkout confirmed
Relay->>Db: Claim unsent outbox rows
Relay->>Queue: Send event with messageId
Queue-->>Publisher: Deliver queued message
Publisher->>Rudder: POST track event
Rudder-->>Publisher: Accept or reject
Publisher->>Queue: Acknowledge accepted messageThe customer receives a business response after the database commit, not after the analytics vendor responds.
The warehouse flow starts from S3 rather than from the publisher. This separation allows you to rebuild models from the archived events without replaying business transactions. S3 notifications can be duplicated or delivered out of order, so the loader should treat object keys as idempotent work items and record completed loads.
sequenceDiagram
participant Rudder
participant S3
participant Queue as Load queue
participant Loader
participant Redshift
participant Model as dbt job
Rudder->>S3: Write raw event files
S3->>Queue: Notify object arrival
Queue-->>Loader: Deliver object notification
Loader->>S3: Read eligible files
Loader->>Redshift: Load and record file checkpoint
Loader->>Queue: Acknowledge notification
Model->>Redshift: Build incremental models
Redshift-->>Model: Publish curated tablesThe S3 archive and loader checkpoint make warehouse recovery independent of event collection.
The loader should batch small files where practical. A high count of tiny objects creates more operational work and inefficient warehouse loads. Configure RudderStack’s destination batching to suit the expected volume, and monitor file size and arrival cadence before tuning aggressively.
8. Build and release
Treat the tracking plan, application code, infrastructure, and warehouse models as one governed change surface. Keep the event catalog and schema validation in the application repository; keep dbt models and tests in the analytics repository if ownership differs, but make their release order explicit.
This Terraform excerpt creates a queue and raw bucket with baseline encryption and versioning. A production module also needs least-privilege IAM policies, lifecycle rules, queue policies, alarms, and environment-specific naming. Use separate AWS accounts or isolated environments for staging and production; never point staging RudderStack sources at production data.
resource "aws_sqs_queue" "server_events_dlq" {
name = "product-server-events-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sqs_queue" "server_events" {
name = "product-server-events"
visibility_timeout_seconds = 60
message_retention_seconds = 345600
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.server_events_dlq.arn
maxReceiveCount = 5
})
}
resource "aws_s3_bucket" "raw_events" {
bucket = var.raw_events_bucket_name
}
resource "aws_s3_bucket_versioning" "raw_events" {
bucket = aws_s3_bucket.raw_events.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "raw_events" {
bucket = aws_s3_bucket.raw_events.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
Notice that the queue’s dead-letter path and the S3 archive are explicit; they are not substitutes for monitoring or access controls.
A GitHub Actions workflow should run on pull requests and deploy only after tests pass. Use GitHub’s AWS OIDC integration rather than long-lived AWS credentials in repository secrets. The example leaves account-specific role configuration to the environment.
name: analytics-pipeline
on:
pull_request:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: npm
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm run test:contract
- run: terraform fmt -check -recursive
- run: terraform validate
deploy-staging:
if: github.ref == 'refs/heads/main'
needs: test
runs-on: ubuntu-latest
environment: staging
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_STAGING_ROLE_ARN }}
aws-region: us-east-1
- run: ./scripts/deploy-staging.sh
- run: npm run test:smoke
Notice that staging promotion is gated by tests and uses a short-lived role assumption rather than stored AWS keys.
Test at four levels:
- Unit: validate event factories, required properties, consent checks, identity rules, and serialization.
- Integration: exercise the outbox, queue, publisher, and loader against test dependencies or isolated AWS resources. Verify retry and dead-letter behavior.
- Contract: compare emitted events against the tracking plan; test representative RudderStack payloads and warehouse schema assumptions.
- Load: generate expected sustained and burst traffic, then measure queue age, publisher throughput, S3 file cadence, and warehouse freshness. Define the load profile from observed traffic, not a generic benchmark.
Do not use production user data in contract or load tests. Test migration and replay procedures in staging before changing event semantics or warehouse keys.
9. Observability
Instrument the publisher, loader, API, and outbox relay. Propagate a trace context across application calls and queue messages where supported by the message metadata. For browser events, correlation is best-effort: browser sessions can be short-lived and may not provide a trace that links cleanly to server work.
Logs should be structured and useful without copying sensitive event payloads:
{
"timestamp": "2026-11-18T14:03:22.481Z",
"level": "WARN",
"service": "rudder-publisher",
"environment": "production",
"message_id": "evt_4f93",
"event_name": "checkout_completed",
"attempt": 3,
"destination": "rudderstack",
"outcome": "retryable_failure",
"http_status": 503,
"trace_id": "a81b7e22f91c0c45",
"duration_ms": 218
}
The log identifies the event and outcome without including user identity, write keys, or arbitrary properties.
Use RED metrics for request-serving services and USE metrics for resources:
| Signal family | Key measurements |
|---|---|
| RED: rate | Events received and published per second; API requests per second; warehouse objects processed per minute. |
| RED: errors | Publish failures by class, rejected events, loader failures, and outbox serialization errors. |
| RED: duration | Publisher request latency, API latency, load-job duration, and end-to-end event freshness. |
| USE: utilization | ECS CPU and memory; database connections and storage; Redshift capacity and query time. |
| USE: saturation | SQS visible and in-flight messages, oldest-message age, outbox backlog, and Redshift queueing. |
Use OpenTelemetry spans around event publishing and queue processing. The AWS Distro for OpenTelemetry collector can receive OTLP telemetry from the service and export it to configured AWS observability backends. This Node.js example records a span around the outbound request; production services should also enable supported HTTP and AWS SDK instrumentation.
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { trace } from "@opentelemetry/api";
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
})
});
sdk.start();
const tracer = trace.getTracer("product-analytics-publisher");
export async function publishWithTrace(
event: { messageId: string; event: string },
send: () => Promise<void>
): Promise<void> {
await tracer.startActiveSpan("rudderstack.publish", async (span) => {
span.setAttribute("event.name", event.event);
span.setAttribute("event.message_id", event.messageId);
try {
await send();
span.setAttribute("delivery.outcome", "accepted");
} catch (error) {
span.recordException(error as Error);
span.setAttribute("delivery.outcome", "failed");
throw error;
} finally {
span.end();
}
});
}
Notice that the trace records event identity and outcome, but does not attach the full payload.
Set service objectives on outcomes users and analysts care about. The values below are initial targets for this assumed scale, not guarantees provided by AWS or RudderStack.
| SLI | Initial SLO |
|---|---|
| API success for business requests, excluding client errors | 99.9% over 30 days |
| Server events accepted by RudderStack within 5 minutes of outbox commit | 99% over 30 days |
| Raw event availability in S3 after vendor acceptance | 99% within 15 minutes |
| Warehouse freshness for routine events | 95% within 60 minutes |
| Required-field validity in modeled events | 99.5% per day |
Create dashboards for end-to-end freshness, event volume by event name and source, queue depth and oldest age, delivery error class, raw-file counts and sizes, warehouse load history, and contract-test failures. Page on sustained growth in oldest SQS message age, a DLQ increase, a prolonged drop in server-event delivery, a broken S3-to-warehouse freshness objective, or an API error-rate/SLO burn. Send low-volume schema drift and unexpected event-name alerts to a team channel rather than paging on every anomaly.
10. Security and compliance
Analytics data is easy to over-collect because properties are flexible. Establish an allowlist-oriented policy: event schemas define permitted properties, and application code rejects or removes unknown fields where feasible. Review the tracking plan for every new property that can identify a person, reveal health or financial information, or contain user-generated text.
- Use separate RudderStack sources and credentials for browser and server events. Rotate server credentials and store them in AWS Secrets Manager.
- Give ECS task roles only the permissions each service needs. The publisher should read its queue and secret; the loader should read designated S3 prefixes and write to Redshift.
- Block public access to S3, encrypt data at rest and in transit, and restrict warehouse access through IAM and database roles.
- Apply consent before sending non-essential browser analytics. Do not assume that server-side collection bypasses consent requirements.
- Define deletion workflows across RudderStack, S3, Redshift, backups, and downstream extracts. Retention and deletion behavior should be documented and tested.
- Restrict raw-data access to the smallest group that needs it. Analysts should generally query curated models rather than unrestricted raw events.
- Keep credentials and payloads out of logs, traces, build artifacts, and support tickets.
This design does not by itself establish compliance with any particular regulation. Privacy counsel and security owners must confirm the applicable data residency, consent, retention, and deletion requirements.
11. Cost
The figures below are an assumption-based monthly estimate for the stated event volume, one production environment, modest staging use, and the architecture described. They are planning ranges, not AWS quotes or RudderStack pricing. They exclude engineering labor, BI licenses, taxes, and exceptional backfill or egress charges.
| Cost area | Monthly planning assumption |
|---|---|
| RudderStack Cloud | $500–$2,000, subject to plan and event volume |
| ECS Fargate services and scheduled tasks | $150–$500 |
| RDS PostgreSQL Multi-AZ and storage | $250–$700 |
| SQS, CloudWatch, and observability ingestion | $100–$400 |
| S3 storage, requests, and lifecycle overhead | $30–$150 |
| Redshift Serverless and snapshots | $300–$1,200 |
| Secrets, KMS, and miscellaneous AWS services | $50–$200 |
| Estimated total | $1,380–$5,150 |
The assumptions that move the bill most are RudderStack’s commercial plan, warehouse query concurrency, retention, and telemetry volume. Keep raw data compressed and lifecycle-managed, avoid unbounded warehouse scans, and sample high-volume diagnostic telemetry where appropriate. Review actual usage monthly before increasing retention or adding another serving warehouse.
12. Failure modes, resilience and disaster recovery
A resilient design makes failure visible and recovery repeatable. It does not assume that queues, vendors, or warehouses never fail.
| Failure mode | Behavior and recovery |
|---|---|
| Browser closes before sending | Accept that some client events are best-effort. Do not infer business completion from a browser-only event. |
| RudderStack endpoint unavailable | Retain server events in SQS and retry with bounded backoff; alert on oldest-message age and DLQ growth. |
| Publisher fails after vendor acceptance | A retry may duplicate the event. Preserve messageId and deduplicate in the warehouse. |
| S3 notification is duplicated or delayed | Treat object keys as idempotent work; periodically reconcile S3 objects against loader checkpoints. |
| Warehouse load fails | Leave the object recoverable in S3, retry the load, and expose the failure in the freshness dashboard. |
| Tracking plan change breaks a model | Stop promotion in CI when contract or data-quality tests fail; retain raw data to repair and rebuild. |
| Queue or regional service disruption | Keep the application transaction independent, use the queue and outbox for recovery, then replay from durable records. |
Use Multi-AZ RDS for availability within a region. S3 versioning protects against some accidental overwrites or deletes; it is not a substitute for a documented recovery plan. Enable appropriate Redshift recovery features and test restoring a representative dataset. Keep infrastructure and transformation code in version control, and store required configuration securely.
For this assumed scale, set an initial analytics recovery objective of restoring warehouse freshness within one business day after a regional or warehouse failure. The raw S3 archive and server-event queue provide replay paths, but the achievable recovery point and time depend on retention, vendor delivery behavior, and recovery capacity. Test a queue drain, a warehouse rebuild from raw files, and a secrets or role rotation at least periodically. The product application should continue operating during analytics recovery.
13. Evolving to the next scale
Do not add streaming infrastructure because the event count appears large in isolation. First identify which objective is failing: freshness, ingestion reliability, query concurrency, cost, or governance.
| Trigger | Change to consider |
|---|---|
| Queue age exceeds the delivery objective during ordinary peaks | Increase publisher concurrency, batch work safely, or split traffic by source or priority. |
| S3 objects become too small or numerous | Adjust destination batching and loader aggregation; avoid starting with one load per object. |
| Warehouse queries compete with ingestion | Separate workloads, tune models and sort keys from observed queries, or consider a different warehouse capacity model. |
| Repeated backfills disrupt normal use | Create a controlled backfill workflow with isolated capacity, checkpoints, and explicit cost approval. |
| Multiple teams change event schemas independently | Introduce a centrally reviewed tracking-plan release process and ownership per event domain. |
| Event routing or transformation needs exceed the managed path | Reassess whether a dedicated streaming platform is justified by a measured requirement, not by fashion. |
The next scale step is often operational rather than architectural: better data-quality ownership, a reliable replay tool, warehouse cost controls, and a clear process for changing metrics. A platform rewrite is rarely the first useful response.
14. Phased implementation plan
- Agree on the contract. Select a small set of product questions, define the first events and properties, assign owners, and document identity, consent, and data classification.
- Build the event boundary. Add typed event factories and contract tests. Instrument a browser event and one important server-side event; keep non-critical events best-effort.
- Deploy the durable path. Add the outbox, SQS queue, publisher, secrets, and dead-letter handling. Verify that analytics failures do not fail the business transaction.
- Create the raw and warehouse path. Route events to S3, configure notification and loader behavior, create Redshift staging and curated models, and add deduplication and freshness checks.
- Release through staging. Run unit, integration, contract, and load tests against isolated sources and data. Validate dashboards, alert routing, and a replay from S3.
- Promote and operate. Release a small, reviewed event set to production. Watch event volume, delivery freshness, data quality, and cost; expand the tracking plan only when the first metrics are trusted.
The success criterion is not that every possible user interaction is tracked. It is that a small, governed set of events reaches analysts with known semantics, visible failure behavior, and a credible path to correction.
This article was generated with AI. #AIGenerated