1) Context and scale
This blueprint instruments a small order-processing product with a Node.js HTTP API and a .NET background worker. The API validates requests, reads cached product data, persists order state in PostgreSQL, and sends work to a queue. The worker consumes queued jobs and completes asynchronous processing. Both services emit OpenTelemetry traces, metrics, and logs through an OpenTelemetry Collector to one managed observability backend.
The example runs on AWS, but keeps application instrumentation vendor-neutral. The goal is not to build a telemetry platform. It is to answer practical production questions: did a request fail, where did its time go, what happened after it entered the queue, and is the problem isolated to an application or a shared dependency?
The runtime targets are Node.js 24 LTS and .NET 10. Keep runtime patching and OpenTelemetry package versions current through normal dependency-update and release processes; pin compatible package versions in lock files rather than copying unpinned dependencies into production.
| Assumption | Planning value |
|---|---|
| Daily active users | 50,000 |
| API traffic | 400,000 requests/day; 5 requests/s average and 100 requests/s peak |
| Background work | 1–2 queue messages per accepted order |
| Application data | 200 GB initially; growth and retention reviewed quarterly |
| Telemetry | 10 GB/month of searchable logs and traces after sampling and retention controls |
| Engineering team | 6–8 engineers across 1–2 teams; shared platform ownership |
| Monthly infrastructure and observability budget | $1,500 target; assumption-based planning limit |
These numbers are planning assumptions, not measured capacity or vendor pricing. The peak rate is intentionally much higher than the average to accommodate traffic bursts. Validate it with representative load tests before sizing production tasks.
2) Requirements
Functional requirements describe what the system must observe; non-functional requirements set operating boundaries. Neither table implies that every signal should be retained forever. Telemetry is operational data, and its volume, sensitivity, and retention require deliberate controls.
| Functional requirement | Acceptance criterion |
|---|---|
| Trace synchronous requests | A request trace shows the API, database, cache, and outbound calls where instrumentation is available. |
| Trace asynchronous work | Trace context is propagated in the queue message so the consumer can link its processing to the producer. |
| Collect metrics | Services expose request rate, errors, duration, queue health, and runtime/resource measurements. |
| Correlate logs | Structured logs include service identity and trace/span identifiers when emitted in an active trace. |
| Search across services | Engineers can filter signals by service, environment, version, and trace ID without embedding user data in telemetry. |
| Support diagnosis | Dashboards and alerts distinguish user-visible failures from resource saturation and dependency trouble. |
| Non-functional requirement | Target or constraint |
|---|---|
| Availability | Application availability target is 99.9% per month, excluding agreed maintenance and external customer-side failures. |
| Performance | Instrumentation should have bounded export queues, batching, and timeouts; telemetry export must not block request handling. |
| Cardinality | Metric labels are bounded. Never use user IDs, order IDs, URLs with identifiers, or exception messages as metric dimensions. |
| Privacy | Do not record credentials, payment details, request bodies, or unrestricted query parameters. |
| Resilience | A telemetry backend or collector outage must not take down the API or stop the worker from processing. |
| Operability | One team can deploy, tune, and troubleshoot the collector configuration and service instrumentation. |
3) High-level design
The request path is synchronous up to durable order acceptance. SQS separates that response from slower processing. Trace context travels in message attributes; the worker creates a consumer span and records the message relationship. This is the important boundary to test: automatic HTTP instrumentation cannot infer causality across a queue by itself.
Each ECS task runs a local OpenTelemetry Collector sidecar. Applications send OTLP to the sidecar over the task’s loopback network. The collector batches, applies resource attributes, and exports all three signals to Grafana Cloud. Application logs are also written as structured JSON to stdout so ECS can retain a fallback copy in CloudWatch Logs. That fallback is not a second source of truth for dashboards; it is useful when the primary telemetry path is impaired.
flowchart LR Client["Clients"] --> Alb["AWS Application Load Balancer"] Alb --> Api["Node.js API on ECS Fargate"] Api --> Redis["Amazon ElastiCache for Redis"] Api --> Db["Amazon Aurora PostgreSQL"] Api --> Queue["Amazon SQS"] Queue --> Worker[".NET worker on ECS Fargate"] Worker --> Db Api --> CollectorA["OpenTelemetry Collector sidecar"] Worker --> CollectorW["OpenTelemetry Collector sidecar"] CollectorA --> Grafana["Grafana Cloud OTLP"] CollectorW --> Grafana Api --> CloudWatch["Amazon CloudWatch Logs"] Worker --> CloudWatch
The API and worker share telemetry conventions while keeping telemetry export out of their request and job critical paths.
Use private subnets for the tasks and database, with a public ALB as the only internet-facing application entry point. The collector sidecars need controlled egress to the OTLP endpoint. Restrict that path using the network controls available in the deployment, and use secret-backed authentication rather than placing a telemetry token in a task definition or image.
4) Cloud service mapping
AWS is the deployment choice, not an instrumentation requirement. The alternatives below are functional counterparts, not claims that their APIs, pricing, or operational behavior are identical. Grafana Cloud is a separate managed SaaS dependency in this design; the cloud-native alternatives are shown for teams that prefer a single-cloud observability boundary.
| Component | AWS choice | Azure equivalent | GCP equivalent |
|---|---|---|---|
| HTTP ingress | Application Load Balancer | Application Gateway | Cloud Load Balancing |
| Container runtime | ECS on Fargate | Azure Container Apps | Cloud Run |
| Relational database | Aurora PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Cache | ElastiCache for Redis OSS | Azure Managed Redis | Memorystore for Redis |
| Work queue | Amazon SQS | Azure Service Bus queues | Cloud Tasks or Pub/Sub, depending on delivery semantics |
| Secret storage | AWS Secrets Manager | Azure Key Vault | Secret Manager |
| Platform logs and infrastructure metrics | CloudWatch | Azure Monitor | Cloud Monitoring and Cloud Logging |
| Managed telemetry backend | Grafana Cloud | Grafana Cloud, or Azure Monitor | Grafana Cloud, or Google Cloud Observability |
Do not treat the database or queue as interchangeable merely because another provider offers a service with a similar name. Re-evaluate delivery guarantees, retention, transaction behavior, networking, identity, and failure semantics during a cloud migration.
5) Data design
The product database stores business state, not telemetry. A concise schema gives the API and worker a shared durable record of order processing. Telemetry belongs in the observability backend, where trace spans, metric time series, and logs have different query and retention characteristics.
erDiagram
CUSTOMER ||--o{ ORDER : places
ORDER ||--o{ ORDER_ITEM : contains
ORDER ||--o{ OUTBOX_EVENT : emits
CUSTOMER {
uuid id
text status
timestamptz created_at
}
ORDER {
uuid id
uuid customer_id
text status
timestamptz created_at
}
ORDER_ITEM {
uuid order_id
int line_number
text sku
int quantity
}
OUTBOX_EVENT {
uuid id
uuid order_id
text event_type
timestamptz created_at
timestamptz published_at
}Business records and the outbox make accepted work durable; telemetry remains outside the transactional schema.
An outbox is justified when the business invariant is “an accepted order must eventually be published.” The API writes the order and outbox event in one PostgreSQL transaction; a dispatcher publishes pending events to SQS and marks them published. This avoids the dual-write failure where the database commit succeeds but queue publication fails. For a simpler product with a tolerable recovery path, direct publication may be acceptable, but that is a product decision, not an observability shortcut.
CREATE TABLE customer (
id uuid PRIMARY KEY,
status text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE orders (
id uuid PRIMARY KEY,
customer_id uuid NOT NULL REFERENCES customer(id),
status text NOT NULL CHECK (status IN ('accepted', 'processing', 'complete', 'failed')),
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX orders_customer_created_idx
ON orders (customer_id, created_at DESC);
CREATE TABLE order_item (
order_id uuid NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
line_number integer NOT NULL,
sku text NOT NULL,
quantity integer NOT NULL CHECK (quantity > 0),
PRIMARY KEY (order_id, line_number)
);
CREATE TABLE outbox_event (
id uuid PRIMARY KEY,
order_id uuid NOT NULL REFERENCES orders(id),
event_type text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz
);
CREATE INDEX outbox_pending_idx
ON outbox_event (created_at)
WHERE published_at IS NULL;
For telemetry, standardize resource attributes such as service.name, service.version, deployment.environment.name, and cloud.region. Use stable span attributes for operation type and dependency identity. Keep tenant IDs and other potentially identifying values out of shared telemetry unless security and privacy review explicitly approves a controlled, low-cardinality representation.
6) Low-level design
Instrument at two layers. Automatic instrumentation covers common framework and client boundaries, including HTTP servers, HTTP clients, and database calls supported by the chosen libraries. Manual instrumentation captures business operations that automatic instrumentation cannot infer: accepting an order, publishing an outbox event, and processing a queue message.
The application depends on OpenTelemetry APIs and exporters, not directly on Grafana SDKs. A service should remain runnable when telemetry export fails. Configure sampling and export limits in the collector and SDK, and give every service a distinct name and version.
classDiagram
class OrderApi {
+createOrder(request)
}
class OrderService {
+acceptOrder(command)
}
class OrderRepository {
+save(order)
}
class OutboxPublisher {
+publishPending()
}
class OrderWorker {
+process(message)
}
class Telemetry {
+startSpan(name)
+recordMetric(name, value)
+emitLog(record)
}
OrderApi --> OrderService
OrderService --> OrderRepository
OrderService --> Telemetry
OrderService --> OutboxPublisher
OutboxPublisher --> Telemetry
OrderWorker --> TelemetryBusiness-level spans supplement, rather than duplicate, framework and dependency instrumentation.
Minimal runnable example: Node.js trace and metric export. Install compatible versions of @opentelemetry/sdk-node, @opentelemetry/api, @opentelemetry/auto-instrumentations-node, @opentelemetry/exporter-trace-otlp-http, and @opentelemetry/exporter-metrics-otlp-http. Start the collector locally with OTLP HTTP enabled on port 4318, then run this as an ES module.
import { NodeSDK } from '@opentelemetry/sdk-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { trace, metrics } from '@opentelemetry/api';
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318';
const sdk = new NodeSDK({
resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'node-api' }),
traceExporter: new OTLPTraceExporter({ url: `${endpoint}/v1/traces` }),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({ url: `${endpoint}/v1/metrics` }),
}),
instrumentations: [getNodeAutoInstrumentations()],
});
await sdk.start();
const meter = metrics.getMeter('node-api');
const requests = meter.createCounter('orders.accepted');
const tracer = trace.getTracer('node-api');
await tracer.startActiveSpan('order.accept', async (span) => {
try {
requests.add(1, { outcome: 'accepted' });
console.log(JSON.stringify({ event: 'order.accepted', outcome: 'accepted' }));
} finally {
span.end();
}
});
await sdk.shutdown();
Notice: auto-instrumentation covers supported libraries, but the named business span and counter express product behavior that a generic HTTP span cannot.
Realistic example: .NET worker setup. Add compatible versions of OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol, and the ASP.NET Core, HTTP, and runtime instrumentation packages. The collector endpoint is supplied through environment configuration; the same application can use a local collector in development and a sidecar in ECS.
using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
var serviceName = "order-worker";
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(serviceName, serviceVersion: "1.0.0")
.AddAttributes(new Dictionary<string, object>
{
["deployment.environment.name"] =
builder.Environment.EnvironmentName
}))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource(serviceName)
.AddOtlpExporter())
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddRuntimeInstrumentation()
.AddMeter(serviceName)
.AddOtlpExporter());
builder.Logging.AddOpenTelemetry(logging =>
{
logging.IncludeFormattedMessage = true;
logging.IncludeScopes = true;
logging.AddOtlpExporter();
});
var app = builder.Build();
var activitySource = new ActivitySource(serviceName);
var meter = new Meter(serviceName);
var processed = meter.CreateCounter<long>("orders.processed");
app.MapPost("/internal/process/{orderId}", (string orderId, ILogger<Program> logger) =>
{
using var activity = activitySource.StartActivity("order.process");
activity?.SetTag("order.operation", "process");
processed.Add(1, new KeyValuePair<string, object?>("outcome", "accepted"));
logger.LogInformation("Order processing started for {OrderId}", orderId);
return Results.Accepted();
});
app.Run();
Notice: log attributes and trace context are exported through the OpenTelemetry logging provider; review the logging configuration and redaction rules before adding business identifiers to log scopes.
Keep metric names and dimensions in a shared instrumentation contract. Counters are monotonic; durations should use histograms; gauges represent current state. Avoid turning every span attribute into a metric label—high-cardinality metrics can become a cost and query-performance problem before they become a diagnostic advantage.
7) Key flows
For HTTP requests, the incoming trace context is extracted by server instrumentation. Supported client instrumentation propagates it to downstream HTTP calls. The API should return the trace ID in an approved response header only if that is useful for support workflows; a trace ID is not an authorization token.
sequenceDiagram participant C as Client participant A as Node API participant R as Redis participant D as Aurora PostgreSQL participant Q as SQS participant O as Collector participant G as Grafana Cloud C->>A: HTTP request with traceparent A->>R: Read product data R-->>A: Cache result A->>D: Commit order and outbox D-->>A: Commit complete A->>Q: Publish message with trace context A-->>C: Accepted response A->>O: OTLP traces, metrics, logs O->>G: Batched telemetry
The API records its own latency and outcomes while preserving trace context for asynchronous processing.
The queue consumer starts a new processing span using the message’s propagated context. For batch delivery, instrument each message or use span links where a single batch is related to multiple producer contexts; do not assign one arbitrary message’s parent to the whole batch. Acknowledgement happens only after the worker’s durable work completes.
sequenceDiagram participant Q as SQS participant W as .NET worker participant D as Aurora PostgreSQL participant O as Collector participant G as Grafana Cloud Q->>W: Deliver message with trace context W->>W: Extract context and start consumer span W->>D: Apply idempotent state transition D-->>W: Commit complete W->>Q: Delete acknowledged message W->>O: OTLP traces, metrics, logs O->>G: Batched telemetry
The consumer trace makes queue delay and processing time separately diagnosable.
8) Build and release
Build and test both services in CI, publish immutable container images, and deploy through staging before production. Infrastructure code owns task definitions, networking, IAM, queue configuration, database connectivity, and collector configuration. Avoid manually editing a running task to “fix” instrumentation; the resulting drift is hard to reproduce and easy to lose.
A Terraform fragment illustrates the sidecar pattern. It is not a complete deployable module: networking, task roles, secret injection, health checks, and the application container are omitted for clarity. The collector image should be pinned to an approved release or digest in the real module.
resource "aws_ecs_task_definition" "orders_api" {
family = "orders-api"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = 1024
memory = 2048
execution_role_arn = aws_iam_role.task_execution.arn
task_role_arn = aws_iam_role.orders_api.arn
container_definitions = jsonencode([
{
name = "api"
image = var.api_image
essential = true
environment = [
{ name = "OTEL_SERVICE_NAME", value = "orders-api" },
{ name = "OTEL_EXPORTER_OTLP_ENDPOINT", value = "http://127.0.0.1:4318" },
{ name = "OTEL_EXPORTER_OTLP_PROTOCOL", value = "http/protobuf" }
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.orders.name
awslogs-region = var.aws_region
awslogs-stream-prefix = "api"
}
}
},
{
name = "otel-collector"
image = var.collector_image
essential = false
portMappings = [
{ containerPort = 4317, protocol = "tcp" },
{ containerPort = 4318, protocol = "tcp" }
]
secrets = [
{ name = "GRAFANA_CLOUD_OTLP_AUTH", valueFrom = var.otlp_auth_secret_arn }
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.orders.name
awslogs-region = var.aws_region
awslogs-stream-prefix = "collector"
}
}
}
])
}
A release workflow should deploy only after tests and image publication succeed. Use GitHub Actions OIDC to assume a narrowly scoped AWS role; do not store long-lived cloud keys in repository secrets.
name: build-test-deploy
on:
pull_request:
push:
branches: [main]
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "24"
cache: npm
- run: npm ci
- run: npm test -- --runInBand
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.0.x"
- run: dotnet test --configuration Release
deploy-staging:
if: github.event_name == 'push'
needs: verify
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.STAGING_DEPLOY_ROLE_ARN }}
aws-region: us-east-1
- run: ./deploy.sh staging "${GITHUB_SHA}"
The test strategy should exercise telemetry as behavior, not just verify that packages compile:
- Unit: test business logic and metric outcomes without requiring a telemetry backend. Check that sensitive values are not attached to logs or spans.
- Integration: run each service with a local collector and dependencies; assert that a known operation produces the expected service name, span, metric, and structured log.
- Contract: verify the message schema and trace-context propagation across producer and consumer. Test compatibility for old messages that lack trace context.
- Load: test the assumed peak and burst duration. Compare latency and resource use with instrumentation enabled and disabled; test collector backpressure and export timeouts.
- Release: deploy to staging, run a synthetic request and queue job, and verify the complete trace and associated signals before production promotion.
9) Observability
Instrument once, then operate the data. Each application uses OpenTelemetry SDKs and OTLP exporters to send traces, metrics, and logs to its local collector. The collector adds deployment metadata, limits memory, batches exports, and sends telemetry to the managed backend. Keep SDK queues and collector queues bounded. On exporter failure, drop or retry according to configured limits rather than allowing telemetry to exhaust application memory.
Use structured logs with a stable schema. The following example is suitable for a JSON logger; trace_id and span_id should come from the active OpenTelemetry context, not from user input.
{
"timestamp": "2026-11-07T12:00:00Z",
"severity": "INFO",
"service.name": "orders-api",
"service.version": "1.8.2",
"deployment.environment.name": "production",
"event.name": "order.accepted",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"outcome": "accepted"
}
Use RED for service behavior: request or job Rate, Errors, and Duration. Use USE for resources: Utilization, Saturation, and Errors. For the API, RED includes HTTP request rate, non-success outcome rate, and latency distribution. For the worker, use message processing rate, failure rate, and processing duration. USE includes CPU and memory utilization, task restarts, database connection-pool saturation, queue age, and throttling or resource errors. Queue depth alone is not sufficient; age of the oldest message better expresses whether work is falling behind.
A trace should represent work, not every implementation detail. Use the standard HTTP, database, messaging, and runtime instrumentation available for each library. Add business spans around meaningful operations, and propagate W3C Trace Context through SQS message attributes. Configure the collector’s otlp receiver and otlphttp exporter; store the destination and authorization material as secrets.
| SLI | SLO target | Measurement window |
|---|---|---|
| Successful API requests | 99.9% of eligible requests succeed | 30-day rolling |
| API latency | 99% under 500 ms for the agreed synchronous endpoint set | 30-day rolling |
| Queue processing freshness | 99% of accepted messages begin processing within 2 minutes | 30-day rolling |
| Telemetry delivery | 99% of sampled application telemetry reaches the backend within 5 minutes | 30-day rolling |
These are initial objectives, not guarantees. Define eligible requests, error classes, and latency boundaries precisely before using them for release decisions. Telemetry delivery is an operational objective; never page merely because one low-priority span was sampled or dropped.
Dashboards should include an API RED overview, worker RED and queue age, database/cache/queue dependency health, and a telemetry pipeline view showing collector restarts, queue occupancy, export failures, and dropped records. Give dashboards environment and service filters. Keep a trace-search workflow from a request ID or trace ID to related logs and downstream spans.
Page on sustained symptoms that threaten a user-facing SLO or data correctness:
- API error-budget burn at a sustained fast or slow rate, using a multi-window burn-rate policy.
- API availability or latency outside the agreed SLO with enough traffic to make the signal meaningful.
- Oldest queue message age exceeding the processing freshness objective, especially with rising failures.
- Database connection exhaustion, repeated failover, or sustained saturation that threatens request success.
- Collector or export failures only when the resulting loss makes incident diagnosis materially unavailable; route lower-severity pipeline degradation to a ticket or team notification.
Keep CPU, memory, and isolated dependency warnings out of the paging path unless they predict user impact. Every page needs a runbook link, a clear owner, and a first diagnostic query or dashboard.
10) Security and compliance
Telemetry is a data export path and must receive the same scrutiny as other production integrations. Apply least privilege to task roles, secret access, and collector egress. Use TLS for external OTLP traffic and rotate credentials through the secrets system. The sidecar should have only the permissions required to read its configuration and send telemetry.
Adopt explicit collection rules:
- Denylist authorization headers, cookies, request bodies, SQL parameters, payment data, and personally identifying fields.
- Prefer route templates such as
/orders/{id}to raw paths containing identifiers. - Do not attach customer IDs or order IDs to metric dimensions. If support needs a search key, use a reviewed, access-controlled log field and a defined retention period.
- Restrict backend access by team and environment. Audit changes to sampling, retention, and redaction rules.
- Treat trace IDs as correlation values, not secrets. They must not grant access to records or be used as proof of identity.
- Set retention by operational need and regulatory obligations; verify deletion and data residency requirements with the chosen backend.
Test redaction in integration tests using synthetic sensitive values. A policy that exists only in documentation will eventually be bypassed by a new logger or instrumentation library.
11) Cost
This is an assumption-based monthly estimate, not a current vendor quote. Actual costs depend on region, service hours, task sizing, database configuration, data transfer, retention, and telemetry plan. Validate the estimate using current AWS and Grafana Cloud pricing calculators before approval.
| Cost area | Monthly planning range (USD) | Main assumption |
|---|---|---|
| ECS Fargate API and worker tasks | $180–$350 | Small always-on service pools with peak scaling headroom |
| Aurora PostgreSQL and storage/backups | $250–$500 | Multi-AZ production database; modest initial data volume |
| Redis, ALB, SQS, networking, and CloudWatch fallback logs | $150–$350 | Medium-scale traffic and bounded platform-log retention |
| Grafana Cloud telemetry | $150–$400 | Approximately 10 GB/month of retained searchable logs and traces; plan-dependent |
| Estimated total | $730–$1,600 | Assumptions must be replaced by measured usage |
Telemetry cost is governed by ingestion volume, retention, and query behavior—not just request count. Start with head sampling for routine traces if volume requires it, but preserve error and high-latency traces through a considered sampling policy. Keep metrics low-cardinality, filter health-check noise, and set retention by signal. Do not sample away the only evidence for rare failures without an alternative diagnostic path.
12) Failure modes, resilience and disaster recovery
| Failure mode | User or operator impact | Response |
|---|---|---|
| Grafana Cloud unavailable | Telemetry export fails; application should continue | Bounded collector queues, retry with limits, monitor drops, and keep CloudWatch fallback logs |
| Collector sidecar exits | Task loses its local OTLP destination | Mark sidecar non-essential only if application behavior is independent; alert on repeated exits and restart tasks through normal orchestration |
| Incorrect sampling or redaction | Missing diagnostic evidence or sensitive data exposure | Review policy changes, test in staging, and make configuration versioned and reversible |
| SQS backlog grows | Delayed work and potentially stale order state | Scale consumers within database limits, inspect oldest-message age, and apply retry/DLQ policy |
| Database failover or exhaustion | API errors or worker retries | Use bounded retries, idempotent processing, connection limits, and an operational runbook |
| Trace context missing or malformed | Broken producer-consumer trace continuity | Treat context as optional metadata; process the message normally and record a bounded diagnostic outcome |
Resilience means observability failure does not become application failure. Set collector memory limits, batch sizes, retry duration, and queue capacities based on the task budget. Avoid unbounded retries and synchronous exporter calls in the request path. A collector should expose its own health and pipeline metrics; otherwise the telemetry pipeline can fail silently.
For disaster recovery, decide whether the application’s recovery objectives require a warm deployment in another region. At this scale, a documented restore-and-redeploy plan may be more appropriate than active-active infrastructure. Back up the database according to the business recovery point objective, test restoration, and keep infrastructure and collector configuration in version control. Telemetry does not replace application data backups; losing historical traces is usually a different class of event from losing orders.
13) Evolving to the next scale
Do not scale the telemetry architecture based on user counts alone. Scale when measured request concurrency, ingestion volume, query latency, queue age, or team boundaries justify it.
| Signal that the current design is strained | Next change |
|---|---|
| Collector sidecars consume meaningful task CPU or memory | Tune batching and sampling; then evaluate a shared, horizontally scaled collector gateway |
| One service’s telemetry overwhelms cost or query performance | Set service-specific retention and sampling; review noisy instrumentation and cardinality |
| Teams need independent release and access boundaries | Separate ownership, dashboards, and backend permissions without changing semantic conventions |
| Queue throughput competes with database capacity | Scale consumers with explicit concurrency limits and protect database connection budgets |
| Regional recovery becomes a business requirement | Add a regional deployment and tested failover plan; re-evaluate data and telemetry residency |
| Trace searches become too expensive or slow | Adjust retention, tail sampling, and indexing strategy based on diagnosed use cases |
A collector gateway is not automatically better than a sidecar. It centralizes policy and can simplify egress, but adds a shared network hop, a scaling unit, and another failure boundary. Keep the sidecar design until there is a concrete need for centralized processing or independent collector scaling.
14) Phased implementation plan
- Agree on signal contracts. Choose service names, environment and version attributes, span naming, metric names, log fields, redaction rules, and ownership. Decide which identifiers are forbidden in telemetry.
- Instrument one synchronous path. Add Node.js automatic HTTP and client instrumentation, a business span, RED metrics, and structured logs. Verify request context and export locally.
- Instrument the .NET worker and queue boundary. Add server/client and runtime instrumentation, consumer spans, and trace-context propagation in message attributes. Test messages without context and batch handling.
- Deploy the collector and backend integration. Version the collector configuration, manage credentials through secrets, add resource limits, and verify all signals in staging. Measure ingestion and cost against the assumptions.
- Add SLOs and operational workflows. Publish RED/USE dashboards, define eligible events and burn-rate alerts, write runbooks, and test synthetic API and worker flows.
- Validate failure behavior and release. Load-test the assumed peak, interrupt telemetry export in staging, verify application behavior, confirm queue recovery and database capacity, then promote gradually.
The durable outcome is not a dashboard count. It is a consistent path from a user-visible symptom to a trace, the associated logs and metrics, and an owner who can act. OpenTelemetry supplies the shared instrumentation and transport model; disciplined naming, bounded telemetry, and operational follow-through make that model useful.
This article was generated with AI. #AIGenerated