Skip to content

OpenTelemetry End to End: Traces, Metrics, and Logs for Node.js and .NET

A medium-scale AWS blueprint for instrumenting Node.js and .NET services with OpenTelemetry, propagating context across a queue, and operating traces, metrics, and logs.

Ekanatha Reddy Urivakili18 min3,411 words

1) Context and scale

This blueprint instruments a small order-processing product with a Node.js HTTP API and a .NET background worker. The API validates requests, reads cached product data, persists order state in PostgreSQL, and sends work to a queue. The worker consumes queued jobs and completes asynchronous processing. Both services emit OpenTelemetry traces, metrics, and logs through an OpenTelemetry Collector to one managed observability backend.

The example runs on AWS, but keeps application instrumentation vendor-neutral. The goal is not to build a telemetry platform. It is to answer practical production questions: did a request fail, where did its time go, what happened after it entered the queue, and is the problem isolated to an application or a shared dependency?

The runtime targets are Node.js 24 LTS and .NET 10. Keep runtime patching and OpenTelemetry package versions current through normal dependency-update and release processes; pin compatible package versions in lock files rather than copying unpinned dependencies into production.

AssumptionPlanning value
Daily active users50,000
API traffic400,000 requests/day; 5 requests/s average and 100 requests/s peak
Background work1–2 queue messages per accepted order
Application data200 GB initially; growth and retention reviewed quarterly
Telemetry10 GB/month of searchable logs and traces after sampling and retention controls
Engineering team6–8 engineers across 1–2 teams; shared platform ownership
Monthly infrastructure and observability budget$1,500 target; assumption-based planning limit

These numbers are planning assumptions, not measured capacity or vendor pricing. The peak rate is intentionally much higher than the average to accommodate traffic bursts. Validate it with representative load tests before sizing production tasks.

2) Requirements

Functional requirements describe what the system must observe; non-functional requirements set operating boundaries. Neither table implies that every signal should be retained forever. Telemetry is operational data, and its volume, sensitivity, and retention require deliberate controls.

Functional requirementAcceptance criterion
Trace synchronous requestsA request trace shows the API, database, cache, and outbound calls where instrumentation is available.
Trace asynchronous workTrace context is propagated in the queue message so the consumer can link its processing to the producer.
Collect metricsServices expose request rate, errors, duration, queue health, and runtime/resource measurements.
Correlate logsStructured logs include service identity and trace/span identifiers when emitted in an active trace.
Search across servicesEngineers can filter signals by service, environment, version, and trace ID without embedding user data in telemetry.
Support diagnosisDashboards and alerts distinguish user-visible failures from resource saturation and dependency trouble.
Non-functional requirementTarget or constraint
AvailabilityApplication availability target is 99.9% per month, excluding agreed maintenance and external customer-side failures.
PerformanceInstrumentation should have bounded export queues, batching, and timeouts; telemetry export must not block request handling.
CardinalityMetric labels are bounded. Never use user IDs, order IDs, URLs with identifiers, or exception messages as metric dimensions.
PrivacyDo not record credentials, payment details, request bodies, or unrestricted query parameters.
ResilienceA telemetry backend or collector outage must not take down the API or stop the worker from processing.
OperabilityOne team can deploy, tune, and troubleshoot the collector configuration and service instrumentation.

3) High-level design

The request path is synchronous up to durable order acceptance. SQS separates that response from slower processing. Trace context travels in message attributes; the worker creates a consumer span and records the message relationship. This is the important boundary to test: automatic HTTP instrumentation cannot infer causality across a queue by itself.

Each ECS task runs a local OpenTelemetry Collector sidecar. Applications send OTLP to the sidecar over the task’s loopback network. The collector batches, applies resource attributes, and exports all three signals to Grafana Cloud. Application logs are also written as structured JSON to stdout so ECS can retain a fallback copy in CloudWatch Logs. That fallback is not a second source of truth for dashboards; it is useful when the primary telemetry path is impaired.

flowchart LR
  Client["Clients"] --> Alb["AWS Application Load Balancer"]
  Alb --> Api["Node.js API on ECS Fargate"]
  Api --> Redis["Amazon ElastiCache for Redis"]
  Api --> Db["Amazon Aurora PostgreSQL"]
  Api --> Queue["Amazon SQS"]
  Queue --> Worker[".NET worker on ECS Fargate"]
  Worker --> Db
  Api --> CollectorA["OpenTelemetry Collector sidecar"]
  Worker --> CollectorW["OpenTelemetry Collector sidecar"]
  CollectorA --> Grafana["Grafana Cloud OTLP"]
  CollectorW --> Grafana
  Api --> CloudWatch["Amazon CloudWatch Logs"]
  Worker --> CloudWatch

The API and worker share telemetry conventions while keeping telemetry export out of their request and job critical paths.

Use private subnets for the tasks and database, with a public ALB as the only internet-facing application entry point. The collector sidecars need controlled egress to the OTLP endpoint. Restrict that path using the network controls available in the deployment, and use secret-backed authentication rather than placing a telemetry token in a task definition or image.

4) Cloud service mapping

AWS is the deployment choice, not an instrumentation requirement. The alternatives below are functional counterparts, not claims that their APIs, pricing, or operational behavior are identical. Grafana Cloud is a separate managed SaaS dependency in this design; the cloud-native alternatives are shown for teams that prefer a single-cloud observability boundary.

ComponentAWS choiceAzure equivalentGCP equivalent
HTTP ingressApplication Load BalancerApplication GatewayCloud Load Balancing
Container runtimeECS on FargateAzure Container AppsCloud Run
Relational databaseAurora PostgreSQLAzure Database for PostgreSQLCloud SQL for PostgreSQL
CacheElastiCache for Redis OSSAzure Managed RedisMemorystore for Redis
Work queueAmazon SQSAzure Service Bus queuesCloud Tasks or Pub/Sub, depending on delivery semantics
Secret storageAWS Secrets ManagerAzure Key VaultSecret Manager
Platform logs and infrastructure metricsCloudWatchAzure MonitorCloud Monitoring and Cloud Logging
Managed telemetry backendGrafana CloudGrafana Cloud, or Azure MonitorGrafana Cloud, or Google Cloud Observability

Do not treat the database or queue as interchangeable merely because another provider offers a service with a similar name. Re-evaluate delivery guarantees, retention, transaction behavior, networking, identity, and failure semantics during a cloud migration.

5) Data design

The product database stores business state, not telemetry. A concise schema gives the API and worker a shared durable record of order processing. Telemetry belongs in the observability backend, where trace spans, metric time series, and logs have different query and retention characteristics.

erDiagram
  CUSTOMER ||--o{ ORDER : places
  ORDER ||--o{ ORDER_ITEM : contains
  ORDER ||--o{ OUTBOX_EVENT : emits
  CUSTOMER {
    uuid id
    text status
    timestamptz created_at
  }
  ORDER {
    uuid id
    uuid customer_id
    text status
    timestamptz created_at
  }
  ORDER_ITEM {
    uuid order_id
    int line_number
    text sku
    int quantity
  }
  OUTBOX_EVENT {
    uuid id
    uuid order_id
    text event_type
    timestamptz created_at
    timestamptz published_at
  }

Business records and the outbox make accepted work durable; telemetry remains outside the transactional schema.

An outbox is justified when the business invariant is “an accepted order must eventually be published.” The API writes the order and outbox event in one PostgreSQL transaction; a dispatcher publishes pending events to SQS and marks them published. This avoids the dual-write failure where the database commit succeeds but queue publication fails. For a simpler product with a tolerable recovery path, direct publication may be acceptable, but that is a product decision, not an observability shortcut.

CREATE TABLE customer (
    id          uuid PRIMARY KEY,
    status      text NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE orders (
    id          uuid PRIMARY KEY,
    customer_id uuid NOT NULL REFERENCES customer(id),
    status      text NOT NULL CHECK (status IN ('accepted', 'processing', 'complete', 'failed')),
    created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX orders_customer_created_idx
    ON orders (customer_id, created_at DESC);

CREATE TABLE order_item (
    order_id    uuid NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
    line_number integer NOT NULL,
    sku         text NOT NULL,
    quantity    integer NOT NULL CHECK (quantity > 0),
    PRIMARY KEY (order_id, line_number)
);

CREATE TABLE outbox_event (
    id           uuid PRIMARY KEY,
    order_id     uuid NOT NULL REFERENCES orders(id),
    event_type   text NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz
);

CREATE INDEX outbox_pending_idx
    ON outbox_event (created_at)
    WHERE published_at IS NULL;

For telemetry, standardize resource attributes such as service.name, service.version, deployment.environment.name, and cloud.region. Use stable span attributes for operation type and dependency identity. Keep tenant IDs and other potentially identifying values out of shared telemetry unless security and privacy review explicitly approves a controlled, low-cardinality representation.

6) Low-level design

Instrument at two layers. Automatic instrumentation covers common framework and client boundaries, including HTTP servers, HTTP clients, and database calls supported by the chosen libraries. Manual instrumentation captures business operations that automatic instrumentation cannot infer: accepting an order, publishing an outbox event, and processing a queue message.

The application depends on OpenTelemetry APIs and exporters, not directly on Grafana SDKs. A service should remain runnable when telemetry export fails. Configure sampling and export limits in the collector and SDK, and give every service a distinct name and version.

classDiagram
  class OrderApi {
    +createOrder(request)
  }
  class OrderService {
    +acceptOrder(command)
  }
  class OrderRepository {
    +save(order)
  }
  class OutboxPublisher {
    +publishPending()
  }
  class OrderWorker {
    +process(message)
  }
  class Telemetry {
    +startSpan(name)
    +recordMetric(name, value)
    +emitLog(record)
  }
  OrderApi --> OrderService
  OrderService --> OrderRepository
  OrderService --> Telemetry
  OrderService --> OutboxPublisher
  OutboxPublisher --> Telemetry
  OrderWorker --> Telemetry

Business-level spans supplement, rather than duplicate, framework and dependency instrumentation.

Minimal runnable example: Node.js trace and metric export. Install compatible versions of @opentelemetry/sdk-node, @opentelemetry/api, @opentelemetry/auto-instrumentations-node, @opentelemetry/exporter-trace-otlp-http, and @opentelemetry/exporter-metrics-otlp-http. Start the collector locally with OTLP HTTP enabled on port 4318, then run this as an ES module.

import { NodeSDK } from '@opentelemetry/sdk-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { trace, metrics } from '@opentelemetry/api';

const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'node-api' }),
  traceExporter: new OTLPTraceExporter({ url: `${endpoint}/v1/traces` }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: `${endpoint}/v1/metrics` }),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

await sdk.start();

const meter = metrics.getMeter('node-api');
const requests = meter.createCounter('orders.accepted');
const tracer = trace.getTracer('node-api');

await tracer.startActiveSpan('order.accept', async (span) => {
  try {
    requests.add(1, { outcome: 'accepted' });
    console.log(JSON.stringify({ event: 'order.accepted', outcome: 'accepted' }));
  } finally {
    span.end();
  }
});

await sdk.shutdown();

Notice: auto-instrumentation covers supported libraries, but the named business span and counter express product behavior that a generic HTTP span cannot.

Realistic example: .NET worker setup. Add compatible versions of OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.OpenTelemetryProtocol, and the ASP.NET Core, HTTP, and runtime instrumentation packages. The collector endpoint is supplied through environment configuration; the same application can use a local collector in development and a sidecar in ECS.

using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
var serviceName = "order-worker";

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName, serviceVersion: "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment.name"] =
                builder.Environment.EnvironmentName
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource(serviceName)
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter(serviceName)
        .AddOtlpExporter());

builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeFormattedMessage = true;
    logging.IncludeScopes = true;
    logging.AddOtlpExporter();
});

var app = builder.Build();
var activitySource = new ActivitySource(serviceName);
var meter = new Meter(serviceName);
var processed = meter.CreateCounter<long>("orders.processed");

app.MapPost("/internal/process/{orderId}", (string orderId, ILogger<Program> logger) =>
{
    using var activity = activitySource.StartActivity("order.process");
    activity?.SetTag("order.operation", "process");
    processed.Add(1, new KeyValuePair<string, object?>("outcome", "accepted"));
    logger.LogInformation("Order processing started for {OrderId}", orderId);
    return Results.Accepted();
});

app.Run();

Notice: log attributes and trace context are exported through the OpenTelemetry logging provider; review the logging configuration and redaction rules before adding business identifiers to log scopes.

Keep metric names and dimensions in a shared instrumentation contract. Counters are monotonic; durations should use histograms; gauges represent current state. Avoid turning every span attribute into a metric label—high-cardinality metrics can become a cost and query-performance problem before they become a diagnostic advantage.

7) Key flows

For HTTP requests, the incoming trace context is extracted by server instrumentation. Supported client instrumentation propagates it to downstream HTTP calls. The API should return the trace ID in an approved response header only if that is useful for support workflows; a trace ID is not an authorization token.

sequenceDiagram
  participant C as Client
  participant A as Node API
  participant R as Redis
  participant D as Aurora PostgreSQL
  participant Q as SQS
  participant O as Collector
  participant G as Grafana Cloud
  C->>A: HTTP request with traceparent
  A->>R: Read product data
  R-->>A: Cache result
  A->>D: Commit order and outbox
  D-->>A: Commit complete
  A->>Q: Publish message with trace context
  A-->>C: Accepted response
  A->>O: OTLP traces, metrics, logs
  O->>G: Batched telemetry

The API records its own latency and outcomes while preserving trace context for asynchronous processing.

The queue consumer starts a new processing span using the message’s propagated context. For batch delivery, instrument each message or use span links where a single batch is related to multiple producer contexts; do not assign one arbitrary message’s parent to the whole batch. Acknowledgement happens only after the worker’s durable work completes.

sequenceDiagram
  participant Q as SQS
  participant W as .NET worker
  participant D as Aurora PostgreSQL
  participant O as Collector
  participant G as Grafana Cloud
  Q->>W: Deliver message with trace context
  W->>W: Extract context and start consumer span
  W->>D: Apply idempotent state transition
  D-->>W: Commit complete
  W->>Q: Delete acknowledged message
  W->>O: OTLP traces, metrics, logs
  O->>G: Batched telemetry

The consumer trace makes queue delay and processing time separately diagnosable.

8) Build and release

Build and test both services in CI, publish immutable container images, and deploy through staging before production. Infrastructure code owns task definitions, networking, IAM, queue configuration, database connectivity, and collector configuration. Avoid manually editing a running task to “fix” instrumentation; the resulting drift is hard to reproduce and easy to lose.

A Terraform fragment illustrates the sidecar pattern. It is not a complete deployable module: networking, task roles, secret injection, health checks, and the application container are omitted for clarity. The collector image should be pinned to an approved release or digest in the real module.

resource "aws_ecs_task_definition" "orders_api" {
  family                   = "orders-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024
  memory                   = 2048
  execution_role_arn        = aws_iam_role.task_execution.arn
  task_role_arn             = aws_iam_role.orders_api.arn

  container_definitions = jsonencode([
    {
      name      = "api"
      image     = var.api_image
      essential = true
      environment = [
        { name = "OTEL_SERVICE_NAME", value = "orders-api" },
        { name = "OTEL_EXPORTER_OTLP_ENDPOINT", value = "http://127.0.0.1:4318" },
        { name = "OTEL_EXPORTER_OTLP_PROTOCOL", value = "http/protobuf" }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.orders.name
          awslogs-region        = var.aws_region
          awslogs-stream-prefix = "api"
        }
      }
    },
    {
      name      = "otel-collector"
      image     = var.collector_image
      essential = false
      portMappings = [
        { containerPort = 4317, protocol = "tcp" },
        { containerPort = 4318, protocol = "tcp" }
      ]
      secrets = [
        { name = "GRAFANA_CLOUD_OTLP_AUTH", valueFrom = var.otlp_auth_secret_arn }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.orders.name
          awslogs-region        = var.aws_region
          awslogs-stream-prefix = "collector"
        }
      }
    }
  ])
}

A release workflow should deploy only after tests and image publication succeed. Use GitHub Actions OIDC to assume a narrowly scoped AWS role; do not store long-lived cloud keys in repository secrets.

name: build-test-deploy

on:
  pull_request:
  push:
    branches: [main]

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "24"
          cache: npm
      - run: npm ci
      - run: npm test -- --runInBand
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"
      - run: dotnet test --configuration Release

  deploy-staging:
    if: github.event_name == 'push'
    needs: verify
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.STAGING_DEPLOY_ROLE_ARN }}
          aws-region: us-east-1
      - run: ./deploy.sh staging "${GITHUB_SHA}"

The test strategy should exercise telemetry as behavior, not just verify that packages compile:

  • Unit: test business logic and metric outcomes without requiring a telemetry backend. Check that sensitive values are not attached to logs or spans.
  • Integration: run each service with a local collector and dependencies; assert that a known operation produces the expected service name, span, metric, and structured log.
  • Contract: verify the message schema and trace-context propagation across producer and consumer. Test compatibility for old messages that lack trace context.
  • Load: test the assumed peak and burst duration. Compare latency and resource use with instrumentation enabled and disabled; test collector backpressure and export timeouts.
  • Release: deploy to staging, run a synthetic request and queue job, and verify the complete trace and associated signals before production promotion.

9) Observability

Instrument once, then operate the data. Each application uses OpenTelemetry SDKs and OTLP exporters to send traces, metrics, and logs to its local collector. The collector adds deployment metadata, limits memory, batches exports, and sends telemetry to the managed backend. Keep SDK queues and collector queues bounded. On exporter failure, drop or retry according to configured limits rather than allowing telemetry to exhaust application memory.

Use structured logs with a stable schema. The following example is suitable for a JSON logger; trace_id and span_id should come from the active OpenTelemetry context, not from user input.

{
  "timestamp": "2026-11-07T12:00:00Z",
  "severity": "INFO",
  "service.name": "orders-api",
  "service.version": "1.8.2",
  "deployment.environment.name": "production",
  "event.name": "order.accepted",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "outcome": "accepted"
}

Use RED for service behavior: request or job Rate, Errors, and Duration. Use USE for resources: Utilization, Saturation, and Errors. For the API, RED includes HTTP request rate, non-success outcome rate, and latency distribution. For the worker, use message processing rate, failure rate, and processing duration. USE includes CPU and memory utilization, task restarts, database connection-pool saturation, queue age, and throttling or resource errors. Queue depth alone is not sufficient; age of the oldest message better expresses whether work is falling behind.

A trace should represent work, not every implementation detail. Use the standard HTTP, database, messaging, and runtime instrumentation available for each library. Add business spans around meaningful operations, and propagate W3C Trace Context through SQS message attributes. Configure the collector’s otlp receiver and otlphttp exporter; store the destination and authorization material as secrets.

SLISLO targetMeasurement window
Successful API requests99.9% of eligible requests succeed30-day rolling
API latency99% under 500 ms for the agreed synchronous endpoint set30-day rolling
Queue processing freshness99% of accepted messages begin processing within 2 minutes30-day rolling
Telemetry delivery99% of sampled application telemetry reaches the backend within 5 minutes30-day rolling

These are initial objectives, not guarantees. Define eligible requests, error classes, and latency boundaries precisely before using them for release decisions. Telemetry delivery is an operational objective; never page merely because one low-priority span was sampled or dropped.

Dashboards should include an API RED overview, worker RED and queue age, database/cache/queue dependency health, and a telemetry pipeline view showing collector restarts, queue occupancy, export failures, and dropped records. Give dashboards environment and service filters. Keep a trace-search workflow from a request ID or trace ID to related logs and downstream spans.

Page on sustained symptoms that threaten a user-facing SLO or data correctness:

  • API error-budget burn at a sustained fast or slow rate, using a multi-window burn-rate policy.
  • API availability or latency outside the agreed SLO with enough traffic to make the signal meaningful.
  • Oldest queue message age exceeding the processing freshness objective, especially with rising failures.
  • Database connection exhaustion, repeated failover, or sustained saturation that threatens request success.
  • Collector or export failures only when the resulting loss makes incident diagnosis materially unavailable; route lower-severity pipeline degradation to a ticket or team notification.

Keep CPU, memory, and isolated dependency warnings out of the paging path unless they predict user impact. Every page needs a runbook link, a clear owner, and a first diagnostic query or dashboard.

10) Security and compliance

Telemetry is a data export path and must receive the same scrutiny as other production integrations. Apply least privilege to task roles, secret access, and collector egress. Use TLS for external OTLP traffic and rotate credentials through the secrets system. The sidecar should have only the permissions required to read its configuration and send telemetry.

Adopt explicit collection rules:

  • Denylist authorization headers, cookies, request bodies, SQL parameters, payment data, and personally identifying fields.
  • Prefer route templates such as /orders/{id} to raw paths containing identifiers.
  • Do not attach customer IDs or order IDs to metric dimensions. If support needs a search key, use a reviewed, access-controlled log field and a defined retention period.
  • Restrict backend access by team and environment. Audit changes to sampling, retention, and redaction rules.
  • Treat trace IDs as correlation values, not secrets. They must not grant access to records or be used as proof of identity.
  • Set retention by operational need and regulatory obligations; verify deletion and data residency requirements with the chosen backend.

Test redaction in integration tests using synthetic sensitive values. A policy that exists only in documentation will eventually be bypassed by a new logger or instrumentation library.

11) Cost

This is an assumption-based monthly estimate, not a current vendor quote. Actual costs depend on region, service hours, task sizing, database configuration, data transfer, retention, and telemetry plan. Validate the estimate using current AWS and Grafana Cloud pricing calculators before approval.

Cost areaMonthly planning range (USD)Main assumption
ECS Fargate API and worker tasks$180–$350Small always-on service pools with peak scaling headroom
Aurora PostgreSQL and storage/backups$250–$500Multi-AZ production database; modest initial data volume
Redis, ALB, SQS, networking, and CloudWatch fallback logs$150–$350Medium-scale traffic and bounded platform-log retention
Grafana Cloud telemetry$150–$400Approximately 10 GB/month of retained searchable logs and traces; plan-dependent
Estimated total$730–$1,600Assumptions must be replaced by measured usage

Telemetry cost is governed by ingestion volume, retention, and query behavior—not just request count. Start with head sampling for routine traces if volume requires it, but preserve error and high-latency traces through a considered sampling policy. Keep metrics low-cardinality, filter health-check noise, and set retention by signal. Do not sample away the only evidence for rare failures without an alternative diagnostic path.

12) Failure modes, resilience and disaster recovery

Failure modeUser or operator impactResponse
Grafana Cloud unavailableTelemetry export fails; application should continueBounded collector queues, retry with limits, monitor drops, and keep CloudWatch fallback logs
Collector sidecar exitsTask loses its local OTLP destinationMark sidecar non-essential only if application behavior is independent; alert on repeated exits and restart tasks through normal orchestration
Incorrect sampling or redactionMissing diagnostic evidence or sensitive data exposureReview policy changes, test in staging, and make configuration versioned and reversible
SQS backlog growsDelayed work and potentially stale order stateScale consumers within database limits, inspect oldest-message age, and apply retry/DLQ policy
Database failover or exhaustionAPI errors or worker retriesUse bounded retries, idempotent processing, connection limits, and an operational runbook
Trace context missing or malformedBroken producer-consumer trace continuityTreat context as optional metadata; process the message normally and record a bounded diagnostic outcome

Resilience means observability failure does not become application failure. Set collector memory limits, batch sizes, retry duration, and queue capacities based on the task budget. Avoid unbounded retries and synchronous exporter calls in the request path. A collector should expose its own health and pipeline metrics; otherwise the telemetry pipeline can fail silently.

For disaster recovery, decide whether the application’s recovery objectives require a warm deployment in another region. At this scale, a documented restore-and-redeploy plan may be more appropriate than active-active infrastructure. Back up the database according to the business recovery point objective, test restoration, and keep infrastructure and collector configuration in version control. Telemetry does not replace application data backups; losing historical traces is usually a different class of event from losing orders.

13) Evolving to the next scale

Do not scale the telemetry architecture based on user counts alone. Scale when measured request concurrency, ingestion volume, query latency, queue age, or team boundaries justify it.

Signal that the current design is strainedNext change
Collector sidecars consume meaningful task CPU or memoryTune batching and sampling; then evaluate a shared, horizontally scaled collector gateway
One service’s telemetry overwhelms cost or query performanceSet service-specific retention and sampling; review noisy instrumentation and cardinality
Teams need independent release and access boundariesSeparate ownership, dashboards, and backend permissions without changing semantic conventions
Queue throughput competes with database capacityScale consumers with explicit concurrency limits and protect database connection budgets
Regional recovery becomes a business requirementAdd a regional deployment and tested failover plan; re-evaluate data and telemetry residency
Trace searches become too expensive or slowAdjust retention, tail sampling, and indexing strategy based on diagnosed use cases

A collector gateway is not automatically better than a sidecar. It centralizes policy and can simplify egress, but adds a shared network hop, a scaling unit, and another failure boundary. Keep the sidecar design until there is a concrete need for centralized processing or independent collector scaling.

14) Phased implementation plan

  1. Agree on signal contracts. Choose service names, environment and version attributes, span naming, metric names, log fields, redaction rules, and ownership. Decide which identifiers are forbidden in telemetry.
  2. Instrument one synchronous path. Add Node.js automatic HTTP and client instrumentation, a business span, RED metrics, and structured logs. Verify request context and export locally.
  3. Instrument the .NET worker and queue boundary. Add server/client and runtime instrumentation, consumer spans, and trace-context propagation in message attributes. Test messages without context and batch handling.
  4. Deploy the collector and backend integration. Version the collector configuration, manage credentials through secrets, add resource limits, and verify all signals in staging. Measure ingestion and cost against the assumptions.
  5. Add SLOs and operational workflows. Publish RED/USE dashboards, define eligible events and burn-rate alerts, write runbooks, and test synthetic API and worker flows.
  6. Validate failure behavior and release. Load-test the assumed peak, interrupt telemetry export in staging, verify application behavior, confirm queue recovery and database capacity, then promote gradually.

The durable outcome is not a dashboard count. It is a consistent path from a user-visible symptom to a trace, the associated logs and metrics, and an owner who can act. OpenTelemetry supplies the shared instrumentation and transport model; disciplined naming, bounded telemetry, and operational follow-through make that model useful.


This article was generated with AI. #AIGenerated