Skip to content

Observability for a Small App: Logs, Uptime Checks, and One Useful Dashboard

A small-team AWS blueprint for turning structured logs, external uptime checks, traces, and a focused dashboard into a practical production feedback loop.

Ekanatha Reddy Urivakili19 min3,816 words

1. Context and scale

A small application does not need an observability platform assembled from several databases and dashboards. It needs to answer a short list of operational questions quickly:

  • Can a user reach the app and complete its main action?
  • If a request fails, what failed, and which deployment introduced it?
  • Is the problem isolated to the app, the database, or an external dependency?
  • Is anyone expected to act on the issue right now?

This blueprint uses a single-region AWS deployment: a .NET web monolith on Amazon ECS with AWS Fargate, an Application Load Balancer, and Amazon RDS for PostgreSQL. Amazon CloudWatch collects logs and metrics; AWS X-Ray stores traces; CloudWatch Synthetics checks the public endpoint from outside the application. The team operates one dashboard and a small number of alerts.

The point is not that AWS is the only reasonable choice. The point is to keep the operational path short: use managed services, emit a few useful signals, and make every alert correspond to an action.

AssumptionPlanning value
Daily active users500
Typical requests per second0.1; bursts up to 5
Application data volume5 GB initially, growing slowly
Team size2 engineers, shared on-call
DeploymentOne AWS region; one production environment
Monthly observability budget$20–$50 within a broader app budget
Availability objective99.5% monthly, an explicit product assumption

These are sizing assumptions, not measured capacity claims. A small workload can still have sharp traffic spikes, slow database queries, or a failed release. Observability should make those conditions visible without creating a second system that the team has to operate.

2. Requirements

Requirements should focus on actions the team can take, not on collecting every possible signal.

Functional requirements

RequirementDesign response
Confirm public availabilityRun an external CloudWatch Synthetics canary against the production URL every five minutes.
Diagnose failed requestsEmit JSON logs with severity, route template, status, trace ID, and release identifier.
Find slow operationsCreate distributed traces for incoming HTTP requests and database calls.
Spot user-visible degradationTrack request rate, error rate, latency, and canary failures.
Relate incidents to releasesAdd the same immutable release identifier to logs, metrics, traces, and deployment records.
Notify the teamRoute actionable CloudWatch alarms to an SNS topic connected to the team’s notification channel.

Non-functional requirements

RequirementTarget or constraint
Operational simplicityOne application service, one managed database, one dashboard, and a short alert list.
Signal costBound log retention and sample routine traces; do not retain unlimited debug output.
Diagnostic contextA failed request should be traceable across application logs and traces using a correlation ID.
Safe health checksLiveness checks must not depend on the database; readiness and external checks must have short timeouts.
Low cardinalityNever use user IDs, request IDs, or raw URLs as metric dimensions.
RecoverabilityRebuild the service from infrastructure and deployment definitions; back up the database independently.

3. High-level design

The public request path goes through Route 53 and an ALB to the Fargate service. The application writes structured logs to CloudWatch Logs and sends OpenTelemetry data to an AWS Distro for OpenTelemetry (ADOT) Collector sidecar. The collector exports traces to X-Ray and application metrics to CloudWatch. RDS stays private to the application network.

A CloudWatch Synthetics canary uses the public hostname, so it exercises DNS, TLS, load balancing, application startup, and the readiness endpoint. This is deliberately different from an internal health check: an internal check cannot prove the public path works.

flowchart LR
    User["User or browser"] --> DNS["Amazon Route 53"]
    DNS --> ALB["Application Load Balancer"]
    ALB --> App["Amazon ECS on AWS Fargate"]
    App --> DB["Amazon RDS for PostgreSQL"]
    App --> Logs["Amazon CloudWatch Logs"]
    App --> Collector["ADOT Collector sidecar"]
    Collector --> XRay["AWS X-Ray"]
    Collector --> Metrics["Amazon CloudWatch metrics"]
    Canary["CloudWatch Synthetics canary"] --> DNS
    Metrics --> Alarm["CloudWatch alarms"]
    Logs --> Alarm
    Alarm --> Notify["Amazon SNS"]

The application emits telemetry at the point of work; the canary tests the path from outside the application.

Keep the first version intentionally small. Do not add a metrics database, log search cluster, or event bus just to gain a second place to query. If a signal cannot help explain a user-visible symptom or support a decision, defer it.

4. Cloud service mapping

AWS is the selected cloud. The other columns show equivalent service categories, not a promise that every product has identical semantics or pricing.

ComponentAWS choiceAzure equivalentGoogle Cloud equivalent
DNS and public entryRoute 53 and Application Load BalancerAzure DNS and Application GatewayCloud DNS and Cloud Load Balancing
Container runtimeAmazon ECS with AWS FargateAzure Container AppsCloud Run
Managed relational databaseAmazon RDS for PostgreSQLAzure Database for PostgreSQLCloud SQL for PostgreSQL
Structured log storage and queriesAmazon CloudWatch LogsAzure Monitor LogsCloud Logging
Application metrics and alarmsAmazon CloudWatchAzure Monitor Metrics and AlertsCloud Monitoring
Distributed tracesAWS X-RayApplication InsightsCloud Trace
External uptime checksCloudWatch SyntheticsAzure Monitor availability testsCloud Monitoring uptime checks
NotificationsAmazon SNSAzure Monitor action groupsCloud Monitoring notification channels
Container image storageAmazon ECRAzure Container RegistryArtifact Registry

A team already committed to a cloud should normally use its managed logging, metrics, tracing, and uptime-check services. The important architecture decision is the signal contract and the operational workflow, not the logo on the dashboard.

5. Data design

Telemetry is operational data, not application business data. Keep it in managed observability services rather than copying it into the application database. The application database remains the source of truth for product records; CloudWatch and X-Ray hold time-bounded diagnostic signals.

Use three logical fact types:

FactGrainMeasures and useful dimensionsStorage and retention approach
Request metricOne service, route template, status class, and time intervalRequest count, error count, latency histogram; dimensions include environment and releaseCloudWatch metrics; use bounded dimensions and a retention policy appropriate to the team’s incident window.
Synthetic runOne canary executionSuccess, duration, failure category; dimensions include check name and environmentCloudWatch Synthetics and its associated logs; retain enough history to review outages and recurring failures.
TraceOne sampled request pathSpan duration, status, service, operation, and parent-child relationshipsX-Ray; sample routine success traffic and retain errors and slow traces at a higher rate where supported.

Logs provide event detail; traces provide the path through a request; metrics provide an inexpensive way to detect changes over time. Do not try to use logs as a high-cardinality metric store or metrics as a substitute for request-level context.

If longer-term analysis becomes a real requirement, archive selected, redacted logs to S3 using date-based prefixes such as year=2026/month=10/day=31/. Add a query layer only when there is a concrete retention, audit, or analysis need. For this workload, a warehouse or lake pipeline would add more operations than value.

Make the telemetry schema predictable:

  • service.name, deployment.environment, and service.version identify the running service.
  • trace_id and span_id connect log events to trace views.
  • http.route is a route template such as /orders/{id}, never a raw URL.
  • http.response.status_code and a bounded status class distinguish success from failure.
  • User identifiers, credentials, request bodies, and other personal data are excluded unless a reviewed, specific requirement says otherwise.

6. Low-level design

The application has one telemetry setup shared by all routes. Instrument the ASP.NET Core server and outbound HTTP calls, and attach service identity once at startup. Database instrumentation can be added if it is supported by the chosen database client and verified to avoid sensitive query capture.

classDiagram
    class WebApplication
    class RequestPipeline
    class HealthEndpoints
    class StructuredLogger
    class OpenTelemetryProvider
    class ADOTCollector
    class CloudWatch
    class XRay
    WebApplication --> RequestPipeline
    WebApplication --> HealthEndpoints
    RequestPipeline --> StructuredLogger
    RequestPipeline --> OpenTelemetryProvider
    OpenTelemetryProvider --> ADOTCollector
    ADOTCollector --> CloudWatch
    ADOTCollector --> XRay

The application owns instrumentation; the collector exports signals to managed backends.

Minimal runnable example

Create an ASP.NET Core minimal web app with dotnet new web, replace Program.cs with the following, then run dotnet run. The health endpoint is intentionally independent of PostgreSQL so it can report whether the process is alive.

var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddJsonConsole();

var app = builder.Build();

app.MapGet("/", (ILogger<Program> logger) =>
{
    logger.LogInformation("Request completed for {Route}", "/");
    return Results.Ok(new { status = "ok" });
});

app.MapGet("/health/live", () => Results.Ok(new { status = "alive" }));

app.Run();

Notice that the log message uses named properties, so the route remains a queryable field rather than being embedded in an unstructured sentence.

Production-oriented telemetry setup

Add the OpenTelemetry.Extensions.Hosting, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http, and OpenTelemetry.Exporter.OpenTelemetryProtocol packages. Run an ADOT Collector sidecar in the same ECS task and configure it to accept OTLP over gRPC on port 4317, then export traces to X-Ray and metrics to CloudWatch. The exact collector image and IAM permissions belong in the task definition and should be tested in the target account.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;

var builder = WebApplication.CreateBuilder(args);
var serviceVersion = Environment.GetEnvironmentVariable("RELEASE_SHA") ?? "local";

builder.Logging.AddJsonConsole();

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "small-app",
            serviceVersion: serviceVersion)
        .AddAttributes(new[]
        {
            new KeyValuePair<string, object>(
                "deployment.environment",
                Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "unknown")
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            options.Filter = context =>
                !context.Request.Path.StartsWithSegments("/health/live");
        })
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri(
                Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
                ?? "http://127.0.0.1:4317");
        }))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri(
                Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
                ?? "http://127.0.0.1:4317");
        }));

var app = builder.Build();

app.MapGet("/", (ILogger<Program> logger, Activity? activity) =>
{
    logger.LogInformation(
        "Serving home page for {Route}; trace {TraceId}",
        "/",
        activity?.TraceId.ToString());

    return Results.Ok(new { status = "ok" });
});

app.MapGet("/health/live", () => Results.Ok(new { status = "alive" }));
app.Run();

Notice that release and environment are resource attributes, while the health probe is excluded from traces to avoid filling the trace view with routine noise.

For structured logs in production, emit fields that answer operational questions. A JSON console formatter is sufficient when the ECS log driver forwards container output to CloudWatch Logs.

{
  "Timestamp": "2026-10-31T12:00:00Z",
  "LogLevel": "Error",
  "Category": "SmallApp.Orders",
  "Message": "Order lookup failed",
  "TraceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "Route": "/orders/{id}",
  "StatusCode": 500,
  "Release": "a1b2c3d",
  "ExceptionType": "NpgsqlException"
}

Never log access tokens, passwords, full payment details, raw request bodies, or unreviewed personal data. A trace ID is useful diagnostic context; it is not a user identifier.

7. Key flows

The normal request flow creates one trace that can connect the load balancer, application work, and instrumented dependencies. The app should return an appropriate response even if telemetry export is temporarily unavailable; telemetry is not a synchronous dependency of user requests.

sequenceDiagram
    participant U as User
    participant A as ALB
    participant S as ECS app
    participant D as RDS PostgreSQL
    participant C as ADOT Collector
    participant X as X-Ray
    U->>A: HTTPS request
    A->>S: Forward request
    S->>D: Execute query
    D-->>S: Return result
    S-->>A: HTTP response
    S->>C: Export trace and metrics
    C->>X: Export sampled trace
    A-->>U: Return response

Telemetry export follows request processing and must not become a reason to fail a successful user operation.

The external check should exercise a stable endpoint that proves the application is ready to serve. A failed canary pages only after a defined failure window, avoiding a notification for a single transient probe failure.

sequenceDiagram
    participant Y as Synthetics canary
    participant R as Route 53
    participant A as ALB
    participant S as ECS app
    participant M as CloudWatch
    participant N as SNS
    Y->>R: Resolve production hostname
    Y->>A: Request HTTPS readiness URL
    A->>S: Forward check
    S-->>Y: Return status and response
    Y->>M: Publish canary result
    M->>M: Evaluate alarm window
    M->>N: Notify on sustained failure

The canary observes the public route and the alarm applies a failure window before notifying the team.

8. Build and release

Infrastructure changes should be reviewed and applied through the same version-controlled workflow as application changes. This Terraform fragment shows the operational resources that are easy to forget: bounded log retention and an alarm for load-balancer server errors. It assumes the ALB and SNS topic are created elsewhere in the same configuration.

resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/small-app"
  retention_in_days = 30
}

resource "aws_cloudwatch_metric_alarm" "alb_5xx" {
  alarm_name          = "small-app-alb-5xx"
  namespace           = "AWS/ApplicationELB"
  metric_name         = "HTTPCode_ELB_5XX_Count"
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 2
  threshold           = 5
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"

  dimensions = {
    LoadBalancer = var.alb_cloudwatch_dimension
  }

  alarm_actions = [aws_sns_topic.on_call.arn]
}

Treat the threshold as a starting assumption. A low-traffic app can produce misleading rate-based alerts, so use a minimum error count and compare it with the canary and application error metrics before paging.

A minimal GitHub Actions release pipeline can test, build, push, and deploy an image. Configure AWS access through GitHub’s OIDC federation rather than long-lived access keys, and provide repository, cluster, and service names as protected workflow variables.

name: deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  test-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"

      - run: dotnet test --configuration Release

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push image
        env:
          REGISTRY: ${{ vars.ECR_REGISTRY }}
          REPOSITORY: ${{ vars.ECR_REPOSITORY }}
          SHA: ${{ github.sha }}
        run: |
          docker build -t "$REGISTRY/$REPOSITORY:$SHA" .
          docker push "$REGISTRY/$REPOSITORY:$SHA"

      - name: Deploy and wait
        env:
          IMAGE: ${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:${{ github.sha }}
        run: |
          aws ecs update-service \
            --cluster "${{ vars.ECS_CLUSTER }}" \
            --service "${{ vars.ECS_SERVICE }}" \
            --force-new-deployment
          aws ecs wait services-stable \
            --cluster "${{ vars.ECS_CLUSTER }}" \
            --services "${{ vars.ECS_SERVICE }}"

In a real pipeline, pass the immutable image digest into a rendered task definition rather than relying on a mutable tag. Add a deployment smoke test after ECS stabilizes; a successful deployment command does not prove that a user-facing route works.

Use a test strategy proportional to the risks:

  • Unit: Validate business rules and log-field construction without AWS dependencies.
  • Integration: Run the app against a disposable PostgreSQL instance and verify health, migrations, and representative routes.
  • Contract: Check the health endpoint response and any external API contracts that the app depends on.
  • Load: Exercise the expected burst assumption and observe latency, database connections, task CPU, and memory. The goal is to find obvious saturation, not to publish a benchmark.

9. Observability

What to emit

A useful log event answers what happened, where it happened, and how to find the related request. Include severity, service, environment, release, route template, status code, and trace ID. Add an error category or dependency name when it helps distinguish likely causes. Keep messages short and fields stable.

Use the RED method for the request-serving service:

SignalSmall-app measures
RateRequests per minute, by service and bounded route template.
ErrorsHTTP 5xx count and ratio; distinguish expected client errors from service failures.
DurationLatency histogram or percentiles for server requests; inspect slow routes rather than only a global average.

Use USE for constrained resources:

ResourceUtilizationSaturationErrors
Fargate taskCPU and memory utilizationPending tasks or repeated scale-out pressureContainer exits and failed health checks
RDS PostgreSQLCPU, storage, and connectionsConnection exhaustion and elevated query latencyDatabase restarts, failed connections, storage alarms
ALBRequest volume and target response timeUnhealthy targets or connection pressureLoad-balancer and target 5xx responses

Distributed tracing with OpenTelemetry

The application code in section 6 uses the OpenTelemetry .NET SDK and OTLP export to the local collector. The ADOT Collector should run as a sidecar in the ECS task, with a task role limited to the required X-Ray and CloudWatch permissions. Keep collector failures out of the user request path, and monitor collector export errors as an operational signal.

Sampling should reflect the diagnostic need and the budget. Start with a modest sampling rate for successful requests, then preserve error and slow-request traces where the sampling configuration supports it. Do not make a high-cardinality trace attribute out of user input. For a low-volume service, retaining more traces can be reasonable, but verify the resulting cost and sensitive-data exposure.

One useful dashboard

Build one CloudWatch dashboard that answers “is the service healthy, and where should I look?” Arrange it from user-visible symptoms toward likely causes:

  1. Top row: Synthetics success and duration, request count, 5xx count, and latency.
  2. Middle row: ALB target health and target response time, ECS CPU and memory, and RDS connections and CPU.
  3. Bottom row: Recent deployment marker, log query for errors grouped by route and release, and a link to the X-Ray service map or relevant traces.

Avoid filling the dashboard with every available metric. A panel earns its place if it helps confirm impact, localize a fault, or check recovery. Keep a saved CloudWatch Logs Insights query for recent errors; include TraceId, route, release, and exception type in the result.

SLI and SLO

These targets are explicit product assumptions for a small service, not a guarantee supplied by AWS. Review them against user expectations and observed traffic.

SLIInitial SLOMeasurement and response
Public availability99.5% per calendar monthSuccessful external canary runs divided by valid runs; investigate sustained failures.
Successful server requestsAt least 99% of non-health requests return below 500 per rolling 30 daysUse application request metrics; exclude expected 4xx responses.
Interactive latency95% of successful requests below 1 second over a rolling 7-day windowUse a latency histogram and inspect route-level outliers.

A low-traffic service needs care when interpreting percentages. One failed request can distort a short window, while a five-minute canary can miss a brief outage. Report both the windowed SLI and the raw event count, and use longer windows for objective evaluation.

Alerts worth paging on

Page only when a human action is likely to reduce user impact:

  • External canary failure sustained across multiple runs.
  • A meaningful increase in 5xx responses that aligns with user-visible failure.
  • All targets unhealthy or the service unable to maintain a healthy task.
  • Database connection exhaustion, storage nearing its configured limit, or persistent database unavailability.

Use ticket or chat notifications for non-urgent conditions such as a gradual rise in latency, a single task restart, or an approaching retention limit. Avoid paging on CPU alone: high utilization is a symptom, not proof of user impact. Every alert needs an owner, a runbook link, and a clear first action.

10. Security and compliance

Observability data can contain sensitive information, so treat it as production data.

  • Grant the application task role permission to write only to its required log group and telemetry destinations. Keep deployment permissions separate from runtime permissions.
  • Restrict access to CloudWatch Logs, dashboards, and traces through least-privilege IAM roles. Review cross-account access explicitly.
  • Encrypt managed storage using service defaults or approved customer-managed keys where policy requires them.
  • Redact secrets and personal data before emitting logs or span attributes. Avoid logging full URLs when query strings may contain tokens.
  • Set log retention deliberately. A 30-day default is an assumption, not a compliance policy; align it with the organization’s retention and deletion requirements.
  • Keep RDS in private subnets, require encrypted connections, and store credentials in AWS Secrets Manager rather than in task definitions or source control.
  • Record who can change alarms, retention, and telemetry access. An accidental dashboard or alarm deletion can be an operational incident.
  • Verify that the canary does not authenticate using a production user account with broad privileges. Prefer a dedicated, low-privilege probe or a public readiness endpoint.

This design does not imply compliance with a particular regulation. Regulatory obligations depend on the data, jurisdiction, contracts, and organizational controls.

11. Cost

The table is an assumption-based monthly planning estimate in USD, not a quote. It assumes one region, modest request volume, one small Fargate service, a small single-AZ RDS PostgreSQL instance, limited log volume, short retention, and a five-minute canary. Actual charges depend on region, task sizing, storage, log ingestion, trace volume, network design, and current service rates. Validate the design with the AWS Pricing Calculator before deployment.

Cost areaPlanning estimate per monthMain cost driver
Fargate application task$10–$30Continuous task size and runtime hours
Application Load Balancer$20–$40Hourly charge and processed traffic
RDS PostgreSQL$20–$50Instance hours, storage, and backups
CloudWatch Logs and metrics$5–$25Ingested log volume, retention, custom metrics
X-Ray tracing$0–$15Traces recorded and retrieved
Synthetics canary and notifications$5–$20Check frequency, runtime, and notification use
Network and data transfer$10–$50NAT gateway, traffic, and topology choices
Indicative total$70–$230Assumption-sensitive; check the regional calculator

The range is intentionally broad because network topology and log volume can dominate a small deployment. A NAT gateway can cost more than the observability signals themselves at this scale. If private-subnet egress is required, include NAT or endpoint charges in the estimate rather than treating them as free infrastructure.

The first cost controls should be boring: set log retention, avoid debug logging in production, keep metric dimensions bounded, and review trace volume after the first month. Do not reduce observability blindly to save a small amount; remove noisy signals before removing the evidence needed to diagnose outages.

12. Failure modes, resilience, and disaster recovery

Failure modeUser effectDetection and response
Canary fails but app is healthyPossible false alarm from DNS, probe, or transient network issueCompare consecutive runs with ALB and application metrics; do not page on one failed sample.
App fails but canary is not configured correctlyUsers see errors without an external alertTest the canary against a deliberate failure in a non-production environment and verify the alarm route.
CloudWatch or collector export is unavailableRequests may succeed while telemetry is delayed or lostKeep exports asynchronous and bounded; alert on sustained collector errors without failing requests.
Database is unavailableReadiness fails and requests depending on data failKeep liveness independent, use short database timeouts, and rely on managed database recovery and backups.
Bad releaseErrors or latency rise after deploymentInclude release identity in signals, run post-deploy smoke checks, and retain a straightforward rollback path.
Logs contain sensitive fieldsIncreased exposure and possible compliance incidentRedact at source, restrict access, set retention, and exercise an incident response process.

Resilience for this scale means avoiding unnecessary dependencies and making recovery procedures testable. ECS can replace unhealthy tasks; RDS backups support database recovery; and the application image plus infrastructure definitions support service recreation. Those mechanisms do not automatically prove recovery time or data recovery point.

Choose and document a recovery point objective (RPO) and recovery time objective (RTO) with the product owner. For a small, single-region app, restoring a database backup may be an acceptable recovery path, but it is a decision that must be tested. Perform a restore exercise, verify that the app can connect to the restored database, and record the actual steps and elapsed time.

Do not claim regional disaster recovery merely because backups exist. A regional failure may require a separate plan for restoring the database, rebuilding infrastructure, changing DNS, and validating secrets and permissions. Add cross-region recovery only when the business impact justifies its cost and operational burden.

13. Evolving to the next scale

Keep the architecture as-is while the team can answer incidents from the dashboard, logs, and traces without excessive manual work. Add complexity in response to a measured limitation.

TriggerChange to considerWhat not to do prematurely
The team needs longer retention or cross-service log analysisArchive selected logs to S3 and add a query layer with defined access and retentionExport every signal indefinitely without a query or compliance requirement
More services make trace navigation difficultStandardize service naming, trace context propagation, and shared dashboard conventionsBuild a custom tracing backend before evaluating managed options
Traffic or latency exceeds one task’s capacityAdd ECS autoscaling based on request load and resource metrics; load-test database limitsScale only from CPU without checking database connections and tail latency
On-call becomes noisyReview alert precision, add ownership and runbook links, and tune evaluation windowsAdd more alerts to compensate for a weak service-level signal
Availability requirements become stricterAdd a tested multi-AZ database and service deployment strategy; revisit SLOs and recovery objectivesPromise high availability based only on a green health endpoint
Several teams need common telemetry conventionsPublish a small instrumentation library and service metadata standardCreate a platform team or telemetry abstraction that a two-person team cannot maintain

Keep the data model stable as the implementation evolves: low-cardinality metrics for trends, structured logs for event detail, and traces for request paths. The backends can change without changing the questions the team needs to answer.

14. Phased implementation plan

Phase 1: Make failures visible

  • Emit JSON logs with stable fields, release identity, and trace correlation.
  • Add separate liveness and readiness endpoints.
  • Set a finite log retention period.
  • Add one external canary against the production hostname.

At the end of this phase, a developer should be able to confirm whether the public endpoint is reachable and find the relevant application error without searching raw text manually.

Phase 2: Connect the request path

  • Add OpenTelemetry server and outbound HTTP instrumentation.
  • Run the ADOT Collector sidecar and verify trace export to X-Ray.
  • Add request rate, error, and duration metrics.
  • Check that application failures still return correctly if telemetry export is interrupted.

At the end of this phase, the team should be able to move from a failed request log to its trace and identify the slow or failing dependency.

Phase 3: Build the operating loop

  • Create the single dashboard described above.
  • Add alarms for sustained public failure, significant 5xx activity, unhealthy targets, and database exhaustion.
  • Route notifications to a monitored SNS destination.
  • Add deployment smoke tests and a short runbook for each paging alert.

At the end of this phase, an alert should tell the on-call engineer what is broken, where to look, and the first safe action.

Phase 4: Review before adding tools

After the first month of production use, review alert quality, telemetry volume, dashboard usefulness, and actual recovery experience. Remove noisy signals; adjust thresholds using observed traffic; and update cost assumptions with real bills. Only then decide whether longer retention, additional availability, or a separate analytics store is worth operating.

The useful baseline is not “all telemetry is collected.” It is that a small team can detect a real user-facing problem, connect it to a request and a release, and recover without guessing.