1. Context and scale
A small application does not need an observability platform assembled from several databases and dashboards. It needs to answer a short list of operational questions quickly:
- Can a user reach the app and complete its main action?
- If a request fails, what failed, and which deployment introduced it?
- Is the problem isolated to the app, the database, or an external dependency?
- Is anyone expected to act on the issue right now?
This blueprint uses a single-region AWS deployment: a .NET web monolith on Amazon ECS with AWS Fargate, an Application Load Balancer, and Amazon RDS for PostgreSQL. Amazon CloudWatch collects logs and metrics; AWS X-Ray stores traces; CloudWatch Synthetics checks the public endpoint from outside the application. The team operates one dashboard and a small number of alerts.
The point is not that AWS is the only reasonable choice. The point is to keep the operational path short: use managed services, emit a few useful signals, and make every alert correspond to an action.
| Assumption | Planning value |
|---|---|
| Daily active users | 500 |
| Typical requests per second | 0.1; bursts up to 5 |
| Application data volume | 5 GB initially, growing slowly |
| Team size | 2 engineers, shared on-call |
| Deployment | One AWS region; one production environment |
| Monthly observability budget | $20–$50 within a broader app budget |
| Availability objective | 99.5% monthly, an explicit product assumption |
These are sizing assumptions, not measured capacity claims. A small workload can still have sharp traffic spikes, slow database queries, or a failed release. Observability should make those conditions visible without creating a second system that the team has to operate.
2. Requirements
Requirements should focus on actions the team can take, not on collecting every possible signal.
Functional requirements
| Requirement | Design response |
|---|---|
| Confirm public availability | Run an external CloudWatch Synthetics canary against the production URL every five minutes. |
| Diagnose failed requests | Emit JSON logs with severity, route template, status, trace ID, and release identifier. |
| Find slow operations | Create distributed traces for incoming HTTP requests and database calls. |
| Spot user-visible degradation | Track request rate, error rate, latency, and canary failures. |
| Relate incidents to releases | Add the same immutable release identifier to logs, metrics, traces, and deployment records. |
| Notify the team | Route actionable CloudWatch alarms to an SNS topic connected to the team’s notification channel. |
Non-functional requirements
| Requirement | Target or constraint |
|---|---|
| Operational simplicity | One application service, one managed database, one dashboard, and a short alert list. |
| Signal cost | Bound log retention and sample routine traces; do not retain unlimited debug output. |
| Diagnostic context | A failed request should be traceable across application logs and traces using a correlation ID. |
| Safe health checks | Liveness checks must not depend on the database; readiness and external checks must have short timeouts. |
| Low cardinality | Never use user IDs, request IDs, or raw URLs as metric dimensions. |
| Recoverability | Rebuild the service from infrastructure and deployment definitions; back up the database independently. |
3. High-level design
The public request path goes through Route 53 and an ALB to the Fargate service. The application writes structured logs to CloudWatch Logs and sends OpenTelemetry data to an AWS Distro for OpenTelemetry (ADOT) Collector sidecar. The collector exports traces to X-Ray and application metrics to CloudWatch. RDS stays private to the application network.
A CloudWatch Synthetics canary uses the public hostname, so it exercises DNS, TLS, load balancing, application startup, and the readiness endpoint. This is deliberately different from an internal health check: an internal check cannot prove the public path works.
flowchart LR
User["User or browser"] --> DNS["Amazon Route 53"]
DNS --> ALB["Application Load Balancer"]
ALB --> App["Amazon ECS on AWS Fargate"]
App --> DB["Amazon RDS for PostgreSQL"]
App --> Logs["Amazon CloudWatch Logs"]
App --> Collector["ADOT Collector sidecar"]
Collector --> XRay["AWS X-Ray"]
Collector --> Metrics["Amazon CloudWatch metrics"]
Canary["CloudWatch Synthetics canary"] --> DNS
Metrics --> Alarm["CloudWatch alarms"]
Logs --> Alarm
Alarm --> Notify["Amazon SNS"]The application emits telemetry at the point of work; the canary tests the path from outside the application.
Keep the first version intentionally small. Do not add a metrics database, log search cluster, or event bus just to gain a second place to query. If a signal cannot help explain a user-visible symptom or support a decision, defer it.
4. Cloud service mapping
AWS is the selected cloud. The other columns show equivalent service categories, not a promise that every product has identical semantics or pricing.
| Component | AWS choice | Azure equivalent | Google Cloud equivalent |
|---|---|---|---|
| DNS and public entry | Route 53 and Application Load Balancer | Azure DNS and Application Gateway | Cloud DNS and Cloud Load Balancing |
| Container runtime | Amazon ECS with AWS Fargate | Azure Container Apps | Cloud Run |
| Managed relational database | Amazon RDS for PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Structured log storage and queries | Amazon CloudWatch Logs | Azure Monitor Logs | Cloud Logging |
| Application metrics and alarms | Amazon CloudWatch | Azure Monitor Metrics and Alerts | Cloud Monitoring |
| Distributed traces | AWS X-Ray | Application Insights | Cloud Trace |
| External uptime checks | CloudWatch Synthetics | Azure Monitor availability tests | Cloud Monitoring uptime checks |
| Notifications | Amazon SNS | Azure Monitor action groups | Cloud Monitoring notification channels |
| Container image storage | Amazon ECR | Azure Container Registry | Artifact Registry |
A team already committed to a cloud should normally use its managed logging, metrics, tracing, and uptime-check services. The important architecture decision is the signal contract and the operational workflow, not the logo on the dashboard.
5. Data design
Telemetry is operational data, not application business data. Keep it in managed observability services rather than copying it into the application database. The application database remains the source of truth for product records; CloudWatch and X-Ray hold time-bounded diagnostic signals.
Use three logical fact types:
| Fact | Grain | Measures and useful dimensions | Storage and retention approach |
|---|---|---|---|
| Request metric | One service, route template, status class, and time interval | Request count, error count, latency histogram; dimensions include environment and release | CloudWatch metrics; use bounded dimensions and a retention policy appropriate to the team’s incident window. |
| Synthetic run | One canary execution | Success, duration, failure category; dimensions include check name and environment | CloudWatch Synthetics and its associated logs; retain enough history to review outages and recurring failures. |
| Trace | One sampled request path | Span duration, status, service, operation, and parent-child relationships | X-Ray; sample routine success traffic and retain errors and slow traces at a higher rate where supported. |
Logs provide event detail; traces provide the path through a request; metrics provide an inexpensive way to detect changes over time. Do not try to use logs as a high-cardinality metric store or metrics as a substitute for request-level context.
If longer-term analysis becomes a real requirement, archive selected, redacted logs to S3 using date-based prefixes such as year=2026/month=10/day=31/. Add a query layer only when there is a concrete retention, audit, or analysis need. For this workload, a warehouse or lake pipeline would add more operations than value.
Make the telemetry schema predictable:
service.name,deployment.environment, andservice.versionidentify the running service.trace_idandspan_idconnect log events to trace views.http.routeis a route template such as/orders/{id}, never a raw URL.http.response.status_codeand a bounded status class distinguish success from failure.- User identifiers, credentials, request bodies, and other personal data are excluded unless a reviewed, specific requirement says otherwise.
6. Low-level design
The application has one telemetry setup shared by all routes. Instrument the ASP.NET Core server and outbound HTTP calls, and attach service identity once at startup. Database instrumentation can be added if it is supported by the chosen database client and verified to avoid sensitive query capture.
classDiagram
class WebApplication
class RequestPipeline
class HealthEndpoints
class StructuredLogger
class OpenTelemetryProvider
class ADOTCollector
class CloudWatch
class XRay
WebApplication --> RequestPipeline
WebApplication --> HealthEndpoints
RequestPipeline --> StructuredLogger
RequestPipeline --> OpenTelemetryProvider
OpenTelemetryProvider --> ADOTCollector
ADOTCollector --> CloudWatch
ADOTCollector --> XRayThe application owns instrumentation; the collector exports signals to managed backends.
Minimal runnable example
Create an ASP.NET Core minimal web app with dotnet new web, replace Program.cs with the following, then run dotnet run. The health endpoint is intentionally independent of PostgreSQL so it can report whether the process is alive.
var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddJsonConsole();
var app = builder.Build();
app.MapGet("/", (ILogger<Program> logger) =>
{
logger.LogInformation("Request completed for {Route}", "/");
return Results.Ok(new { status = "ok" });
});
app.MapGet("/health/live", () => Results.Ok(new { status = "alive" }));
app.Run();
Notice that the log message uses named properties, so the route remains a queryable field rather than being embedded in an unstructured sentence.
Production-oriented telemetry setup
Add the OpenTelemetry.Extensions.Hosting, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http, and OpenTelemetry.Exporter.OpenTelemetryProtocol packages. Run an ADOT Collector sidecar in the same ECS task and configure it to accept OTLP over gRPC on port 4317, then export traces to X-Ray and metrics to CloudWatch. The exact collector image and IAM permissions belong in the task definition and should be tested in the target account.
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
var builder = WebApplication.CreateBuilder(args);
var serviceVersion = Environment.GetEnvironmentVariable("RELEASE_SHA") ?? "local";
builder.Logging.AddJsonConsole();
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(
serviceName: "small-app",
serviceVersion: serviceVersion)
.AddAttributes(new[]
{
new KeyValuePair<string, object>(
"deployment.environment",
Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "unknown")
}))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation(options =>
{
options.Filter = context =>
!context.Request.Path.StartsWithSegments("/health/live");
})
.AddHttpClientInstrumentation()
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(
Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
?? "http://127.0.0.1:4317");
}))
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(
Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT")
?? "http://127.0.0.1:4317");
}));
var app = builder.Build();
app.MapGet("/", (ILogger<Program> logger, Activity? activity) =>
{
logger.LogInformation(
"Serving home page for {Route}; trace {TraceId}",
"/",
activity?.TraceId.ToString());
return Results.Ok(new { status = "ok" });
});
app.MapGet("/health/live", () => Results.Ok(new { status = "alive" }));
app.Run();
Notice that release and environment are resource attributes, while the health probe is excluded from traces to avoid filling the trace view with routine noise.
For structured logs in production, emit fields that answer operational questions. A JSON console formatter is sufficient when the ECS log driver forwards container output to CloudWatch Logs.
{
"Timestamp": "2026-10-31T12:00:00Z",
"LogLevel": "Error",
"Category": "SmallApp.Orders",
"Message": "Order lookup failed",
"TraceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"Route": "/orders/{id}",
"StatusCode": 500,
"Release": "a1b2c3d",
"ExceptionType": "NpgsqlException"
}
Never log access tokens, passwords, full payment details, raw request bodies, or unreviewed personal data. A trace ID is useful diagnostic context; it is not a user identifier.
7. Key flows
The normal request flow creates one trace that can connect the load balancer, application work, and instrumented dependencies. The app should return an appropriate response even if telemetry export is temporarily unavailable; telemetry is not a synchronous dependency of user requests.
sequenceDiagram
participant U as User
participant A as ALB
participant S as ECS app
participant D as RDS PostgreSQL
participant C as ADOT Collector
participant X as X-Ray
U->>A: HTTPS request
A->>S: Forward request
S->>D: Execute query
D-->>S: Return result
S-->>A: HTTP response
S->>C: Export trace and metrics
C->>X: Export sampled trace
A-->>U: Return responseTelemetry export follows request processing and must not become a reason to fail a successful user operation.
The external check should exercise a stable endpoint that proves the application is ready to serve. A failed canary pages only after a defined failure window, avoiding a notification for a single transient probe failure.
sequenceDiagram
participant Y as Synthetics canary
participant R as Route 53
participant A as ALB
participant S as ECS app
participant M as CloudWatch
participant N as SNS
Y->>R: Resolve production hostname
Y->>A: Request HTTPS readiness URL
A->>S: Forward check
S-->>Y: Return status and response
Y->>M: Publish canary result
M->>M: Evaluate alarm window
M->>N: Notify on sustained failureThe canary observes the public route and the alarm applies a failure window before notifying the team.
8. Build and release
Infrastructure changes should be reviewed and applied through the same version-controlled workflow as application changes. This Terraform fragment shows the operational resources that are easy to forget: bounded log retention and an alarm for load-balancer server errors. It assumes the ALB and SNS topic are created elsewhere in the same configuration.
resource "aws_cloudwatch_log_group" "app" {
name = "/ecs/small-app"
retention_in_days = 30
}
resource "aws_cloudwatch_metric_alarm" "alb_5xx" {
alarm_name = "small-app-alb-5xx"
namespace = "AWS/ApplicationELB"
metric_name = "HTTPCode_ELB_5XX_Count"
statistic = "Sum"
period = 300
evaluation_periods = 2
threshold = 5
comparison_operator = "GreaterThanOrEqualToThreshold"
treat_missing_data = "notBreaching"
dimensions = {
LoadBalancer = var.alb_cloudwatch_dimension
}
alarm_actions = [aws_sns_topic.on_call.arn]
}
Treat the threshold as a starting assumption. A low-traffic app can produce misleading rate-based alerts, so use a minimum error count and compare it with the canary and application error metrics before paging.
A minimal GitHub Actions release pipeline can test, build, push, and deploy an image. Configure AWS access through GitHub’s OIDC federation rather than long-lived access keys, and provide repository, cluster, and service names as protected workflow variables.
name: deploy
on:
push:
branches: [main]
permissions:
id-token: write
contents: read
jobs:
test-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.0.x"
- run: dotnet test --configuration Release
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.AWS_DEPLOY_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- uses: aws-actions/amazon-ecr-login@v2
- name: Build and push image
env:
REGISTRY: ${{ vars.ECR_REGISTRY }}
REPOSITORY: ${{ vars.ECR_REPOSITORY }}
SHA: ${{ github.sha }}
run: |
docker build -t "$REGISTRY/$REPOSITORY:$SHA" .
docker push "$REGISTRY/$REPOSITORY:$SHA"
- name: Deploy and wait
env:
IMAGE: ${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:${{ github.sha }}
run: |
aws ecs update-service \
--cluster "${{ vars.ECS_CLUSTER }}" \
--service "${{ vars.ECS_SERVICE }}" \
--force-new-deployment
aws ecs wait services-stable \
--cluster "${{ vars.ECS_CLUSTER }}" \
--services "${{ vars.ECS_SERVICE }}"
In a real pipeline, pass the immutable image digest into a rendered task definition rather than relying on a mutable tag. Add a deployment smoke test after ECS stabilizes; a successful deployment command does not prove that a user-facing route works.
Use a test strategy proportional to the risks:
- Unit: Validate business rules and log-field construction without AWS dependencies.
- Integration: Run the app against a disposable PostgreSQL instance and verify health, migrations, and representative routes.
- Contract: Check the health endpoint response and any external API contracts that the app depends on.
- Load: Exercise the expected burst assumption and observe latency, database connections, task CPU, and memory. The goal is to find obvious saturation, not to publish a benchmark.
9. Observability
What to emit
A useful log event answers what happened, where it happened, and how to find the related request. Include severity, service, environment, release, route template, status code, and trace ID. Add an error category or dependency name when it helps distinguish likely causes. Keep messages short and fields stable.
Use the RED method for the request-serving service:
| Signal | Small-app measures |
|---|---|
| Rate | Requests per minute, by service and bounded route template. |
| Errors | HTTP 5xx count and ratio; distinguish expected client errors from service failures. |
| Duration | Latency histogram or percentiles for server requests; inspect slow routes rather than only a global average. |
Use USE for constrained resources:
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| Fargate task | CPU and memory utilization | Pending tasks or repeated scale-out pressure | Container exits and failed health checks |
| RDS PostgreSQL | CPU, storage, and connections | Connection exhaustion and elevated query latency | Database restarts, failed connections, storage alarms |
| ALB | Request volume and target response time | Unhealthy targets or connection pressure | Load-balancer and target 5xx responses |
Distributed tracing with OpenTelemetry
The application code in section 6 uses the OpenTelemetry .NET SDK and OTLP export to the local collector. The ADOT Collector should run as a sidecar in the ECS task, with a task role limited to the required X-Ray and CloudWatch permissions. Keep collector failures out of the user request path, and monitor collector export errors as an operational signal.
Sampling should reflect the diagnostic need and the budget. Start with a modest sampling rate for successful requests, then preserve error and slow-request traces where the sampling configuration supports it. Do not make a high-cardinality trace attribute out of user input. For a low-volume service, retaining more traces can be reasonable, but verify the resulting cost and sensitive-data exposure.
One useful dashboard
Build one CloudWatch dashboard that answers “is the service healthy, and where should I look?” Arrange it from user-visible symptoms toward likely causes:
- Top row: Synthetics success and duration, request count, 5xx count, and latency.
- Middle row: ALB target health and target response time, ECS CPU and memory, and RDS connections and CPU.
- Bottom row: Recent deployment marker, log query for errors grouped by route and release, and a link to the X-Ray service map or relevant traces.
Avoid filling the dashboard with every available metric. A panel earns its place if it helps confirm impact, localize a fault, or check recovery. Keep a saved CloudWatch Logs Insights query for recent errors; include TraceId, route, release, and exception type in the result.
SLI and SLO
These targets are explicit product assumptions for a small service, not a guarantee supplied by AWS. Review them against user expectations and observed traffic.
| SLI | Initial SLO | Measurement and response |
|---|---|---|
| Public availability | 99.5% per calendar month | Successful external canary runs divided by valid runs; investigate sustained failures. |
| Successful server requests | At least 99% of non-health requests return below 500 per rolling 30 days | Use application request metrics; exclude expected 4xx responses. |
| Interactive latency | 95% of successful requests below 1 second over a rolling 7-day window | Use a latency histogram and inspect route-level outliers. |
A low-traffic service needs care when interpreting percentages. One failed request can distort a short window, while a five-minute canary can miss a brief outage. Report both the windowed SLI and the raw event count, and use longer windows for objective evaluation.
Alerts worth paging on
Page only when a human action is likely to reduce user impact:
- External canary failure sustained across multiple runs.
- A meaningful increase in 5xx responses that aligns with user-visible failure.
- All targets unhealthy or the service unable to maintain a healthy task.
- Database connection exhaustion, storage nearing its configured limit, or persistent database unavailability.
Use ticket or chat notifications for non-urgent conditions such as a gradual rise in latency, a single task restart, or an approaching retention limit. Avoid paging on CPU alone: high utilization is a symptom, not proof of user impact. Every alert needs an owner, a runbook link, and a clear first action.
10. Security and compliance
Observability data can contain sensitive information, so treat it as production data.
- Grant the application task role permission to write only to its required log group and telemetry destinations. Keep deployment permissions separate from runtime permissions.
- Restrict access to CloudWatch Logs, dashboards, and traces through least-privilege IAM roles. Review cross-account access explicitly.
- Encrypt managed storage using service defaults or approved customer-managed keys where policy requires them.
- Redact secrets and personal data before emitting logs or span attributes. Avoid logging full URLs when query strings may contain tokens.
- Set log retention deliberately. A 30-day default is an assumption, not a compliance policy; align it with the organization’s retention and deletion requirements.
- Keep RDS in private subnets, require encrypted connections, and store credentials in AWS Secrets Manager rather than in task definitions or source control.
- Record who can change alarms, retention, and telemetry access. An accidental dashboard or alarm deletion can be an operational incident.
- Verify that the canary does not authenticate using a production user account with broad privileges. Prefer a dedicated, low-privilege probe or a public readiness endpoint.
This design does not imply compliance with a particular regulation. Regulatory obligations depend on the data, jurisdiction, contracts, and organizational controls.
11. Cost
The table is an assumption-based monthly planning estimate in USD, not a quote. It assumes one region, modest request volume, one small Fargate service, a small single-AZ RDS PostgreSQL instance, limited log volume, short retention, and a five-minute canary. Actual charges depend on region, task sizing, storage, log ingestion, trace volume, network design, and current service rates. Validate the design with the AWS Pricing Calculator before deployment.
| Cost area | Planning estimate per month | Main cost driver |
|---|---|---|
| Fargate application task | $10–$30 | Continuous task size and runtime hours |
| Application Load Balancer | $20–$40 | Hourly charge and processed traffic |
| RDS PostgreSQL | $20–$50 | Instance hours, storage, and backups |
| CloudWatch Logs and metrics | $5–$25 | Ingested log volume, retention, custom metrics |
| X-Ray tracing | $0–$15 | Traces recorded and retrieved |
| Synthetics canary and notifications | $5–$20 | Check frequency, runtime, and notification use |
| Network and data transfer | $10–$50 | NAT gateway, traffic, and topology choices |
| Indicative total | $70–$230 | Assumption-sensitive; check the regional calculator |
The range is intentionally broad because network topology and log volume can dominate a small deployment. A NAT gateway can cost more than the observability signals themselves at this scale. If private-subnet egress is required, include NAT or endpoint charges in the estimate rather than treating them as free infrastructure.
The first cost controls should be boring: set log retention, avoid debug logging in production, keep metric dimensions bounded, and review trace volume after the first month. Do not reduce observability blindly to save a small amount; remove noisy signals before removing the evidence needed to diagnose outages.
12. Failure modes, resilience, and disaster recovery
| Failure mode | User effect | Detection and response |
|---|---|---|
| Canary fails but app is healthy | Possible false alarm from DNS, probe, or transient network issue | Compare consecutive runs with ALB and application metrics; do not page on one failed sample. |
| App fails but canary is not configured correctly | Users see errors without an external alert | Test the canary against a deliberate failure in a non-production environment and verify the alarm route. |
| CloudWatch or collector export is unavailable | Requests may succeed while telemetry is delayed or lost | Keep exports asynchronous and bounded; alert on sustained collector errors without failing requests. |
| Database is unavailable | Readiness fails and requests depending on data fail | Keep liveness independent, use short database timeouts, and rely on managed database recovery and backups. |
| Bad release | Errors or latency rise after deployment | Include release identity in signals, run post-deploy smoke checks, and retain a straightforward rollback path. |
| Logs contain sensitive fields | Increased exposure and possible compliance incident | Redact at source, restrict access, set retention, and exercise an incident response process. |
Resilience for this scale means avoiding unnecessary dependencies and making recovery procedures testable. ECS can replace unhealthy tasks; RDS backups support database recovery; and the application image plus infrastructure definitions support service recreation. Those mechanisms do not automatically prove recovery time or data recovery point.
Choose and document a recovery point objective (RPO) and recovery time objective (RTO) with the product owner. For a small, single-region app, restoring a database backup may be an acceptable recovery path, but it is a decision that must be tested. Perform a restore exercise, verify that the app can connect to the restored database, and record the actual steps and elapsed time.
Do not claim regional disaster recovery merely because backups exist. A regional failure may require a separate plan for restoring the database, rebuilding infrastructure, changing DNS, and validating secrets and permissions. Add cross-region recovery only when the business impact justifies its cost and operational burden.
13. Evolving to the next scale
Keep the architecture as-is while the team can answer incidents from the dashboard, logs, and traces without excessive manual work. Add complexity in response to a measured limitation.
| Trigger | Change to consider | What not to do prematurely |
|---|---|---|
| The team needs longer retention or cross-service log analysis | Archive selected logs to S3 and add a query layer with defined access and retention | Export every signal indefinitely without a query or compliance requirement |
| More services make trace navigation difficult | Standardize service naming, trace context propagation, and shared dashboard conventions | Build a custom tracing backend before evaluating managed options |
| Traffic or latency exceeds one task’s capacity | Add ECS autoscaling based on request load and resource metrics; load-test database limits | Scale only from CPU without checking database connections and tail latency |
| On-call becomes noisy | Review alert precision, add ownership and runbook links, and tune evaluation windows | Add more alerts to compensate for a weak service-level signal |
| Availability requirements become stricter | Add a tested multi-AZ database and service deployment strategy; revisit SLOs and recovery objectives | Promise high availability based only on a green health endpoint |
| Several teams need common telemetry conventions | Publish a small instrumentation library and service metadata standard | Create a platform team or telemetry abstraction that a two-person team cannot maintain |
Keep the data model stable as the implementation evolves: low-cardinality metrics for trends, structured logs for event detail, and traces for request paths. The backends can change without changing the questions the team needs to answer.
14. Phased implementation plan
Phase 1: Make failures visible
- Emit JSON logs with stable fields, release identity, and trace correlation.
- Add separate liveness and readiness endpoints.
- Set a finite log retention period.
- Add one external canary against the production hostname.
At the end of this phase, a developer should be able to confirm whether the public endpoint is reachable and find the relevant application error without searching raw text manually.
Phase 2: Connect the request path
- Add OpenTelemetry server and outbound HTTP instrumentation.
- Run the ADOT Collector sidecar and verify trace export to X-Ray.
- Add request rate, error, and duration metrics.
- Check that application failures still return correctly if telemetry export is interrupted.
At the end of this phase, the team should be able to move from a failed request log to its trace and identify the slow or failing dependency.
Phase 3: Build the operating loop
- Create the single dashboard described above.
- Add alarms for sustained public failure, significant 5xx activity, unhealthy targets, and database exhaustion.
- Route notifications to a monitored SNS destination.
- Add deployment smoke tests and a short runbook for each paging alert.
At the end of this phase, an alert should tell the on-call engineer what is broken, where to look, and the first safe action.
Phase 4: Review before adding tools
After the first month of production use, review alert quality, telemetry volume, dashboard usefulness, and actual recovery experience. Remove noisy signals; adjust thresholds using observed traffic; and update cost assumptions with real bills. Only then decide whether longer retention, additional availability, or a separate analytics store is worth operating.
The useful baseline is not “all telemetry is collected.” It is that a small team can detect a real user-facing problem, connect it to a request and a release, and recover without guessing.