1) Context and scale
This blueprint deploys the Grafana LGTM stack—Loki for logs, Grafana for exploration and dashboards, Tempo for traces, and Mimir for metrics—on AWS EKS. It is for a small product organization that needs to correlate application signals without adopting a separate managed observability product for each signal.
The application is a handful of services, including a modular monolith, running on Kubernetes. It uses a multi-AZ database, a cache, and a queue. Application traffic is bursty, deployments happen through CI/CD, and a staging environment is required. The observability platform is an internal production service: if it becomes unavailable, the customer-facing application should keep working.
The design uses EKS for compute and S3 for durable signal storage. Grafana, Loki, Tempo, and Mimir run as workloads in Kubernetes; they are not AWS-managed services. The initial deployment is single-region and multi-AZ, with three separate S3 buckets and workload identities that grant each backend access only to its own bucket.
| Assumption | Planning value |
|---|---|
| Daily active users | 50,000 |
| Application traffic | 250 requests/s peak; 60 requests/s average |
| Telemetry volume | 40 GB/day compressed, across logs, traces, and metrics |
| Retention | Logs 30 days; traces 7 days; metrics 13 months |
| Engineering team | 2 product teams; 1 platform engineer shares ownership |
| Budget | $2,000–$4,000/month for the observability platform, excluding application compute |
The initial topology is deliberately modest: one EKS cluster, one region, multi-AZ node groups, and replicated backend components where a single pod would be an unacceptable failure point. It does not attempt multi-region active-active telemetry ingestion. That adds operational and cost complexity before the expected scale demands it.
2) Requirements
Functional requirements define what the platform must let engineers do. Non-functional requirements bound how it behaves during normal operation and failure.
| Functional requirement | Acceptance criterion |
|---|---|
| Ingest application traces | Services export OTLP traces through an in-cluster collector gateway. |
| Ingest application and Kubernetes logs | Collectors attach stable resource metadata and send logs to Loki. |
| Ingest platform and application metrics | Scrape Kubernetes targets and accept application metrics into Mimir. |
| Correlate signals | Grafana links a trace to logs and metrics using service, environment, and trace identifiers. |
| Alert on service health | SLO and infrastructure alerts route through Alertmanager to the team’s paging system. |
| Explore telemetry | Engineers can search logs, query metrics, and inspect traces through authenticated Grafana access. |
| Operate staging safely | Staging telemetry is separated from production by namespace, storage bucket, credentials, and dashboard conventions. |
| Non-functional requirement | Target |
|---|---|
| Availability | 99.9% monthly availability for Grafana query and ingestion paths, excluding planned maintenance. |
| Ingestion durability | A collector retry buffer absorbs short backend disruptions; durable object storage retains accepted backend data. |
| Query latency | Common dashboard queries complete in under 10 seconds at the assumed scale; this is a design target to validate, not a guarantee. |
| Isolation | Telemetry storage and credentials are separate from application data and from staging. |
| Security | Private cluster access for internal components, TLS at ingress and between external clients and endpoints, least-privilege workload identity. |
| Operability | Backend health, ingestion lag, capacity, and query failures are visible independently of the stack being monitored. |
| Cost control | Retention, sampling, label policy, and ingestion limits are explicit and reviewed. |
For the first release, prioritize a reliable ingestion path and useful dashboards over elaborate tenancy. A single production tenant is sufficient for two teams if access is controlled at Grafana and write access is limited to trusted collectors. Add tenant isolation when teams have materially different access or retention requirements.
3) High-level design
Applications export traces and metrics using OpenTelemetry. Grafana Alloy runs as a DaemonSet for Kubernetes and node collection, and as a gateway Deployment for OTLP intake and processing. Alloy sends logs to Loki, metrics to Mimir, and traces to Tempo. Grafana reads from all three backends. The backends use S3 for durable storage; their short-lived working state and caches run in EKS.
flowchart LR
app["Application pods with OpenTelemetry"] --> alloygw["Alloy OTLP gateway"]
nodes["Kubernetes nodes and pod logs"] --> alloyds["Alloy DaemonSet"]
alloyds --> loki["Loki on EKS"]
alloygw --> loki
alloygw --> mimir["Mimir on EKS"]
alloygw --> tempo["Tempo on EKS"]
grafana["Grafana on EKS"] --> loki
grafana --> mimir
grafana --> tempo
loki --> s3logs["Amazon S3 logs bucket"]
mimir --> s3metrics["Amazon S3 metrics bucket"]
tempo --> s3traces["Amazon S3 traces bucket"]
alb["AWS Application Load Balancer"] --> grafana
grafana --> albCaption: Applications and collectors write to the backends; Grafana queries them, and S3 holds durable signal data.
The public-facing surface should be small. An AWS Application Load Balancer exposes Grafana through an internal or tightly restricted ingress. Loki, Mimir, and Tempo endpoints remain cluster-internal. If an external collector must write to the platform, expose a separate authenticated OTLP endpoint rather than exposing backend APIs.
Use EKS Pod Identity or IAM Roles for Service Accounts to grant each backend access to its own S3 bucket. Encrypt buckets with AWS KMS, keep backend-to-backend traffic on the cluster network, and use network policies to limit who can reach write and query endpoints.
4) Cloud service mapping
The AWS column is the chosen implementation. Azure and GCP equivalents are infrastructure mappings, not drop-in replacements for LGTM. Each cloud’s native observability products have different query languages, data models, and operational characteristics.
| Component or responsibility | AWS choice | Azure equivalent | GCP equivalent |
|---|---|---|---|
| Kubernetes control plane | Amazon EKS | Azure Kubernetes Service (AKS) | Google Kubernetes Engine (GKE) |
| Log, trace, and metric backends | Loki, Tempo, and Mimir on EKS | Same open-source components on AKS | Same open-source components on GKE |
| Grafana UI | Grafana on EKS | Grafana on AKS | Grafana on GKE |
| Durable backend storage | Amazon S3, one bucket per backend | Azure Blob Storage | Google Cloud Storage |
| Workload identity | EKS Pod Identity or IAM Roles for Service Accounts | Microsoft Entra Workload ID | Workload Identity Federation for GKE |
| Encryption keys | AWS Key Management Service | Azure Key Vault | Cloud Key Management Service |
| Ingress and TLS | AWS Load Balancer Controller and Application Load Balancer | Application Gateway Ingress Controller or Azure Load Balancer | GKE Ingress or Gateway |
| DNS | Amazon Route 53 | Azure DNS | Cloud DNS |
| Secrets | AWS Secrets Manager or Kubernetes secrets backed by an external secrets integration | Azure Key Vault | Secret Manager |
| Native alternative | Amazon CloudWatch and AWS X-Ray; different product model | Azure Monitor and Application Insights; different product model | Cloud Logging, Cloud Monitoring, and Cloud Trace; different product model |
Do not expose S3 credentials as static Kubernetes secrets. Use a workload identity bound to each backend’s service account and a narrowly scoped bucket policy. The same principle applies to the equivalent identity systems on Azure and GCP.
5) Data design
LGTM stores different signal types in purpose-built backends; it is not a relational data model. Treat each backend’s object storage as a separate telemetry lake with a schema and lifecycle owned by that backend.
| Signal | Backend and logical data | Useful dimensions | Partitioning and retention |
|---|---|---|---|
| Logs | Loki streams and compressed chunks | service_name, environment, cluster, namespace, level | Loki owns chunk layout in S3; keep 30 days initially. |
| Traces | Tempo traces and blocks | service.name, deployment.environment, http.route, status | Tempo owns block layout in S3; keep 7 days initially. |
| Metrics | Mimir time series and blocks | Prometheus labels such as service, environment, region | Mimir owns block layout in S3; keep 13 months if query and cost tests support it. |
| Shared resource metadata | OpenTelemetry resource attributes | Service, environment, version, cluster | Keep names and values consistent across all signals. |
Use low-cardinality labels for Loki stream identity and Prometheus series dimensions. service_name and environment are useful; user IDs, request IDs, raw URLs, and trace IDs are not. Put high-cardinality detail in log bodies or trace attributes, where the backend can store it without creating a new metric series or log stream for every value.
Define a telemetry contract that is reviewed alongside application changes:
- Every service supplies a stable service name, environment, and version.
- HTTP metrics use a route template such as
/orders/{id}, not the raw URL. - Logs are structured and include severity, service, and trace context when available.
- Trace sampling is consistent with debugging needs and explicitly documented.
- Retention changes require a storage and cost review.
The initial deployment can use separate S3 buckets for logs, metrics, and traces, with encryption and access policies per backend. Avoid applying generic expiration rules to backend data directories: Loki, Tempo, and Mimir implement retention and object cleanup according to their own metadata and compaction behavior. An S3 lifecycle rule that deletes objects behind a backend’s back can corrupt queryability or interfere with compaction.
6) Low-level design
The platform has four user-facing and data-serving components, plus two collection roles. Keep collection close to workloads and put backend write paths behind internal Kubernetes Services.
classDiagram
class Application {
+serviceName
+environment
+exportOTLP()
}
class AlloyDaemonSet {
+collectPodLogs()
+collectNodeSignals()
}
class AlloyGateway {
+receiveOTLP()
+applyProcessors()
+routeSignals()
}
class Loki {
+writeLogs()
+queryLogs()
}
class Mimir {
+writeMetrics()
+queryMetrics()
}
class Tempo {
+writeTraces()
+queryTraces()
}
class Grafana {
+queryDataSources()
+renderDashboards()
}
Application --> AlloyGateway
AlloyDaemonSet --> Loki
AlloyGateway --> Loki
AlloyGateway --> Mimir
AlloyGateway --> Tempo
Grafana --> Loki
Grafana --> Mimir
Grafana --> TempoCaption: Collection and storage are separate responsibilities; Grafana is the shared query and visualization layer.
A minimal Go application can emit a trace to a local OTLP HTTP collector. Set OTEL_EXPORTER_OTLP_ENDPOINT to the collector host and port, such as localhost:4318; the exporter adds the traces path.
package main
import (
"context"
"log"
"os"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
func main() {
ctx := context.Background()
endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
if endpoint == "" {
endpoint = "localhost:4318"
}
exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(endpoint))
if err != nil {
log.Fatal(err)
}
res, err := resource.New(ctx, resource.WithAttributes(
semconv.ServiceName("checkout-api"),
attribute.String("deployment.environment", "dev"),
))
if err != nil {
log.Fatal(err)
}
provider := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
)
otel.SetTracerProvider(provider)
defer func() {
if err := provider.Shutdown(ctx); err != nil {
log.Printf("shutdown tracer provider: %v", err)
}
}()
_, span := otel.Tracer("checkout-api").Start(ctx, "checkout.create")
span.SetAttributes(attribute.String("checkout.payment_method", "card"))
span.End()
}
Notice: set resource attributes once at startup, then attach operation-specific attributes to spans rather than turning request-specific values into metric labels.
A realistic HTTP handler should create spans around useful work, propagate the request context to downstream calls, and record errors without storing sensitive payloads. This complete example adds a handler span and returns a response; pin compatible OpenTelemetry Go modules in the application’s module files.
package main
import (
"context"
"errors"
"log"
"net/http"
"os"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
func main() {
ctx := context.Background()
endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
if endpoint == "" {
endpoint = "localhost:4318"
}
exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(endpoint))
if err != nil {
log.Fatal(err)
}
res, err := resource.New(ctx, resource.WithAttributes(
semconv.ServiceName("orders-api"),
attribute.String("deployment.environment", "production"),
))
if err != nil {
log.Fatal(err)
}
provider := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(res),
)
otel.SetTracerProvider(provider)
defer func() {
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := provider.Shutdown(shutdownCtx); err != nil {
log.Printf("shutdown tracer provider: %v", err)
}
}()
mux := http.NewServeMux()
mux.HandleFunc("POST /orders", func(w http.ResponseWriter, r *http.Request) {
ctx, span := otel.Tracer("orders-api").Start(r.Context(), "orders.create")
defer span.End()
span.SetAttributes(
attribute.String("http.request.method", r.Method),
attribute.String("http.route", "/orders"),
)
if err := createOrder(ctx); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "order creation failed")
http.Error(w, "order creation failed", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte(`{"status":"created"}`))
})
log.Fatal(http.ListenAndServe(":8080", mux))
}
func createOrder(ctx context.Context) error {
if ctx.Err() != nil {
return ctx.Err()
}
return errors.New("replace with order persistence")
}
Notice: use a route template and bounded attributes; do not add order IDs, customer details, or raw request bodies as metric labels or default span attributes.
7) Key flows
For ingestion, the collector gateway is the controlled boundary. It batches and routes signals, while backend services perform their own durable writes. The product should continue serving requests if an observability backend is temporarily impaired; configure bounded retries and queues rather than allowing telemetry export to block application work.
sequenceDiagram
participant app as Application
participant alloy as Alloy gateway
participant tempo as Tempo
participant mimir as Mimir
participant loki as Loki
participant s3 as Amazon S3
app->>alloy: Send OTLP traces and metrics
alloy->>tempo: Export traces
alloy->>mimir: Remote write metrics
app->>alloy: Emit structured logs
alloy->>loki: Push logs
tempo->>s3: Persist trace blocks
mimir->>s3: Persist metric blocks
loki->>s3: Persist log chunksCaption: The gateway routes telemetry to dedicated backends, which persist data in their own S3 buckets.
For exploration, Grafana queries each backend through its data-source configuration. Configure trace-to-logs and trace-to-metrics links using shared service and environment attributes. Grafana should not be the only place where alert rules exist: keep critical rules and dashboards provisioned as code so they can be reviewed and restored.
sequenceDiagram
participant engineer as Engineer
participant grafana as Grafana
participant mimir as Mimir
participant tempo as Tempo
participant loki as Loki
engineer->>grafana: Open service dashboard
grafana->>mimir: Query request rate and errors
grafana->>tempo: Load selected trace
grafana->>loki: Find logs by service and trace ID
mimir-->>grafana: Return metric series
tempo-->>grafana: Return trace spans
loki-->>grafana: Return matching log lines
grafana-->>engineer: Show correlated signalsCaption: A useful debugging path begins with a service-level symptom and narrows to a trace and its related logs.
8) Build and release
Manage Kubernetes resources and storage prerequisites separately. Terraform owns EKS dependencies, S3 buckets, KMS keys, identity policies, and the workload identities. Helm releases own Grafana, Loki, Tempo, Mimir, and Alloy. Pin chart versions, image digests, and Terraform provider versions in the repository; upgrade them through review rather than floating to the newest version on every deploy.
This Terraform sketch provisions dedicated encrypted buckets. It assumes the AWS provider is configured and the deployment role has permission to create these resources. In production, also manage bucket policies, KMS key policies, and IAM identity bindings as code.
variable "name_prefix" {
type = string
default = "platform-observability-prod"
}
resource "aws_kms_key" "telemetry" {
description = "Encryption key for observability data"
deletion_window_in_days = 30
enable_key_rotation = true
}
resource "aws_s3_bucket" "telemetry" {
for_each = toset(["logs", "metrics", "traces"])
bucket = "${var.name_prefix}-${each.key}"
tags = {
system = "observability"
environment = "production"
signal = each.key
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "telemetry" {
for_each = aws_s3_bucket.telemetry
bucket = each.value.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.telemetry.arn
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_public_access_block" "telemetry" {
for_each = aws_s3_bucket.telemetry
bucket = each.value.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
A staging namespace can run a smaller stack with separate buckets and identities. Keep staging data out of production dashboards and production SLO calculations. Use resource quotas and pod disruption budgets to prevent test workloads from consuming production capacity if both environments share a cluster.
A CI pipeline should validate configuration and deploy to staging before promotion. The example assumes repository scripts wrap the chosen Terraform and Helm versions and that the deployment role is obtained through GitHub Actions OIDC.
name: observability-platform
on:
pull_request:
push:
branches: [main]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Terraform format and validate
run: |
terraform -chdir=infra fmt -check -recursive
terraform -chdir=infra init -backend=false
terraform -chdir=infra validate
- name: Helm lint
run: make helm-lint
- name: Unit and contract tests
run: make test
staging:
if: github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
environment: observability-staging
steps:
- uses: actions/checkout@v4
- name: Configure cloud identity
run: make aws-oidc-login
- name: Deploy staging
run: make deploy-staging
- name: Smoke test
run: make smoke-test-staging
production:
if: github.event_name == 'push'
needs: staging
runs-on: ubuntu-latest
environment: observability-production
steps:
- uses: actions/checkout@v4
- name: Configure cloud identity
run: make aws-oidc-login
- name: Promote reviewed release
run: make deploy-production
Treat the chart values and dashboards as release artifacts. Validate rendered Kubernetes resources, review changes to retention and resource requests, and use a staged rollout for backend upgrades. A rollback plan should include both Helm release rollback and a compatibility check for backend data formats; reverting a chart does not necessarily reverse a storage-format migration.
Test at four levels:
- Unit: Validate configuration generators, label normalization, routing rules, and dashboard provisioning inputs.
- Integration: Deploy a temporary or staging stack, write representative logs, metrics, and traces, then query each backend.
- Contract: Assert that services emit required resource attributes and that dashboards and alerts use the agreed metric and label names.
- Load: Replay expected peak ingestion and query patterns. Test backend restarts, S3 throttling behavior, and collector backpressure; measure dropped telemetry and recovery time rather than only successful request throughput.
9) Observability
The observability platform needs independent health checks. Monitor EKS, S3 access, collectors, backend ingestion, compaction, query services, and Grafana. If all alerting depends on Mimir and Mimir is down, the platform may fail silently; retain an independent cloud-level alert path for cluster, node, and bucket problems.
Use structured logs with stable fields, and avoid logging secrets, credentials, or raw customer payloads. A useful collector or backend event might look like this:
{
"timestamp": "2026-11-19T12:00:00Z",
"level": "warn",
"service": "alloy-gateway",
"environment": "production",
"event": "export_retry",
"backend": "tempo",
"retry_attempt": 3,
"queue_depth": 1200,
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}
The example includes a trace identifier for correlation, but that identifier should remain a log field, not a Loki stream label. A small set of stable labels keeps queries useful and limits index growth.
| Area | RED signals for services | USE signals for resources |
|---|---|---|
| Alloy | Export request rate, export errors, export duration | CPU and memory utilization, queue use, receiver and exporter saturation |
| Loki | Push and query rate, failed pushes and queries, request duration | CPU, memory, disk, object-store request pressure, ingester capacity |
| Mimir | Write and query rate, rejected samples, query duration | CPU, memory, compaction capacity, cache utilization, object-store pressure |
| Tempo | Trace write and query rate, rejected spans, query duration | CPU, memory, block processing capacity, object-store pressure |
| Grafana | Dashboard and API request rate, failures, response duration | CPU, memory, connection and query concurrency saturation |
| EKS nodes | Node and pod service errors, where applicable | CPU, memory, filesystem, network utilization and saturation |
Instrument application code with OpenTelemetry as shown in the low-level design. Export OTLP to an in-cluster Alloy gateway, use batch processors, and set bounded timeouts. Keep application request paths independent of telemetry export success: a failed export should produce a measurable drop or queue event, not a customer-facing failure.
| SLI | SLO target | Measurement |
|---|---|---|
| Grafana availability | 99.9% monthly | Successful authenticated health and query requests |
| Metric ingestion success | 99.5% over 30 days | Accepted samples divided by attempted samples, excluding explicitly rejected invalid data |
| Trace ingestion success | 99.5% over 30 days | Accepted spans divided by attempted spans |
| Log ingestion success | 99.5% over 30 days | Accepted log records divided by attempted records |
| Common dashboard latency | 95% under 10 seconds | Representative provisioned dashboard queries |
| Telemetry freshness | 95% visible within 2 minutes | Time between collector receipt and query visibility |
These are starting targets, not claims about default LGTM behavior. Define the denominator carefully: collector drops, malformed payloads, and intentional sampling need separate treatment, or the SLI can hide data loss.
Dashboards should answer operational questions rather than merely display component internals:
- Service overview: request rate, errors, latency, saturation, and a link to relevant traces.
- Ingestion health: collector queue depth, exporter failures, rejected data, and backend freshness.
- Backend health: read/write errors, compaction or block-processing backlog, query latency, pod restarts, and object-store errors.
- Capacity and cost: signal volume, retention horizon, S3 growth, and query load by team or service.
- SLO status: error-budget burn rates with links from service-level symptoms to supporting telemetry.
Page on sustained SLO burn, loss of ingestion beyond the recovery window, backend unavailability, rapidly growing queues, or a forecast that indicates storage or compute exhaustion before the team can act. Route slow queries, isolated pod restarts, and moderate capacity drift to tickets unless they are causing an active user or SLO impact. Alerts should identify the affected signal, the likely failure domain, and the runbook.
10) Security and compliance
Telemetry is sensitive data. Logs and traces can accidentally capture tokens, email addresses, payment details, or internal identifiers. Treat them as production data even when the application team considers them “just diagnostics.”
- Keep Grafana and backend APIs private by default. Use an authenticated ingress for Grafana and restrict access through the organization’s identity provider.
- Grant each backend access only to its own S3 bucket through workload identity. Avoid shared access keys and broad bucket permissions.
- Encrypt S3 with KMS and enable public access blocks. Restrict KMS use to the necessary workload identities and administrators.
- Use TLS at the ingress and for external telemetry submission. Restrict internal service access with Kubernetes network policies where the CNI supports them.
- Redact or drop sensitive attributes at instrumentation or collector processing stages. Do not rely on dashboard permissions to protect data already stored in a shared backend.
- Review tenant, retention, and deletion needs with security and compliance owners. Confirm that backups, exported dashboards, and incident attachments follow the same policy.
- Keep administrative access separate from read-only investigation access, and audit configuration changes and access to sensitive dashboards.
A single shared production tenant is a reasonable starting point for a small team, but it is not a security boundary between strongly isolated customer populations. If telemetry contains customer-specific data with different access controls, design tenancy and authorization before ingestion rather than trying to reconstruct isolation later.
11) Cost
The table is an assumption-based monthly estimate in USD for the scenario above. It is a planning envelope, not a vendor quote; actual cost depends on region, data shape, query frequency, storage compression, node selection, transfer paths, and current prices. It excludes the application’s existing EKS cost if the platform shares that cluster.
| Cost area | Assumption-based monthly estimate | Main cost drivers |
|---|---|---|
| EKS platform compute | $700–$1,400 | Backend replicas, collector DaemonSets, memory-optimized node capacity, and reserved headroom |
| S3 storage and requests | $150–$600 | 40 GB/day compressed, retention policy, compaction, query reads, and object count |
| Load balancing and network | $100–$400 | Ingress, cross-AZ traffic, egress, and inter-service data movement |
| KMS and supporting services | $30–$150 | Key usage, secrets, DNS, and supporting control-plane integrations |
| Operational headroom | $200–$500 | Temporary scale-up for upgrades, backfills, and incident investigation |
| Estimated total | $1,180–$3,050 | Validate with measured ingestion and region-specific pricing |
Storage is only one part of the bill. Expensive broad queries, high-cardinality metrics, chatty logs, cross-AZ traffic, and excessive replication can raise cost even when retention is short. Set ingestion limits and query timeouts, review data growth monthly, and alert on S3 growth and backend capacity before budgets are exceeded.
12) Failure modes, resilience and disaster recovery
| Failure mode | User impact | Response |
|---|---|---|
| Alloy pod restart or node loss | Temporary collection gap or retry | Run a DaemonSet across nodes; use bounded queues and retries; alert on drops and queue growth. |
| Backend pod or AZ loss | Reduced write or query capacity | Distribute replicas across AZs with topology constraints and disruption budgets; test failover. |
| S3 throttling or access failure | Delayed writes, failed queries, or compaction backlog | Alert on object-store errors; verify IAM, bucket policy, KMS access, and retry behavior. |
| Cardinality explosion | Memory pressure, rejected writes, slow queries, rising cost | Enforce label policy, ingestion limits, and dashboard reviews; remove unstable dimensions. |
| Bad Helm or configuration release | Ingestion or query outage | Promote through staging, keep configuration in version control, and rehearse rollback. |
| Region outage | Platform unavailable | Application continues without synchronous observability dependencies; restore in a second region if the recovery objective justifies it. |
The baseline recovery strategy is single-region resilience across multiple AZs, with S3 durability and reproducible infrastructure. Back up Terraform state, Helm values, dashboards, alert rules, and identity policies. Treat backend object data as recoverable only after testing that the necessary metadata, bucket access, and component configuration are available together.
Set explicit recovery objectives with the teams that use the platform. A practical starting assumption is a four-hour recovery time objective for restoring query and ingestion service in-region, with telemetry loss bounded by the collector’s configured buffering and backend durability behavior. Those are design targets, not guaranteed outcomes. Validate them by deleting a staging backend, restoring its configuration, and testing reads from retained data.
For regional recovery, replicate the required S3 data and deploy the stack from code in a second region. Decide whether the recovery mode is cold standby or warm standby based on the value of immediate access during a regional outage. Cross-region replication adds cost and operational complexity; it does not automatically make Grafana dashboards, identity, DNS, or alert routing fail over correctly.
13) Evolving to the next scale
Do not scale every component at once. Scale from observed bottlenecks and distinguish ingestion pressure from query pressure.
| Trigger | Next change |
|---|---|
| Collector queues grow or exports are dropped at peak | Increase gateway replicas and resources; review batching, retries, sampling, and network limits. |
| Loki writes or queries compete for resources | Separate read and write capacity, add replicas, and tune caches and query limits. |
| Mimir ingestion or compaction falls behind | Increase the relevant component capacity and review cardinality, retention, and write patterns. |
| Tempo trace queries or block processing become slow | Add query capacity and review trace volume, sampling, and retention. |
| Teams need independent access or retention | Introduce tenancy and explicit authorization boundaries; do not treat a dashboard folder as data isolation. |
| One cluster creates an unacceptable blast radius | Move the platform to a dedicated EKS cluster and isolate its upgrades and capacity. |
| Regional loss exceeds the accepted recovery objective | Add a tested secondary-region recovery plan, including object replication, identity, DNS, and alert routing. |
At larger scale, separate backend read and write paths, use dedicated node groups, and consider dedicated platform clusters. Re-evaluate storage and retention with actual signal volumes before increasing replica counts. More replicas can improve capacity or availability, but they also increase resource use and do not fix poor labels, unbounded queries, or noisy instrumentation.
14) Phased implementation plan
- Agree on the telemetry contract. Define service identity, environment naming, required resource attributes, sensitive-data rules, and initial retention. Review the contract with application and security owners.
- Build the storage and identity foundation. Provision encrypted S3 buckets, KMS permissions, workload identity, network boundaries, and staging/prod separation through Terraform. Test that each workload can access only its assigned bucket.
- Deploy the smallest useful stack in staging. Install pinned Helm releases for Alloy, Loki, Mimir, Tempo, and Grafana. Provision data sources, one service dashboard, and an ingestion-health dashboard as code.
- Instrument one representative service. Export traces and metrics through OTLP, standardize structured logs, and verify signal correlation. Measure the real data volume and inspect labels before onboarding all services.
- Test failure and capacity behavior. Load-test ingestion and common queries, restart pods and nodes, interrupt backend connectivity, and validate queue limits, alerting, and recovery procedures.
- Promote production with operational ownership. Roll out the stack and application instrumentation gradually. Assign owners for upgrades, SLO review, retention changes, security review, and cost checks.
- Review after real usage. Use measured ingestion, query latency, storage growth, and incident experience to adjust resource requests, sampling, retention, and SLOs. Add complexity only when a measured limit or isolation need requires it.
This article was generated with AI. #AIGenerated