Skip to content

Grafana LGTM on Kubernetes: An Architecture Blueprint

A practical AWS EKS blueprint for running Loki, Grafana, Tempo, and Mimir with bounded retention, controlled cardinality, safe releases, and useful SLOs.

Ekanatha Reddy Urivakili19 min3,744 words

1) Context and scale

This blueprint deploys the Grafana LGTM stack—Loki for logs, Grafana for exploration and dashboards, Tempo for traces, and Mimir for metrics—on AWS EKS. It is for a small product organization that needs to correlate application signals without adopting a separate managed observability product for each signal.

The application is a handful of services, including a modular monolith, running on Kubernetes. It uses a multi-AZ database, a cache, and a queue. Application traffic is bursty, deployments happen through CI/CD, and a staging environment is required. The observability platform is an internal production service: if it becomes unavailable, the customer-facing application should keep working.

The design uses EKS for compute and S3 for durable signal storage. Grafana, Loki, Tempo, and Mimir run as workloads in Kubernetes; they are not AWS-managed services. The initial deployment is single-region and multi-AZ, with three separate S3 buckets and workload identities that grant each backend access only to its own bucket.

AssumptionPlanning value
Daily active users50,000
Application traffic250 requests/s peak; 60 requests/s average
Telemetry volume40 GB/day compressed, across logs, traces, and metrics
RetentionLogs 30 days; traces 7 days; metrics 13 months
Engineering team2 product teams; 1 platform engineer shares ownership
Budget$2,000–$4,000/month for the observability platform, excluding application compute

The initial topology is deliberately modest: one EKS cluster, one region, multi-AZ node groups, and replicated backend components where a single pod would be an unacceptable failure point. It does not attempt multi-region active-active telemetry ingestion. That adds operational and cost complexity before the expected scale demands it.

2) Requirements

Functional requirements define what the platform must let engineers do. Non-functional requirements bound how it behaves during normal operation and failure.

Functional requirementAcceptance criterion
Ingest application tracesServices export OTLP traces through an in-cluster collector gateway.
Ingest application and Kubernetes logsCollectors attach stable resource metadata and send logs to Loki.
Ingest platform and application metricsScrape Kubernetes targets and accept application metrics into Mimir.
Correlate signalsGrafana links a trace to logs and metrics using service, environment, and trace identifiers.
Alert on service healthSLO and infrastructure alerts route through Alertmanager to the team’s paging system.
Explore telemetryEngineers can search logs, query metrics, and inspect traces through authenticated Grafana access.
Operate staging safelyStaging telemetry is separated from production by namespace, storage bucket, credentials, and dashboard conventions.
Non-functional requirementTarget
Availability99.9% monthly availability for Grafana query and ingestion paths, excluding planned maintenance.
Ingestion durabilityA collector retry buffer absorbs short backend disruptions; durable object storage retains accepted backend data.
Query latencyCommon dashboard queries complete in under 10 seconds at the assumed scale; this is a design target to validate, not a guarantee.
IsolationTelemetry storage and credentials are separate from application data and from staging.
SecurityPrivate cluster access for internal components, TLS at ingress and between external clients and endpoints, least-privilege workload identity.
OperabilityBackend health, ingestion lag, capacity, and query failures are visible independently of the stack being monitored.
Cost controlRetention, sampling, label policy, and ingestion limits are explicit and reviewed.

For the first release, prioritize a reliable ingestion path and useful dashboards over elaborate tenancy. A single production tenant is sufficient for two teams if access is controlled at Grafana and write access is limited to trusted collectors. Add tenant isolation when teams have materially different access or retention requirements.

3) High-level design

Applications export traces and metrics using OpenTelemetry. Grafana Alloy runs as a DaemonSet for Kubernetes and node collection, and as a gateway Deployment for OTLP intake and processing. Alloy sends logs to Loki, metrics to Mimir, and traces to Tempo. Grafana reads from all three backends. The backends use S3 for durable storage; their short-lived working state and caches run in EKS.

flowchart LR
    app["Application pods with OpenTelemetry"] --> alloygw["Alloy OTLP gateway"]
    nodes["Kubernetes nodes and pod logs"] --> alloyds["Alloy DaemonSet"]
    alloyds --> loki["Loki on EKS"]
    alloygw --> loki
    alloygw --> mimir["Mimir on EKS"]
    alloygw --> tempo["Tempo on EKS"]
    grafana["Grafana on EKS"] --> loki
    grafana --> mimir
    grafana --> tempo
    loki --> s3logs["Amazon S3 logs bucket"]
    mimir --> s3metrics["Amazon S3 metrics bucket"]
    tempo --> s3traces["Amazon S3 traces bucket"]
    alb["AWS Application Load Balancer"] --> grafana
    grafana --> alb

Caption: Applications and collectors write to the backends; Grafana queries them, and S3 holds durable signal data.

The public-facing surface should be small. An AWS Application Load Balancer exposes Grafana through an internal or tightly restricted ingress. Loki, Mimir, and Tempo endpoints remain cluster-internal. If an external collector must write to the platform, expose a separate authenticated OTLP endpoint rather than exposing backend APIs.

Use EKS Pod Identity or IAM Roles for Service Accounts to grant each backend access to its own S3 bucket. Encrypt buckets with AWS KMS, keep backend-to-backend traffic on the cluster network, and use network policies to limit who can reach write and query endpoints.

4) Cloud service mapping

The AWS column is the chosen implementation. Azure and GCP equivalents are infrastructure mappings, not drop-in replacements for LGTM. Each cloud’s native observability products have different query languages, data models, and operational characteristics.

Component or responsibilityAWS choiceAzure equivalentGCP equivalent
Kubernetes control planeAmazon EKSAzure Kubernetes Service (AKS)Google Kubernetes Engine (GKE)
Log, trace, and metric backendsLoki, Tempo, and Mimir on EKSSame open-source components on AKSSame open-source components on GKE
Grafana UIGrafana on EKSGrafana on AKSGrafana on GKE
Durable backend storageAmazon S3, one bucket per backendAzure Blob StorageGoogle Cloud Storage
Workload identityEKS Pod Identity or IAM Roles for Service AccountsMicrosoft Entra Workload IDWorkload Identity Federation for GKE
Encryption keysAWS Key Management ServiceAzure Key VaultCloud Key Management Service
Ingress and TLSAWS Load Balancer Controller and Application Load BalancerApplication Gateway Ingress Controller or Azure Load BalancerGKE Ingress or Gateway
DNSAmazon Route 53Azure DNSCloud DNS
SecretsAWS Secrets Manager or Kubernetes secrets backed by an external secrets integrationAzure Key VaultSecret Manager
Native alternativeAmazon CloudWatch and AWS X-Ray; different product modelAzure Monitor and Application Insights; different product modelCloud Logging, Cloud Monitoring, and Cloud Trace; different product model

Do not expose S3 credentials as static Kubernetes secrets. Use a workload identity bound to each backend’s service account and a narrowly scoped bucket policy. The same principle applies to the equivalent identity systems on Azure and GCP.

5) Data design

LGTM stores different signal types in purpose-built backends; it is not a relational data model. Treat each backend’s object storage as a separate telemetry lake with a schema and lifecycle owned by that backend.

SignalBackend and logical dataUseful dimensionsPartitioning and retention
LogsLoki streams and compressed chunksservice_name, environment, cluster, namespace, levelLoki owns chunk layout in S3; keep 30 days initially.
TracesTempo traces and blocksservice.name, deployment.environment, http.route, statusTempo owns block layout in S3; keep 7 days initially.
MetricsMimir time series and blocksPrometheus labels such as service, environment, regionMimir owns block layout in S3; keep 13 months if query and cost tests support it.
Shared resource metadataOpenTelemetry resource attributesService, environment, version, clusterKeep names and values consistent across all signals.

Use low-cardinality labels for Loki stream identity and Prometheus series dimensions. service_name and environment are useful; user IDs, request IDs, raw URLs, and trace IDs are not. Put high-cardinality detail in log bodies or trace attributes, where the backend can store it without creating a new metric series or log stream for every value.

Define a telemetry contract that is reviewed alongside application changes:

  • Every service supplies a stable service name, environment, and version.
  • HTTP metrics use a route template such as /orders/{id}, not the raw URL.
  • Logs are structured and include severity, service, and trace context when available.
  • Trace sampling is consistent with debugging needs and explicitly documented.
  • Retention changes require a storage and cost review.

The initial deployment can use separate S3 buckets for logs, metrics, and traces, with encryption and access policies per backend. Avoid applying generic expiration rules to backend data directories: Loki, Tempo, and Mimir implement retention and object cleanup according to their own metadata and compaction behavior. An S3 lifecycle rule that deletes objects behind a backend’s back can corrupt queryability or interfere with compaction.

6) Low-level design

The platform has four user-facing and data-serving components, plus two collection roles. Keep collection close to workloads and put backend write paths behind internal Kubernetes Services.

classDiagram
    class Application {
        +serviceName
        +environment
        +exportOTLP()
    }
    class AlloyDaemonSet {
        +collectPodLogs()
        +collectNodeSignals()
    }
    class AlloyGateway {
        +receiveOTLP()
        +applyProcessors()
        +routeSignals()
    }
    class Loki {
        +writeLogs()
        +queryLogs()
    }
    class Mimir {
        +writeMetrics()
        +queryMetrics()
    }
    class Tempo {
        +writeTraces()
        +queryTraces()
    }
    class Grafana {
        +queryDataSources()
        +renderDashboards()
    }
    Application --> AlloyGateway
    AlloyDaemonSet --> Loki
    AlloyGateway --> Loki
    AlloyGateway --> Mimir
    AlloyGateway --> Tempo
    Grafana --> Loki
    Grafana --> Mimir
    Grafana --> Tempo

Caption: Collection and storage are separate responsibilities; Grafana is the shared query and visualization layer.

A minimal Go application can emit a trace to a local OTLP HTTP collector. Set OTEL_EXPORTER_OTLP_ENDPOINT to the collector host and port, such as localhost:4318; the exporter adds the traces path.

package main

import (
	"context"
	"log"
	"os"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func main() {
	ctx := context.Background()
	endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
	if endpoint == "" {
		endpoint = "localhost:4318"
	}

	exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(endpoint))
	if err != nil {
		log.Fatal(err)
	}

	res, err := resource.New(ctx, resource.WithAttributes(
		semconv.ServiceName("checkout-api"),
		attribute.String("deployment.environment", "dev"),
	))
	if err != nil {
		log.Fatal(err)
	}

	provider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)
	otel.SetTracerProvider(provider)
	defer func() {
		if err := provider.Shutdown(ctx); err != nil {
			log.Printf("shutdown tracer provider: %v", err)
		}
	}()

	_, span := otel.Tracer("checkout-api").Start(ctx, "checkout.create")
	span.SetAttributes(attribute.String("checkout.payment_method", "card"))
	span.End()
}

Notice: set resource attributes once at startup, then attach operation-specific attributes to spans rather than turning request-specific values into metric labels.

A realistic HTTP handler should create spans around useful work, propagate the request context to downstream calls, and record errors without storing sensitive payloads. This complete example adds a handler span and returns a response; pin compatible OpenTelemetry Go modules in the application’s module files.

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func main() {
	ctx := context.Background()
	endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
	if endpoint == "" {
		endpoint = "localhost:4318"
	}

	exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(endpoint))
	if err != nil {
		log.Fatal(err)
	}

	res, err := resource.New(ctx, resource.WithAttributes(
		semconv.ServiceName("orders-api"),
		attribute.String("deployment.environment", "production"),
	))
	if err != nil {
		log.Fatal(err)
	}

	provider := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)
	otel.SetTracerProvider(provider)
	defer func() {
		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := provider.Shutdown(shutdownCtx); err != nil {
			log.Printf("shutdown tracer provider: %v", err)
		}
	}()

	mux := http.NewServeMux()
	mux.HandleFunc("POST /orders", func(w http.ResponseWriter, r *http.Request) {
		ctx, span := otel.Tracer("orders-api").Start(r.Context(), "orders.create")
		defer span.End()

		span.SetAttributes(
			attribute.String("http.request.method", r.Method),
			attribute.String("http.route", "/orders"),
		)

		if err := createOrder(ctx); err != nil {
			span.RecordError(err)
			span.SetStatus(codes.Error, "order creation failed")
			http.Error(w, "order creation failed", http.StatusInternalServerError)
			return
		}

		w.WriteHeader(http.StatusCreated)
		_, _ = w.Write([]byte(`{"status":"created"}`))
	})

	log.Fatal(http.ListenAndServe(":8080", mux))
}

func createOrder(ctx context.Context) error {
	if ctx.Err() != nil {
		return ctx.Err()
	}
	return errors.New("replace with order persistence")
}

Notice: use a route template and bounded attributes; do not add order IDs, customer details, or raw request bodies as metric labels or default span attributes.

7) Key flows

For ingestion, the collector gateway is the controlled boundary. It batches and routes signals, while backend services perform their own durable writes. The product should continue serving requests if an observability backend is temporarily impaired; configure bounded retries and queues rather than allowing telemetry export to block application work.

sequenceDiagram
    participant app as Application
    participant alloy as Alloy gateway
    participant tempo as Tempo
    participant mimir as Mimir
    participant loki as Loki
    participant s3 as Amazon S3
    app->>alloy: Send OTLP traces and metrics
    alloy->>tempo: Export traces
    alloy->>mimir: Remote write metrics
    app->>alloy: Emit structured logs
    alloy->>loki: Push logs
    tempo->>s3: Persist trace blocks
    mimir->>s3: Persist metric blocks
    loki->>s3: Persist log chunks

Caption: The gateway routes telemetry to dedicated backends, which persist data in their own S3 buckets.

For exploration, Grafana queries each backend through its data-source configuration. Configure trace-to-logs and trace-to-metrics links using shared service and environment attributes. Grafana should not be the only place where alert rules exist: keep critical rules and dashboards provisioned as code so they can be reviewed and restored.

sequenceDiagram
    participant engineer as Engineer
    participant grafana as Grafana
    participant mimir as Mimir
    participant tempo as Tempo
    participant loki as Loki
    engineer->>grafana: Open service dashboard
    grafana->>mimir: Query request rate and errors
    grafana->>tempo: Load selected trace
    grafana->>loki: Find logs by service and trace ID
    mimir-->>grafana: Return metric series
    tempo-->>grafana: Return trace spans
    loki-->>grafana: Return matching log lines
    grafana-->>engineer: Show correlated signals

Caption: A useful debugging path begins with a service-level symptom and narrows to a trace and its related logs.

8) Build and release

Manage Kubernetes resources and storage prerequisites separately. Terraform owns EKS dependencies, S3 buckets, KMS keys, identity policies, and the workload identities. Helm releases own Grafana, Loki, Tempo, Mimir, and Alloy. Pin chart versions, image digests, and Terraform provider versions in the repository; upgrade them through review rather than floating to the newest version on every deploy.

This Terraform sketch provisions dedicated encrypted buckets. It assumes the AWS provider is configured and the deployment role has permission to create these resources. In production, also manage bucket policies, KMS key policies, and IAM identity bindings as code.

variable "name_prefix" {
  type    = string
  default = "platform-observability-prod"
}

resource "aws_kms_key" "telemetry" {
  description             = "Encryption key for observability data"
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

resource "aws_s3_bucket" "telemetry" {
  for_each = toset(["logs", "metrics", "traces"])
  bucket   = "${var.name_prefix}-${each.key}"

  tags = {
    system      = "observability"
    environment = "production"
    signal      = each.key
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "telemetry" {
  for_each = aws_s3_bucket.telemetry
  bucket   = each.value.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.telemetry.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "telemetry" {
  for_each                = aws_s3_bucket.telemetry
  bucket                  = each.value.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

A staging namespace can run a smaller stack with separate buckets and identities. Keep staging data out of production dashboards and production SLO calculations. Use resource quotas and pod disruption budgets to prevent test workloads from consuming production capacity if both environments share a cluster.

A CI pipeline should validate configuration and deploy to staging before promotion. The example assumes repository scripts wrap the chosen Terraform and Helm versions and that the deployment role is obtained through GitHub Actions OIDC.

name: observability-platform

on:
  pull_request:
  push:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Terraform format and validate
        run: |
          terraform -chdir=infra fmt -check -recursive
          terraform -chdir=infra init -backend=false
          terraform -chdir=infra validate
      - name: Helm lint
        run: make helm-lint
      - name: Unit and contract tests
        run: make test

  staging:
    if: github.event_name == 'push'
    needs: validate
    runs-on: ubuntu-latest
    environment: observability-staging
    steps:
      - uses: actions/checkout@v4
      - name: Configure cloud identity
        run: make aws-oidc-login
      - name: Deploy staging
        run: make deploy-staging
      - name: Smoke test
        run: make smoke-test-staging

  production:
    if: github.event_name == 'push'
    needs: staging
    runs-on: ubuntu-latest
    environment: observability-production
    steps:
      - uses: actions/checkout@v4
      - name: Configure cloud identity
        run: make aws-oidc-login
      - name: Promote reviewed release
        run: make deploy-production

Treat the chart values and dashboards as release artifacts. Validate rendered Kubernetes resources, review changes to retention and resource requests, and use a staged rollout for backend upgrades. A rollback plan should include both Helm release rollback and a compatibility check for backend data formats; reverting a chart does not necessarily reverse a storage-format migration.

Test at four levels:

  • Unit: Validate configuration generators, label normalization, routing rules, and dashboard provisioning inputs.
  • Integration: Deploy a temporary or staging stack, write representative logs, metrics, and traces, then query each backend.
  • Contract: Assert that services emit required resource attributes and that dashboards and alerts use the agreed metric and label names.
  • Load: Replay expected peak ingestion and query patterns. Test backend restarts, S3 throttling behavior, and collector backpressure; measure dropped telemetry and recovery time rather than only successful request throughput.

9) Observability

The observability platform needs independent health checks. Monitor EKS, S3 access, collectors, backend ingestion, compaction, query services, and Grafana. If all alerting depends on Mimir and Mimir is down, the platform may fail silently; retain an independent cloud-level alert path for cluster, node, and bucket problems.

Use structured logs with stable fields, and avoid logging secrets, credentials, or raw customer payloads. A useful collector or backend event might look like this:

{
  "timestamp": "2026-11-19T12:00:00Z",
  "level": "warn",
  "service": "alloy-gateway",
  "environment": "production",
  "event": "export_retry",
  "backend": "tempo",
  "retry_attempt": 3,
  "queue_depth": 1200,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

The example includes a trace identifier for correlation, but that identifier should remain a log field, not a Loki stream label. A small set of stable labels keeps queries useful and limits index growth.

AreaRED signals for servicesUSE signals for resources
AlloyExport request rate, export errors, export durationCPU and memory utilization, queue use, receiver and exporter saturation
LokiPush and query rate, failed pushes and queries, request durationCPU, memory, disk, object-store request pressure, ingester capacity
MimirWrite and query rate, rejected samples, query durationCPU, memory, compaction capacity, cache utilization, object-store pressure
TempoTrace write and query rate, rejected spans, query durationCPU, memory, block processing capacity, object-store pressure
GrafanaDashboard and API request rate, failures, response durationCPU, memory, connection and query concurrency saturation
EKS nodesNode and pod service errors, where applicableCPU, memory, filesystem, network utilization and saturation

Instrument application code with OpenTelemetry as shown in the low-level design. Export OTLP to an in-cluster Alloy gateway, use batch processors, and set bounded timeouts. Keep application request paths independent of telemetry export success: a failed export should produce a measurable drop or queue event, not a customer-facing failure.

SLISLO targetMeasurement
Grafana availability99.9% monthlySuccessful authenticated health and query requests
Metric ingestion success99.5% over 30 daysAccepted samples divided by attempted samples, excluding explicitly rejected invalid data
Trace ingestion success99.5% over 30 daysAccepted spans divided by attempted spans
Log ingestion success99.5% over 30 daysAccepted log records divided by attempted records
Common dashboard latency95% under 10 secondsRepresentative provisioned dashboard queries
Telemetry freshness95% visible within 2 minutesTime between collector receipt and query visibility

These are starting targets, not claims about default LGTM behavior. Define the denominator carefully: collector drops, malformed payloads, and intentional sampling need separate treatment, or the SLI can hide data loss.

Dashboards should answer operational questions rather than merely display component internals:

  • Service overview: request rate, errors, latency, saturation, and a link to relevant traces.
  • Ingestion health: collector queue depth, exporter failures, rejected data, and backend freshness.
  • Backend health: read/write errors, compaction or block-processing backlog, query latency, pod restarts, and object-store errors.
  • Capacity and cost: signal volume, retention horizon, S3 growth, and query load by team or service.
  • SLO status: error-budget burn rates with links from service-level symptoms to supporting telemetry.

Page on sustained SLO burn, loss of ingestion beyond the recovery window, backend unavailability, rapidly growing queues, or a forecast that indicates storage or compute exhaustion before the team can act. Route slow queries, isolated pod restarts, and moderate capacity drift to tickets unless they are causing an active user or SLO impact. Alerts should identify the affected signal, the likely failure domain, and the runbook.

10) Security and compliance

Telemetry is sensitive data. Logs and traces can accidentally capture tokens, email addresses, payment details, or internal identifiers. Treat them as production data even when the application team considers them “just diagnostics.”

  • Keep Grafana and backend APIs private by default. Use an authenticated ingress for Grafana and restrict access through the organization’s identity provider.
  • Grant each backend access only to its own S3 bucket through workload identity. Avoid shared access keys and broad bucket permissions.
  • Encrypt S3 with KMS and enable public access blocks. Restrict KMS use to the necessary workload identities and administrators.
  • Use TLS at the ingress and for external telemetry submission. Restrict internal service access with Kubernetes network policies where the CNI supports them.
  • Redact or drop sensitive attributes at instrumentation or collector processing stages. Do not rely on dashboard permissions to protect data already stored in a shared backend.
  • Review tenant, retention, and deletion needs with security and compliance owners. Confirm that backups, exported dashboards, and incident attachments follow the same policy.
  • Keep administrative access separate from read-only investigation access, and audit configuration changes and access to sensitive dashboards.

A single shared production tenant is a reasonable starting point for a small team, but it is not a security boundary between strongly isolated customer populations. If telemetry contains customer-specific data with different access controls, design tenancy and authorization before ingestion rather than trying to reconstruct isolation later.

11) Cost

The table is an assumption-based monthly estimate in USD for the scenario above. It is a planning envelope, not a vendor quote; actual cost depends on region, data shape, query frequency, storage compression, node selection, transfer paths, and current prices. It excludes the application’s existing EKS cost if the platform shares that cluster.

Cost areaAssumption-based monthly estimateMain cost drivers
EKS platform compute$700–$1,400Backend replicas, collector DaemonSets, memory-optimized node capacity, and reserved headroom
S3 storage and requests$150–$60040 GB/day compressed, retention policy, compaction, query reads, and object count
Load balancing and network$100–$400Ingress, cross-AZ traffic, egress, and inter-service data movement
KMS and supporting services$30–$150Key usage, secrets, DNS, and supporting control-plane integrations
Operational headroom$200–$500Temporary scale-up for upgrades, backfills, and incident investigation
Estimated total$1,180–$3,050Validate with measured ingestion and region-specific pricing

Storage is only one part of the bill. Expensive broad queries, high-cardinality metrics, chatty logs, cross-AZ traffic, and excessive replication can raise cost even when retention is short. Set ingestion limits and query timeouts, review data growth monthly, and alert on S3 growth and backend capacity before budgets are exceeded.

12) Failure modes, resilience and disaster recovery

Failure modeUser impactResponse
Alloy pod restart or node lossTemporary collection gap or retryRun a DaemonSet across nodes; use bounded queues and retries; alert on drops and queue growth.
Backend pod or AZ lossReduced write or query capacityDistribute replicas across AZs with topology constraints and disruption budgets; test failover.
S3 throttling or access failureDelayed writes, failed queries, or compaction backlogAlert on object-store errors; verify IAM, bucket policy, KMS access, and retry behavior.
Cardinality explosionMemory pressure, rejected writes, slow queries, rising costEnforce label policy, ingestion limits, and dashboard reviews; remove unstable dimensions.
Bad Helm or configuration releaseIngestion or query outagePromote through staging, keep configuration in version control, and rehearse rollback.
Region outagePlatform unavailableApplication continues without synchronous observability dependencies; restore in a second region if the recovery objective justifies it.

The baseline recovery strategy is single-region resilience across multiple AZs, with S3 durability and reproducible infrastructure. Back up Terraform state, Helm values, dashboards, alert rules, and identity policies. Treat backend object data as recoverable only after testing that the necessary metadata, bucket access, and component configuration are available together.

Set explicit recovery objectives with the teams that use the platform. A practical starting assumption is a four-hour recovery time objective for restoring query and ingestion service in-region, with telemetry loss bounded by the collector’s configured buffering and backend durability behavior. Those are design targets, not guaranteed outcomes. Validate them by deleting a staging backend, restoring its configuration, and testing reads from retained data.

For regional recovery, replicate the required S3 data and deploy the stack from code in a second region. Decide whether the recovery mode is cold standby or warm standby based on the value of immediate access during a regional outage. Cross-region replication adds cost and operational complexity; it does not automatically make Grafana dashboards, identity, DNS, or alert routing fail over correctly.

13) Evolving to the next scale

Do not scale every component at once. Scale from observed bottlenecks and distinguish ingestion pressure from query pressure.

TriggerNext change
Collector queues grow or exports are dropped at peakIncrease gateway replicas and resources; review batching, retries, sampling, and network limits.
Loki writes or queries compete for resourcesSeparate read and write capacity, add replicas, and tune caches and query limits.
Mimir ingestion or compaction falls behindIncrease the relevant component capacity and review cardinality, retention, and write patterns.
Tempo trace queries or block processing become slowAdd query capacity and review trace volume, sampling, and retention.
Teams need independent access or retentionIntroduce tenancy and explicit authorization boundaries; do not treat a dashboard folder as data isolation.
One cluster creates an unacceptable blast radiusMove the platform to a dedicated EKS cluster and isolate its upgrades and capacity.
Regional loss exceeds the accepted recovery objectiveAdd a tested secondary-region recovery plan, including object replication, identity, DNS, and alert routing.

At larger scale, separate backend read and write paths, use dedicated node groups, and consider dedicated platform clusters. Re-evaluate storage and retention with actual signal volumes before increasing replica counts. More replicas can improve capacity or availability, but they also increase resource use and do not fix poor labels, unbounded queries, or noisy instrumentation.

14) Phased implementation plan

  1. Agree on the telemetry contract. Define service identity, environment naming, required resource attributes, sensitive-data rules, and initial retention. Review the contract with application and security owners.
  2. Build the storage and identity foundation. Provision encrypted S3 buckets, KMS permissions, workload identity, network boundaries, and staging/prod separation through Terraform. Test that each workload can access only its assigned bucket.
  3. Deploy the smallest useful stack in staging. Install pinned Helm releases for Alloy, Loki, Mimir, Tempo, and Grafana. Provision data sources, one service dashboard, and an ingestion-health dashboard as code.
  4. Instrument one representative service. Export traces and metrics through OTLP, standardize structured logs, and verify signal correlation. Measure the real data volume and inspect labels before onboarding all services.
  5. Test failure and capacity behavior. Load-test ingestion and common queries, restart pods and nodes, interrupt backend connectivity, and validate queue limits, alerting, and recovery procedures.
  6. Promote production with operational ownership. Roll out the stack and application instrumentation gradually. Assign owners for upgrades, SLO review, retention changes, security review, and cost checks.
  7. Review after real usage. Use measured ingestion, query latency, storage growth, and incident experience to adjust resource requests, sampling, retention, and SLOs. Add complexity only when a measured limit or isolation need requires it.

This article was generated with AI. #AIGenerated