Skip to content

Testing Across the SDLC: An Architecture Blueprint with Testcontainers

A practical testing strategy for a medium-scale SaaS: use unit, integration, contract, and end-to-end tests at the right points in the SDLC, with Testcontainers for realistic dependencies.

Ekanatha Reddy Urivakili21 min4,107 words

1. Context and scale

This blueprint is for a small SaaS team shipping an order-management API. It uses a modular monolith on ASP.NET Core, PostgreSQL as the system of record, Redis for cache-aside reads, and Amazon SQS for asynchronous order processing. The specific domain is illustrative; the testing strategy applies to any application whose correctness depends on database constraints, external interfaces, or asynchronous work.

The problem is not a shortage of tests. It is misplaced confidence: unit tests pass while SQL migrations fail, a mocked queue hides an incompatible message, or an end-to-end test becomes so slow and flaky that engineers stop trusting it. The architecture assigns each class of test a clear purpose, runs the cheapest useful checks early, and uses Testcontainers to run real dependencies in developer and CI environments.

The deployment target is AWS. GitHub Actions runs CI, builds an immutable container image, deploys it to an AWS staging environment, and runs a small end-to-end suite before production promotion. Testcontainers is for test infrastructure—not a substitute for staging, production-like cloud configuration, or provider-level verification.

AssumptionPlanning value
Daily active users10,000–100,000
Request volume20 requests/s average; 150 requests/s peak
Data volume100 GB primary relational data, excluding backups
Team size1–2 teams; 6–12 engineers in total
Monthly cloud budgetUS$1,000–3,000, including staging and test-related services
Availability target99.9% monthly for the order API
Application shapeOne modular monolith, one asynchronous worker
Release frequencySeveral production releases per week; assumptions, not a requirement

These are sizing assumptions, not measured demand or a capacity guarantee. Revisit them with production telemetry. In particular, daily users alone do not predict peak concurrency, query cost, or queue depth.

The operating principle is to test a behavior at the narrowest boundary that can establish it. A pricing rule belongs in a unit test. PostgreSQL uniqueness belongs in a test against PostgreSQL. An HTTP message contract belongs in a consumer/provider contract test. Authentication wiring and the deployed route belong in a small end-to-end test against staging. Use the real dependency when the behavior under test belongs to that dependency; use a fake when the behavior belongs to your code.

2. Requirements

Functional requirements

IDRequirementTesting implication
F1A customer can submit an order with one or more items.Unit-test validation and pricing; API integration-test persistence and response semantics.
F2An order is stored with a stable identifier and initial status.Verify persistence, transaction behavior, and database constraints against PostgreSQL.
F3A committed order produces an event for asynchronous processing.Test the outbox transaction and message schema; verify dispatch to a queue in a separate integration boundary.
F4Clients can retrieve an order by identifier.Test query behavior and API serialization; cover the deployed route with a smoke test.
F5API and worker releases remain compatible with the event contract.Validate producer and consumer expectations independently, then run a small staging flow.

Non-functional requirements

IDRequirementVerification approach
N1API availability is at least 99.9% monthly.Request-based SLI, production dashboard, and actionable paging thresholds.
N2Normal API responses have a p95 latency below 300 ms.Staging smoke and scheduled load tests; production RED metrics are authoritative.
N3A release must not silently drop committed order events.Database-backed outbox tests, queue delivery monitoring, and failure-injection exercises.
N4A pull request should give useful feedback within 10 minutes.Keep unit and container-backed tests parallelizable; reserve broad load tests for scheduled or pre-release runs.
N5Schema changes are reversible or safely forward-compatible.Migration tests from the previous schema and staged rollout checks.
N6Test data must not leak across test cases or into production.Disposable containers or isolated schemas; no production credentials in test jobs.

Latency and CI timing targets are goals to validate, not benchmark claims. A test suite that regularly exceeds its feedback budget needs better boundaries, parallelism, or fewer redundant scenarios—not a blanket increase in timeouts.

3. High-level design

The application runs on Amazon ECS Fargate behind an Application Load Balancer. PostgreSQL is Amazon RDS for PostgreSQL deployed across multiple Availability Zones. Redis caching uses Amazon ElastiCache, and asynchronous work uses Amazon SQS with a dead-letter queue. Images are stored in Amazon ECR. GitHub Actions owns CI orchestration; AWS CloudWatch and AWS Distro for OpenTelemetry (ADOT) provide the AWS-side telemetry path.

The testing flow is intentionally separate from the runtime flow. Pull requests run unit tests and Testcontainers-backed integration tests on an isolated CI runner. Contract checks validate API and event compatibility. A successful main-branch build is deployed to staging, where a small set of end-to-end tests verifies the integrated deployment. Production promotion uses the same image digest that passed staging.

flowchart LR
    Dev["Developer"]
    PR["GitHub pull request"]
    CI["GitHub Actions"]
    Unit["Unit tests"]
    Containers["Testcontainers: PostgreSQL and LocalStack"]
    Contract["API and event contract checks"]
    Image["Amazon ECR image"]
    Stage["Amazon ECS staging"]
    Smoke["Staging end-to-end tests"]
    Prod["Amazon ECS production"]
    RDS["Amazon RDS PostgreSQL"]
    SQS["Amazon SQS"]

    Dev --> PR --> CI
    CI --> Unit
    Unit --> Containers
    Containers --> Contract
    Contract --> Image
    Image --> Stage
    Stage --> RDS
    Stage --> SQS
    Stage --> Smoke
    Smoke --> Prod

The CI path uses disposable dependencies before staging, then promotes the tested image rather than rebuilding it.

Testcontainers shortens the distance between a test and the dependency it claims to verify. It does not make local PostgreSQL equivalent to every RDS setting, nor LocalStack equivalent to every SQS behavior. Use it to catch SQL, migration, serialization, and integration mistakes early; use staging to verify AWS identity, networking, configuration, and deployed composition.

4. Cloud service mapping

ComponentChosen service or toolAWS equivalentAzure equivalentGCP equivalent
API and worker runtimeManaged containersAmazon ECS on AWS FargateAzure Container AppsCloud Run
Container registryManaged registryAmazon ECRAzure Container RegistryArtifact Registry
Relational databaseManaged PostgreSQLAmazon RDS for PostgreSQLAzure Database for PostgreSQLCloud SQL for PostgreSQL
CacheManaged Redis-compatible cacheAmazon ElastiCacheAzure Managed RedisMemorystore
Work queueManaged queueAmazon SQSAzure Service BusCloud Tasks or Pub/Sub, depending on delivery model
CI orchestrationGitHub ActionsGitHub Actions with AWS OIDCGitHub Actions with federated identityGitHub Actions with Workload Identity Federation
Test dependenciesTestcontainers for .NETDocker containers on CI runnersDocker containers on CI runnersDocker containers on CI runners
Logs and metricsCloud monitoringAmazon CloudWatchAzure MonitorCloud Monitoring
OpenTelemetry collectionManaged collector integrationAWS Distro for OpenTelemetryAzure Monitor OpenTelemetry distributionGoogle-built OpenTelemetry components

The cloud equivalents are categories, not feature-for-feature matches. For example, Cloud Tasks and Pub/Sub have different delivery and subscription models; choose based on the application’s semantics rather than the table’s row.

5. Data design

The test strategy needs a deliberately small but meaningful relational model. An order references a customer, and an outbox row records the event to dispatch. The outbox is in the same database transaction as the order: a test that only asserts “the publisher was called” would not establish that atomicity.

erDiagram
    CUSTOMER ||--o{ ORDER : places
    ORDER ||--o{ OUTBOX_MESSAGE : produces
    CUSTOMER {
        uuid id PK
        text email UK
        timestamptz created_at
    }
    ORDER {
        uuid id PK
        uuid customer_id FK
        text status
        numeric total
        timestamptz created_at
    }
    OUTBOX_MESSAGE {
        uuid id PK
        uuid order_id FK
        text event_type
        jsonb payload
        timestamptz created_at
        timestamptz dispatched_at
    }

The outbox is persisted alongside the order so its creation can share the order transaction.

A representative schema follows. Production migrations should be versioned and applied by a controlled deployment step; tests should apply the same migrations as the application rather than maintain a separate test-only schema.

CREATE TABLE customer (
    id uuid PRIMARY KEY,
    email text NOT NULL UNIQUE,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE customer_order (
    id uuid PRIMARY KEY,
    customer_id uuid NOT NULL REFERENCES customer(id),
    status text NOT NULL CHECK (status IN ('pending', 'accepted', 'cancelled')),
    total numeric(12, 2) NOT NULL CHECK (total >= 0),
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox_message (
    id uuid PRIMARY KEY,
    order_id uuid NOT NULL REFERENCES customer_order(id),
    event_type text NOT NULL,
    payload jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    dispatched_at timestamptz NULL
);

CREATE INDEX ix_customer_order_customer_created
    ON customer_order (customer_id, created_at DESC);

CREATE INDEX ix_outbox_pending
    ON outbox_message (created_at)
    WHERE dispatched_at IS NULL;

The high-value database tests are not just happy-path inserts. They verify uniqueness, foreign keys, check constraints, transaction rollback, migration from the previous schema, and the query plan for known hot paths when data volume makes that relevant. Keep each test’s data isolated: disposable database per test class or per test worker is a good default at this scale. Do not share a mutable database between parallel tests unless the fixture provides explicit isolation.

6. Low-level design

The test suite is organized around behavior and dependency boundaries. Domain tests instantiate domain code directly. Persistence tests run the real PostgreSQL engine and real migrations. API tests host the application in-process while replacing its database connection with a container-backed connection. Contract tests compare published API schemas and event payloads with consumer expectations. End-to-end tests use the deployed staging URL and a dedicated test account.

classDiagram
    class OrderRules
    class OrderService
    class OrderRepository
    class OutboxWriter
    class ApiHost
    class PostgresFixture
    class ContractSuite
    class StagingE2ESuite

    OrderService --> OrderRules
    OrderService --> OrderRepository
    OrderService --> OutboxWriter
    ApiHost --> OrderService
    PostgresFixture --> OrderRepository
    ContractSuite --> ApiHost
    StagingE2ESuite --> ApiHost

Tests focus on explicit boundaries, from pure rules to the deployed application.

Minimal Testcontainers test

This minimal xUnit example starts PostgreSQL, opens a connection, and proves that the test is talking to a live database rather than a mock. Add the Testcontainers PostgreSQL module, Npgsql, and xUnit packages to the test project.

using Npgsql;
using Testcontainers.PostgreSql;
using Xunit;

public sealed class PostgreSqlSmokeTests
{
    [Fact]
    public async Task Container_accepts_a_database_connection()
    {
        await using var postgres = new PostgreSqlBuilder("postgres:16-alpine")
            .WithDatabase("app")
            .WithUsername("app")
            .WithPassword("test")
            .Build();

        await postgres.StartAsync();

        await using var connection = new NpgsqlConnection(postgres.GetConnectionString());
        await connection.OpenAsync();

        Assert.Equal(System.Data.ConnectionState.Open, connection.State);
    }
}

Notice that the test owns the container lifecycle and uses the container’s generated connection string rather than a developer machine’s database.

This is deliberately a smoke test, not a pattern to copy for every test method. Starting a container per method is simple but expensive. For a suite, create a shared fixture per test collection or per worker, apply migrations once, and reset data between tests with transactions, truncation, or isolated schemas. Choose one reset strategy and make parallel behavior explicit.

Realistic API integration test

The following example assumes the application exposes POST /orders, uses the ConnectionStrings:App setting, and applies migrations when its host starts. The test starts PostgreSQL once for the test class, hosts the actual ASP.NET Core application, inserts a customer, and checks the API response. The application’s Program type must be visible to the test project.

using System.Net;
using System.Net.Http.Json;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.Configuration;
using Npgsql;
using Testcontainers.PostgreSql;
using Xunit;

public sealed class OrderApiTests : IAsyncLifetime
{
    private readonly PostgreSqlContainer _postgres =
        new PostgreSqlBuilder("postgres:16-alpine")
            .WithDatabase("app")
            .WithUsername("app")
            .WithPassword("test")
            .Build();

    private WebApplicationFactory<Program>? _factory;
    private HttpClient? _client;

    public async Task InitializeAsync()
    {
        await _postgres.StartAsync();

        _factory = new WebApplicationFactory<Program>()
            .WithWebHostBuilder(builder =>
                builder.ConfigureAppConfiguration((_, config) =>
                    config.AddInMemoryCollection(new Dictionary<string, string?>
                    {
                        ["ConnectionStrings:App"] = _postgres.GetConnectionString()
                    })));

        _client = _factory.CreateClient();

        await using var connection = new NpgsqlConnection(_postgres.GetConnectionString());
        await connection.OpenAsync();

        await using var command = new NpgsqlCommand(
            "INSERT INTO customer (id, email) VALUES (@id, @email)", connection);
        command.Parameters.AddWithValue("id", Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"));
        command.Parameters.AddWithValue("email", "test@example.invalid");
        await command.ExecuteNonQueryAsync();
    }

    [Fact]
    public async Task Create_order_persists_and_returns_created()
    {
        var response = await _client!.PostAsJsonAsync("/orders", new
        {
            customerId = Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"),
            items = new[] { new { sku = "book-1", quantity = 2 } }
        });

        Assert.Equal(HttpStatusCode.Created, response.StatusCode);
        Assert.NotNull(response.Headers.Location);
    }

    public async Task DisposeAsync()
    {
        _client?.Dispose();
        _factory?.Dispose();
        await _postgres.DisposeAsync();
    }
}

Notice that the test crosses the HTTP, application, migration, and database boundaries, while keeping the external cloud out of the test.

This is an integration test, not an end-to-end test: the host is in-process and does not verify AWS networking, deployed identity, or the public load balancer. Its value is that it can catch invalid SQL, migration drift, dependency injection mistakes, and HTTP behavior cheaply. Keep those claims precise when reporting test results.

7. Key flows

Pull request validation

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub Actions
    participant Unit as Unit test job
    participant TC as Testcontainers
    participant Contract as Contract checks
    participant Review as Pull request

    Dev->>GH: Push commit
    GH->>Unit: Run unit tests
    Unit-->>GH: Results and coverage
    GH->>TC: Start PostgreSQL and queue test dependencies
    TC-->>GH: Integration test results
    GH->>Contract: Validate API and event contracts
    Contract-->>GH: Compatibility result
    GH-->>Review: Required checks pass or fail

Pull requests get fast feedback first, then validate dependency-backed behavior and contracts.

The order is a feedback optimization, not a statement that later tests are less important. Unit tests should fail quickly on domain mistakes; container-backed tests catch integration defects before merge. Contract checks should fail before a consumer discovers a breaking event or API change. A failed required check blocks merge; an engineer should not be able to wave away a red integration suite merely because the unit suite is green.

Staging release and promotion

sequenceDiagram
    participant Build as GitHub Actions
    participant ECR as Amazon ECR
    participant Stage as ECS staging
    participant Smoke as End-to-end suite
    participant Prod as ECS production
    participant Monitor as CloudWatch

    Build->>ECR: Push immutable image digest
    Build->>Stage: Deploy the digest
    Stage-->>Smoke: Health and readiness checks
    Smoke->>Stage: Create and retrieve test order
    Stage-->>Smoke: API and queue-backed result
    Smoke-->>Build: Pass or fail
    Build->>Prod: Promote the same digest
    Prod->>Monitor: Emit logs, metrics, and traces

Staging verifies deployment composition before the already-tested image is promoted.

Do not rebuild the image between staging and production. A rebuild creates a different artifact and weakens the evidence that the tested code is the released code. Staging tests should use dedicated accounts and data, clean up after themselves, and assert externally observable outcomes—not implementation details such as private method calls.

8. Build and release

Infrastructure as code

Use the same Terraform module for staging and production with different variables and state. Keep production credentials out of CI; configure GitHub Actions to assume an AWS role through OIDC. The snippet below is a partial RDS resource, not a complete deployable module: subnet groups, security groups, parameter groups, and secret access must be supplied by the surrounding infrastructure.

resource "aws_db_instance" "app" {
  identifier                  = "${var.environment}-orders-db"
  engine                      = "postgres"
  instance_class              = var.db_instance_class
  allocated_storage           = var.db_allocated_storage
  db_name                     = "orders"
  username                    = "app_user"
  manage_master_user_password = true

  multi_az                    = true
  storage_encrypted           = true
  publicly_accessible         = false
  backup_retention_period     = 14
  deletion_protection         = var.environment == "production"
  skip_final_snapshot         = false

  db_subnet_group_name        = aws_db_subnet_group.app.name
  vpc_security_group_ids      = [aws_security_group.db.id]

  tags = {
    Environment = var.environment
    Application = "orders"
  }
}

Notice that database availability and secret management are infrastructure concerns; a passing container test does not validate them.

Terraform plans should be reviewed and applied through a controlled workflow. Testcontainers tests should not create cloud infrastructure on every pull request. Reserve ephemeral AWS environments for cases that genuinely need AWS semantics, and give them automatic cleanup and budget controls.

CI/CD pipeline

This illustrative GitHub Actions workflow runs unit and integration tests on pull requests, builds and pushes an image on the main branch, and deploys it to staging. Production promotion is a protected environment step using the same image digest. The deployment actions require appropriately scoped OIDC roles, ECR, ECS task-definition files, and GitHub environment configuration.

name: orders-ci

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"
      - run: dotnet restore
      - run: dotnet test --no-restore --filter Category=Unit
      - run: dotnet test --no-restore --filter Category=Integration

  build-and-stage:
    if: github.event_name == 'push'
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}
      - uses: aws-actions/amazon-ecr-login@v2
      - name: Build and push image
        run: |
          IMAGE="${{ vars.ECR_REPOSITORY }}:${GITHUB_SHA}"
          docker build --tag "$IMAGE" .
          docker push "$IMAGE"
          echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
      - name: Deploy staging
        run: ./deploy-staging.sh "$IMAGE"
      - name: Run staging end-to-end tests
        run: dotnet test tests/Orders.E2E --logger "console;verbosity=normal"
        env:
          BASE_URL: ${{ vars.STAGING_BASE_URL }}

Notice that container-backed tests run on the CI runner, while staging tests use the deployed service and a configured base URL.

Pin dependencies and actions according to your organization’s supply-chain policy. Add an explicit contract job and a protected production promotion job in a production workflow; they are omitted here to keep the pipeline example readable. In practice, report test results and image digest as build artifacts, and avoid treating a skipped job as a passing check.

Test strategy by boundary

Test typeWhat it provesDependenciesWhen to runTypical failure caught
UnitA small rule or component behaves as specified.No network; substitute collaborators at boundaries.Every commit, first.Incorrect validation, pricing, state transitions.
IntegrationApplication code works with a real dependency or framework boundary.Testcontainers PostgreSQL; optionally LocalStack for selected queue tests.Every pull request.SQL or migration errors, serialization mismatches, transaction bugs.
ContractProducer and consumer agree on an interface or message shape.Schemas, generated clients, or a contract broker if used.Every pull request and provider release.Removed field, incompatible type, changed required semantics.
End-to-endA user-visible flow works in the deployed composition.Staging API, identity, queue, and test data.After staging deploy; a small smoke set after release.Bad configuration, identity, routing, deployment integration.
LoadCapacity and degradation behavior meet the stated target.Production-like staging environment and controlled load generator.Scheduled and before major capacity changes.Saturation, queuing, tail-latency growth, resource leaks.

Do not turn the test pyramid into a target percentage. A codebase with many database constraints may need more integration tests than one with mostly pure computation. Optimize for risk coverage and feedback time. Keep tests deterministic, make retries visible, and fix flaky tests rather than normalizing repeated reruns.

9. Observability

Testing infrastructure also needs observability. A CI job should report which dependency image and migration version it used, how long each test category took, and enough failure context to reproduce a problem. Application telemetry should let an operator distinguish a bad release from database saturation or a stuck queue. Use structured fields, not a sentence that must be parsed later.

Structured logging

An application log event can use ILogger structured properties and a JSON formatter. Avoid logging customer payloads, credentials, access tokens, or full payment details.

{
  "timestamp": "2026-11-20T12:00:00Z",
  "level": "Information",
  "event_name": "order_created",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "request_id": "req-7f31",
  "order_id": "b3135e4c-1a27-4fa3-9d79-1dded285d9ab",
  "result": "success",
  "duration_ms": 42
}

Use stable event names and correlation identifiers; keep personally identifying data out of routine logs.

RED and USE signals

Signal setResource or serviceKey measurements
REDHTTP APIRequest rate, error rate by route/status class, duration histogram and p95/p99.
REDQueue consumerMessages processed, processing failures, processing duration, age of oldest message.
USEECS tasksCPU utilization, memory utilization, task restarts, desired versus running tasks.
USERDS PostgreSQLCPU, storage, connections, I/O pressure, replica or failover events where applicable.
USERedisMemory, evictions, connections, cache hit/miss ratio.
USECI runnersJob duration, failure and retry counts, container startup failures, disk pressure.

Keep metric labels bounded. Do not label a metric with order IDs, customer IDs, or raw URLs. Traces and logs can carry carefully governed identifiers; metrics need a finite set of dimensions to stay useful and affordable.

OpenTelemetry tracing

Instrument the ASP.NET Core host and outbound HTTP clients. In production, export OTLP to an ADOT collector or an approved OpenTelemetry Collector endpoint, then route telemetry to the chosen backend. The test environment can use the same instrumentation with a local or CI exporter, but should not send test traffic into production telemetry.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource =>
        resource.AddService(
            serviceName: "orders-api",
            serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString()))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter());

var app = builder.Build();
app.MapGet("/health/ready", () => Results.Ok());
app.Run();

public partial class Program;

Trace the incoming request and outbound dependency calls together, and configure the OTLP endpoint through environment-specific settings.

A trace should connect an HTTP request to database or queue work where instrumentation supports it. Add explicit spans for business operations such as order submission and outbox dispatch, but avoid creating one span per trivial helper call. Propagate trace context through asynchronous messages where the chosen messaging instrumentation and message contract support it.

SLI, SLOs, dashboards, and alerts

SLISLO targetMeasurement windowNotes
Successful API requests99.9%Rolling 30 daysExclude intentional client validation errors from the availability denominator, and document the rule.
API latency95% under 300 msRolling 30 daysTrack by route class; avoid hiding slow endpoints in an aggregate.
Order event processing99% processed within 60 secondsRolling 30 daysMeasure from durable enqueue/outbox time to successful processing.
Release smoke success100% of production promotions have a passing staging runPer releaseA release gate, not a user-facing availability SLI.

Use separate dashboards for API RED, database/cache/compute USE, queue health, and release quality. A release dashboard should show test duration and failures by test category alongside deployed image digest. This helps identify whether slower feedback comes from the application, container startup, or a growing end-to-end suite.

Page on sustained customer impact: API error-budget burn, sustained latency SLO breach, a growing age of oldest queue message that threatens the processing objective, or a database failure that makes the API unavailable. Alert on symptoms, not every CPU excursion. Test-suite flakes and runner failures should notify the owning team through CI; they should page only when they block a release for a defined operational reason.

10. Security and compliance

Test environments are still environments that handle credentials, network access, and data. Make their boundaries explicit.

  • Use GitHub OIDC to assume short-lived AWS roles. Scope permissions to the necessary ECR repositories, ECS services, and staging resources.
  • Keep production deploy approval separate from pull-request execution. Untrusted fork code must not receive secrets or production-capable credentials.
  • Use synthetic test data. Do not copy production records into CI or staging unless a documented, approved process irreversibly masks sensitive fields.
  • Pin container images and dependencies according to policy. Use a supported PostgreSQL image, scan application images, and rebuild dependencies through a controlled update process.
  • Run containers with disposable credentials and networks. Do not mount a developer’s Docker socket into an application container or expose database ports publicly.
  • Test authorization and tenant isolation at the API boundary. A unit test of an authorization helper is not a substitute for verifying that an actual endpoint rejects a cross-tenant request.
  • Keep test logs free of secrets and sensitive payloads. Treat test artifacts, traces, and failure screenshots as potentially sensitive.
  • Make migrations additive where practical: expand schema, deploy compatible code, backfill, then remove obsolete schema in a later release. Test both the migration path and the application’s compatibility window.

Compliance requirements vary by sector and jurisdiction. Record which test data is synthetic, where CI logs are retained, and who can access staging. A passing test suite does not establish regulatory compliance; it provides evidence for specific technical controls.

11. Cost

This is an assumption-based monthly estimate, not a cloud quote. Prices depend on region, workload shape, retention, discounting, and service configuration. The table includes a modest production stack and always-on staging allowance appropriate to the stated medium scale; it excludes engineering labor and unusual data-transfer costs.

Cost areaAssumption-based monthly estimate (USD)Main cost drivers
ECS Fargate application and worker$150–400Task count, CPU/memory, running hours, scaling headroom.
Application Load Balancer and networking$80–250Load balancer hours, traffic, NAT gateways, data transfer.
Multi-AZ RDS PostgreSQL and backups$250–650Instance class, storage, I/O, backup retention.
ElastiCache Redis$60–200Node size, replicas, hours, data transfer.
SQS, CloudWatch logs and metrics$50–250Message volume, log ingestion, retention, custom metrics.
Staging runtime and database$150–500Whether staging is always on, and how production-like it is.
GitHub Actions and test execution$50–250Runner minutes, parallel jobs, container image pulls.
Illustrative total$790–2,500Assumption-based range; validate with a regional calculator and actual usage.

Testcontainers is not a separate managed-service charge, but its compute time, image downloads, and CI minutes are real costs. Optimize by sharing a container within a test collection, caching package layers safely, and keeping the end-to-end suite small. Do not cut critical dependency tests just to lower runner minutes; fix redundant coverage and slow fixture setup first.

12. Failure modes, resilience, and disaster recovery

Failure modeDetectionResponse and test evidence
Container startup or image pull fails in CITest job setup failure and runner logsSurface container logs and image tag; retry only transient infrastructure failures, not assertions.
PostgreSQL migration breaks after releaseMigration test from prior schema; staging deploy failureBlock promotion, repair forward or roll back code using the documented migration strategy.
Queue consumer is unavailableOldest-message age and DLQ growthScale or restore consumers; preserve messages and rehearse redrive procedures.
Redis fails or evicts heavilyCache errors, eviction rate, latencyFall back to the database for cacheable reads with bounded load; do not make the cache the source of truth.
A test passes locally but fails in CIEnvironment and dependency metadataPin supported images, remove hidden machine dependencies, and reproduce using the same test command.
Staging passes but production failsProduction RED signals and deployment comparisonPromote the same digest, compare configuration, and use a rollback or forward-fix decision based on data compatibility.
Availability Zone or database failureCloud health events and API errorsRely on Multi-AZ deployment and tested recovery procedures; verify recovery objectives through exercises.

Test resilience deliberately. Unit tests can verify retry and timeout decisions with a controllable clock or injected fault. Integration tests can verify transaction rollback and constraint behavior. Staging exercises can verify permissions, queue redrive, and deployment rollback. Do not make ordinary pull requests depend on an uncontrolled production-like failure injection environment.

Back up the database and verify restoration periodically. Define recovery point and recovery time objectives with the business rather than inferring them from a backup-retention setting. Queue messages and outbox rows need their own retention and replay policy. A backup that has never been restored is an unverified assumption.

13. Evolving to the next scale

Do not split the application into services because the test suite has become inconvenient. First improve the boundaries inside the modular monolith, fixture lifecycle, suite parallelism, and ownership of slow tests.

SignalFirst responseConsider a larger change when
Pull-request feedback exceeds its budgetReport duration by test class; parallelize independent suites; reuse container fixtures safely.A sustained increase remains after removing duplicate coverage and fixing fixture bottlenecks.
PostgreSQL test container becomes a bottleneckUse per-worker containers or isolated schemas; keep migrations representative.Specialized database test infrastructure is required for team throughput or stronger isolation.
Queue integration coverage misses provider behaviorAdd a small staging test against real SQS and verify IAM and DLQ policy.The queue is a critical independently owned service with separate lifecycle and release needs.
Staging is too different from productionCompare Terraform plans, environment settings, and runtime telemetry.More production-like ephemeral environments are justified by release risk and cost.
Teams need independent release cadenceDefine module ownership and stable contracts first.A module has an independent scaling, security, or deployment requirement that outweighs distributed-systems cost.

At higher scale, consider dedicated integration-test workers, a shared contract-testing service, and ephemeral cloud environments for selected releases. Those add operational complexity and need ownership. Keep Testcontainers for fast dependency-backed tests even if the deployment architecture later grows; the purpose is repeatable test dependencies, not a particular production topology.

14. Phased implementation plan

  1. Agree on boundaries and risk. List critical user journeys, database invariants, external contracts, and failure consequences. Assign each behavior to the narrowest useful test type. Establish explicit exclusions, such as tests that require live payment credentials.
  2. Build a fast unit-test baseline. Add pure domain tests for validation, pricing, state transitions, and event creation. Keep them independent of ASP.NET Core, PostgreSQL, and AWS.
  3. Introduce one Testcontainers fixture. Start with PostgreSQL and the application’s real migrations. Add tests for a persistence invariant and one HTTP-to-database path. Make fixture lifetime and parallelization policy explicit.
  4. Test asynchronous boundaries. Validate event schemas and producer/consumer compatibility. Use a containerized queue emulator where it provides useful local feedback, then retain a staging test for AWS-specific behavior.
  5. Make CI enforce the strategy. Order checks for fast feedback, publish clear test-category results, and require appropriate checks for merge. Track duration and flake rate before expanding the suite.
  6. Add safe staging promotion. Deploy one immutable image digest, run health checks and a small user-visible flow, then promote that digest through an approval gate.
  7. Instrument the system and the pipeline. Add structured logs, RED/USE dashboards, traces, SLOs, and actionable alerts. Give test failures enough context to reproduce without leaking secrets or customer data.
  8. Review evidence, not just coverage. After several releases, identify incidents or escaped defects the tests did not catch. Add the missing boundary test, improve the contract, or change the deployment check—without automatically adding another broad end-to-end test.

The result is a layered strategy with honest claims: unit tests establish domain behavior, Testcontainers-backed tests establish behavior against real dependencies, contract tests establish compatibility, and staging end-to-end tests establish that a deployed release works across its actual boundaries. A test suite is valuable when the team can explain what a green result proves—and what it does not.


This article was generated with AI. #AIGenerated