1. Context and scale
This blueprint is for a small SaaS team shipping an order-management API. It uses a modular monolith on ASP.NET Core, PostgreSQL as the system of record, Redis for cache-aside reads, and Amazon SQS for asynchronous order processing. The specific domain is illustrative; the testing strategy applies to any application whose correctness depends on database constraints, external interfaces, or asynchronous work.
The problem is not a shortage of tests. It is misplaced confidence: unit tests pass while SQL migrations fail, a mocked queue hides an incompatible message, or an end-to-end test becomes so slow and flaky that engineers stop trusting it. The architecture assigns each class of test a clear purpose, runs the cheapest useful checks early, and uses Testcontainers to run real dependencies in developer and CI environments.
The deployment target is AWS. GitHub Actions runs CI, builds an immutable container image, deploys it to an AWS staging environment, and runs a small end-to-end suite before production promotion. Testcontainers is for test infrastructure—not a substitute for staging, production-like cloud configuration, or provider-level verification.
| Assumption | Planning value |
|---|---|
| Daily active users | 10,000–100,000 |
| Request volume | 20 requests/s average; 150 requests/s peak |
| Data volume | 100 GB primary relational data, excluding backups |
| Team size | 1–2 teams; 6–12 engineers in total |
| Monthly cloud budget | US$1,000–3,000, including staging and test-related services |
| Availability target | 99.9% monthly for the order API |
| Application shape | One modular monolith, one asynchronous worker |
| Release frequency | Several production releases per week; assumptions, not a requirement |
These are sizing assumptions, not measured demand or a capacity guarantee. Revisit them with production telemetry. In particular, daily users alone do not predict peak concurrency, query cost, or queue depth.
The operating principle is to test a behavior at the narrowest boundary that can establish it. A pricing rule belongs in a unit test. PostgreSQL uniqueness belongs in a test against PostgreSQL. An HTTP message contract belongs in a consumer/provider contract test. Authentication wiring and the deployed route belong in a small end-to-end test against staging. Use the real dependency when the behavior under test belongs to that dependency; use a fake when the behavior belongs to your code.
2. Requirements
Functional requirements
| ID | Requirement | Testing implication |
|---|---|---|
| F1 | A customer can submit an order with one or more items. | Unit-test validation and pricing; API integration-test persistence and response semantics. |
| F2 | An order is stored with a stable identifier and initial status. | Verify persistence, transaction behavior, and database constraints against PostgreSQL. |
| F3 | A committed order produces an event for asynchronous processing. | Test the outbox transaction and message schema; verify dispatch to a queue in a separate integration boundary. |
| F4 | Clients can retrieve an order by identifier. | Test query behavior and API serialization; cover the deployed route with a smoke test. |
| F5 | API and worker releases remain compatible with the event contract. | Validate producer and consumer expectations independently, then run a small staging flow. |
Non-functional requirements
| ID | Requirement | Verification approach |
|---|---|---|
| N1 | API availability is at least 99.9% monthly. | Request-based SLI, production dashboard, and actionable paging thresholds. |
| N2 | Normal API responses have a p95 latency below 300 ms. | Staging smoke and scheduled load tests; production RED metrics are authoritative. |
| N3 | A release must not silently drop committed order events. | Database-backed outbox tests, queue delivery monitoring, and failure-injection exercises. |
| N4 | A pull request should give useful feedback within 10 minutes. | Keep unit and container-backed tests parallelizable; reserve broad load tests for scheduled or pre-release runs. |
| N5 | Schema changes are reversible or safely forward-compatible. | Migration tests from the previous schema and staged rollout checks. |
| N6 | Test data must not leak across test cases or into production. | Disposable containers or isolated schemas; no production credentials in test jobs. |
Latency and CI timing targets are goals to validate, not benchmark claims. A test suite that regularly exceeds its feedback budget needs better boundaries, parallelism, or fewer redundant scenarios—not a blanket increase in timeouts.
3. High-level design
The application runs on Amazon ECS Fargate behind an Application Load Balancer. PostgreSQL is Amazon RDS for PostgreSQL deployed across multiple Availability Zones. Redis caching uses Amazon ElastiCache, and asynchronous work uses Amazon SQS with a dead-letter queue. Images are stored in Amazon ECR. GitHub Actions owns CI orchestration; AWS CloudWatch and AWS Distro for OpenTelemetry (ADOT) provide the AWS-side telemetry path.
The testing flow is intentionally separate from the runtime flow. Pull requests run unit tests and Testcontainers-backed integration tests on an isolated CI runner. Contract checks validate API and event compatibility. A successful main-branch build is deployed to staging, where a small set of end-to-end tests verifies the integrated deployment. Production promotion uses the same image digest that passed staging.
flowchart LR
Dev["Developer"]
PR["GitHub pull request"]
CI["GitHub Actions"]
Unit["Unit tests"]
Containers["Testcontainers: PostgreSQL and LocalStack"]
Contract["API and event contract checks"]
Image["Amazon ECR image"]
Stage["Amazon ECS staging"]
Smoke["Staging end-to-end tests"]
Prod["Amazon ECS production"]
RDS["Amazon RDS PostgreSQL"]
SQS["Amazon SQS"]
Dev --> PR --> CI
CI --> Unit
Unit --> Containers
Containers --> Contract
Contract --> Image
Image --> Stage
Stage --> RDS
Stage --> SQS
Stage --> Smoke
Smoke --> ProdThe CI path uses disposable dependencies before staging, then promotes the tested image rather than rebuilding it.
Testcontainers shortens the distance between a test and the dependency it claims to verify. It does not make local PostgreSQL equivalent to every RDS setting, nor LocalStack equivalent to every SQS behavior. Use it to catch SQL, migration, serialization, and integration mistakes early; use staging to verify AWS identity, networking, configuration, and deployed composition.
4. Cloud service mapping
| Component | Chosen service or tool | AWS equivalent | Azure equivalent | GCP equivalent |
|---|---|---|---|---|
| API and worker runtime | Managed containers | Amazon ECS on AWS Fargate | Azure Container Apps | Cloud Run |
| Container registry | Managed registry | Amazon ECR | Azure Container Registry | Artifact Registry |
| Relational database | Managed PostgreSQL | Amazon RDS for PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Cache | Managed Redis-compatible cache | Amazon ElastiCache | Azure Managed Redis | Memorystore |
| Work queue | Managed queue | Amazon SQS | Azure Service Bus | Cloud Tasks or Pub/Sub, depending on delivery model |
| CI orchestration | GitHub Actions | GitHub Actions with AWS OIDC | GitHub Actions with federated identity | GitHub Actions with Workload Identity Federation |
| Test dependencies | Testcontainers for .NET | Docker containers on CI runners | Docker containers on CI runners | Docker containers on CI runners |
| Logs and metrics | Cloud monitoring | Amazon CloudWatch | Azure Monitor | Cloud Monitoring |
| OpenTelemetry collection | Managed collector integration | AWS Distro for OpenTelemetry | Azure Monitor OpenTelemetry distribution | Google-built OpenTelemetry components |
The cloud equivalents are categories, not feature-for-feature matches. For example, Cloud Tasks and Pub/Sub have different delivery and subscription models; choose based on the application’s semantics rather than the table’s row.
5. Data design
The test strategy needs a deliberately small but meaningful relational model. An order references a customer, and an outbox row records the event to dispatch. The outbox is in the same database transaction as the order: a test that only asserts “the publisher was called” would not establish that atomicity.
erDiagram
CUSTOMER ||--o{ ORDER : places
ORDER ||--o{ OUTBOX_MESSAGE : produces
CUSTOMER {
uuid id PK
text email UK
timestamptz created_at
}
ORDER {
uuid id PK
uuid customer_id FK
text status
numeric total
timestamptz created_at
}
OUTBOX_MESSAGE {
uuid id PK
uuid order_id FK
text event_type
jsonb payload
timestamptz created_at
timestamptz dispatched_at
}The outbox is persisted alongside the order so its creation can share the order transaction.
A representative schema follows. Production migrations should be versioned and applied by a controlled deployment step; tests should apply the same migrations as the application rather than maintain a separate test-only schema.
CREATE TABLE customer (
id uuid PRIMARY KEY,
email text NOT NULL UNIQUE,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE customer_order (
id uuid PRIMARY KEY,
customer_id uuid NOT NULL REFERENCES customer(id),
status text NOT NULL CHECK (status IN ('pending', 'accepted', 'cancelled')),
total numeric(12, 2) NOT NULL CHECK (total >= 0),
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE outbox_message (
id uuid PRIMARY KEY,
order_id uuid NOT NULL REFERENCES customer_order(id),
event_type text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
dispatched_at timestamptz NULL
);
CREATE INDEX ix_customer_order_customer_created
ON customer_order (customer_id, created_at DESC);
CREATE INDEX ix_outbox_pending
ON outbox_message (created_at)
WHERE dispatched_at IS NULL;
The high-value database tests are not just happy-path inserts. They verify uniqueness, foreign keys, check constraints, transaction rollback, migration from the previous schema, and the query plan for known hot paths when data volume makes that relevant. Keep each test’s data isolated: disposable database per test class or per test worker is a good default at this scale. Do not share a mutable database between parallel tests unless the fixture provides explicit isolation.
6. Low-level design
The test suite is organized around behavior and dependency boundaries. Domain tests instantiate domain code directly. Persistence tests run the real PostgreSQL engine and real migrations. API tests host the application in-process while replacing its database connection with a container-backed connection. Contract tests compare published API schemas and event payloads with consumer expectations. End-to-end tests use the deployed staging URL and a dedicated test account.
classDiagram
class OrderRules
class OrderService
class OrderRepository
class OutboxWriter
class ApiHost
class PostgresFixture
class ContractSuite
class StagingE2ESuite
OrderService --> OrderRules
OrderService --> OrderRepository
OrderService --> OutboxWriter
ApiHost --> OrderService
PostgresFixture --> OrderRepository
ContractSuite --> ApiHost
StagingE2ESuite --> ApiHostTests focus on explicit boundaries, from pure rules to the deployed application.
Minimal Testcontainers test
This minimal xUnit example starts PostgreSQL, opens a connection, and proves that the test is talking to a live database rather than a mock. Add the Testcontainers PostgreSQL module, Npgsql, and xUnit packages to the test project.
using Npgsql;
using Testcontainers.PostgreSql;
using Xunit;
public sealed class PostgreSqlSmokeTests
{
[Fact]
public async Task Container_accepts_a_database_connection()
{
await using var postgres = new PostgreSqlBuilder("postgres:16-alpine")
.WithDatabase("app")
.WithUsername("app")
.WithPassword("test")
.Build();
await postgres.StartAsync();
await using var connection = new NpgsqlConnection(postgres.GetConnectionString());
await connection.OpenAsync();
Assert.Equal(System.Data.ConnectionState.Open, connection.State);
}
}
Notice that the test owns the container lifecycle and uses the container’s generated connection string rather than a developer machine’s database.
This is deliberately a smoke test, not a pattern to copy for every test method. Starting a container per method is simple but expensive. For a suite, create a shared fixture per test collection or per worker, apply migrations once, and reset data between tests with transactions, truncation, or isolated schemas. Choose one reset strategy and make parallel behavior explicit.
Realistic API integration test
The following example assumes the application exposes POST /orders, uses the ConnectionStrings:App setting, and applies migrations when its host starts. The test starts PostgreSQL once for the test class, hosts the actual ASP.NET Core application, inserts a customer, and checks the API response. The application’s Program type must be visible to the test project.
using System.Net;
using System.Net.Http.Json;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.Configuration;
using Npgsql;
using Testcontainers.PostgreSql;
using Xunit;
public sealed class OrderApiTests : IAsyncLifetime
{
private readonly PostgreSqlContainer _postgres =
new PostgreSqlBuilder("postgres:16-alpine")
.WithDatabase("app")
.WithUsername("app")
.WithPassword("test")
.Build();
private WebApplicationFactory<Program>? _factory;
private HttpClient? _client;
public async Task InitializeAsync()
{
await _postgres.StartAsync();
_factory = new WebApplicationFactory<Program>()
.WithWebHostBuilder(builder =>
builder.ConfigureAppConfiguration((_, config) =>
config.AddInMemoryCollection(new Dictionary<string, string?>
{
["ConnectionStrings:App"] = _postgres.GetConnectionString()
})));
_client = _factory.CreateClient();
await using var connection = new NpgsqlConnection(_postgres.GetConnectionString());
await connection.OpenAsync();
await using var command = new NpgsqlCommand(
"INSERT INTO customer (id, email) VALUES (@id, @email)", connection);
command.Parameters.AddWithValue("id", Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"));
command.Parameters.AddWithValue("email", "test@example.invalid");
await command.ExecuteNonQueryAsync();
}
[Fact]
public async Task Create_order_persists_and_returns_created()
{
var response = await _client!.PostAsJsonAsync("/orders", new
{
customerId = Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"),
items = new[] { new { sku = "book-1", quantity = 2 } }
});
Assert.Equal(HttpStatusCode.Created, response.StatusCode);
Assert.NotNull(response.Headers.Location);
}
public async Task DisposeAsync()
{
_client?.Dispose();
_factory?.Dispose();
await _postgres.DisposeAsync();
}
}
Notice that the test crosses the HTTP, application, migration, and database boundaries, while keeping the external cloud out of the test.
This is an integration test, not an end-to-end test: the host is in-process and does not verify AWS networking, deployed identity, or the public load balancer. Its value is that it can catch invalid SQL, migration drift, dependency injection mistakes, and HTTP behavior cheaply. Keep those claims precise when reporting test results.
7. Key flows
Pull request validation
sequenceDiagram
participant Dev as Developer
participant GH as GitHub Actions
participant Unit as Unit test job
participant TC as Testcontainers
participant Contract as Contract checks
participant Review as Pull request
Dev->>GH: Push commit
GH->>Unit: Run unit tests
Unit-->>GH: Results and coverage
GH->>TC: Start PostgreSQL and queue test dependencies
TC-->>GH: Integration test results
GH->>Contract: Validate API and event contracts
Contract-->>GH: Compatibility result
GH-->>Review: Required checks pass or failPull requests get fast feedback first, then validate dependency-backed behavior and contracts.
The order is a feedback optimization, not a statement that later tests are less important. Unit tests should fail quickly on domain mistakes; container-backed tests catch integration defects before merge. Contract checks should fail before a consumer discovers a breaking event or API change. A failed required check blocks merge; an engineer should not be able to wave away a red integration suite merely because the unit suite is green.
Staging release and promotion
sequenceDiagram
participant Build as GitHub Actions
participant ECR as Amazon ECR
participant Stage as ECS staging
participant Smoke as End-to-end suite
participant Prod as ECS production
participant Monitor as CloudWatch
Build->>ECR: Push immutable image digest
Build->>Stage: Deploy the digest
Stage-->>Smoke: Health and readiness checks
Smoke->>Stage: Create and retrieve test order
Stage-->>Smoke: API and queue-backed result
Smoke-->>Build: Pass or fail
Build->>Prod: Promote the same digest
Prod->>Monitor: Emit logs, metrics, and tracesStaging verifies deployment composition before the already-tested image is promoted.
Do not rebuild the image between staging and production. A rebuild creates a different artifact and weakens the evidence that the tested code is the released code. Staging tests should use dedicated accounts and data, clean up after themselves, and assert externally observable outcomes—not implementation details such as private method calls.
8. Build and release
Infrastructure as code
Use the same Terraform module for staging and production with different variables and state. Keep production credentials out of CI; configure GitHub Actions to assume an AWS role through OIDC. The snippet below is a partial RDS resource, not a complete deployable module: subnet groups, security groups, parameter groups, and secret access must be supplied by the surrounding infrastructure.
resource "aws_db_instance" "app" {
identifier = "${var.environment}-orders-db"
engine = "postgres"
instance_class = var.db_instance_class
allocated_storage = var.db_allocated_storage
db_name = "orders"
username = "app_user"
manage_master_user_password = true
multi_az = true
storage_encrypted = true
publicly_accessible = false
backup_retention_period = 14
deletion_protection = var.environment == "production"
skip_final_snapshot = false
db_subnet_group_name = aws_db_subnet_group.app.name
vpc_security_group_ids = [aws_security_group.db.id]
tags = {
Environment = var.environment
Application = "orders"
}
}
Notice that database availability and secret management are infrastructure concerns; a passing container test does not validate them.
Terraform plans should be reviewed and applied through a controlled workflow. Testcontainers tests should not create cloud infrastructure on every pull request. Reserve ephemeral AWS environments for cases that genuinely need AWS semantics, and give them automatic cleanup and budget controls.
CI/CD pipeline
This illustrative GitHub Actions workflow runs unit and integration tests on pull requests, builds and pushes an image on the main branch, and deploys it to staging. Production promotion is a protected environment step using the same image digest. The deployment actions require appropriately scoped OIDC roles, ECR, ECS task-definition files, and GitHub environment configuration.
name: orders-ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.0.x"
- run: dotnet restore
- run: dotnet test --no-restore --filter Category=Unit
- run: dotnet test --no-restore --filter Category=Integration
build-and-stage:
if: github.event_name == 'push'
needs: test
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.0.x"
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- uses: aws-actions/amazon-ecr-login@v2
- name: Build and push image
run: |
IMAGE="${{ vars.ECR_REPOSITORY }}:${GITHUB_SHA}"
docker build --tag "$IMAGE" .
docker push "$IMAGE"
echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
- name: Deploy staging
run: ./deploy-staging.sh "$IMAGE"
- name: Run staging end-to-end tests
run: dotnet test tests/Orders.E2E --logger "console;verbosity=normal"
env:
BASE_URL: ${{ vars.STAGING_BASE_URL }}
Notice that container-backed tests run on the CI runner, while staging tests use the deployed service and a configured base URL.
Pin dependencies and actions according to your organization’s supply-chain policy. Add an explicit contract job and a protected production promotion job in a production workflow; they are omitted here to keep the pipeline example readable. In practice, report test results and image digest as build artifacts, and avoid treating a skipped job as a passing check.
Test strategy by boundary
| Test type | What it proves | Dependencies | When to run | Typical failure caught |
|---|---|---|---|---|
| Unit | A small rule or component behaves as specified. | No network; substitute collaborators at boundaries. | Every commit, first. | Incorrect validation, pricing, state transitions. |
| Integration | Application code works with a real dependency or framework boundary. | Testcontainers PostgreSQL; optionally LocalStack for selected queue tests. | Every pull request. | SQL or migration errors, serialization mismatches, transaction bugs. |
| Contract | Producer and consumer agree on an interface or message shape. | Schemas, generated clients, or a contract broker if used. | Every pull request and provider release. | Removed field, incompatible type, changed required semantics. |
| End-to-end | A user-visible flow works in the deployed composition. | Staging API, identity, queue, and test data. | After staging deploy; a small smoke set after release. | Bad configuration, identity, routing, deployment integration. |
| Load | Capacity and degradation behavior meet the stated target. | Production-like staging environment and controlled load generator. | Scheduled and before major capacity changes. | Saturation, queuing, tail-latency growth, resource leaks. |
Do not turn the test pyramid into a target percentage. A codebase with many database constraints may need more integration tests than one with mostly pure computation. Optimize for risk coverage and feedback time. Keep tests deterministic, make retries visible, and fix flaky tests rather than normalizing repeated reruns.
9. Observability
Testing infrastructure also needs observability. A CI job should report which dependency image and migration version it used, how long each test category took, and enough failure context to reproduce a problem. Application telemetry should let an operator distinguish a bad release from database saturation or a stuck queue. Use structured fields, not a sentence that must be parsed later.
Structured logging
An application log event can use ILogger structured properties and a JSON formatter. Avoid logging customer payloads, credentials, access tokens, or full payment details.
{
"timestamp": "2026-11-20T12:00:00Z",
"level": "Information",
"event_name": "order_created",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"request_id": "req-7f31",
"order_id": "b3135e4c-1a27-4fa3-9d79-1dded285d9ab",
"result": "success",
"duration_ms": 42
}
Use stable event names and correlation identifiers; keep personally identifying data out of routine logs.
RED and USE signals
| Signal set | Resource or service | Key measurements |
|---|---|---|
| RED | HTTP API | Request rate, error rate by route/status class, duration histogram and p95/p99. |
| RED | Queue consumer | Messages processed, processing failures, processing duration, age of oldest message. |
| USE | ECS tasks | CPU utilization, memory utilization, task restarts, desired versus running tasks. |
| USE | RDS PostgreSQL | CPU, storage, connections, I/O pressure, replica or failover events where applicable. |
| USE | Redis | Memory, evictions, connections, cache hit/miss ratio. |
| USE | CI runners | Job duration, failure and retry counts, container startup failures, disk pressure. |
Keep metric labels bounded. Do not label a metric with order IDs, customer IDs, or raw URLs. Traces and logs can carry carefully governed identifiers; metrics need a finite set of dimensions to stay useful and affordable.
OpenTelemetry tracing
Instrument the ASP.NET Core host and outbound HTTP clients. In production, export OTLP to an ADOT collector or an approved OpenTelemetry Collector endpoint, then route telemetry to the chosen backend. The test environment can use the same instrumentation with a local or CI exporter, but should not send test traffic into production telemetry.
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource =>
resource.AddService(
serviceName: "orders-api",
serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString()))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddOtlpExporter());
var app = builder.Build();
app.MapGet("/health/ready", () => Results.Ok());
app.Run();
public partial class Program;
Trace the incoming request and outbound dependency calls together, and configure the OTLP endpoint through environment-specific settings.
A trace should connect an HTTP request to database or queue work where instrumentation supports it. Add explicit spans for business operations such as order submission and outbox dispatch, but avoid creating one span per trivial helper call. Propagate trace context through asynchronous messages where the chosen messaging instrumentation and message contract support it.
SLI, SLOs, dashboards, and alerts
| SLI | SLO target | Measurement window | Notes |
|---|---|---|---|
| Successful API requests | 99.9% | Rolling 30 days | Exclude intentional client validation errors from the availability denominator, and document the rule. |
| API latency | 95% under 300 ms | Rolling 30 days | Track by route class; avoid hiding slow endpoints in an aggregate. |
| Order event processing | 99% processed within 60 seconds | Rolling 30 days | Measure from durable enqueue/outbox time to successful processing. |
| Release smoke success | 100% of production promotions have a passing staging run | Per release | A release gate, not a user-facing availability SLI. |
Use separate dashboards for API RED, database/cache/compute USE, queue health, and release quality. A release dashboard should show test duration and failures by test category alongside deployed image digest. This helps identify whether slower feedback comes from the application, container startup, or a growing end-to-end suite.
Page on sustained customer impact: API error-budget burn, sustained latency SLO breach, a growing age of oldest queue message that threatens the processing objective, or a database failure that makes the API unavailable. Alert on symptoms, not every CPU excursion. Test-suite flakes and runner failures should notify the owning team through CI; they should page only when they block a release for a defined operational reason.
10. Security and compliance
Test environments are still environments that handle credentials, network access, and data. Make their boundaries explicit.
- Use GitHub OIDC to assume short-lived AWS roles. Scope permissions to the necessary ECR repositories, ECS services, and staging resources.
- Keep production deploy approval separate from pull-request execution. Untrusted fork code must not receive secrets or production-capable credentials.
- Use synthetic test data. Do not copy production records into CI or staging unless a documented, approved process irreversibly masks sensitive fields.
- Pin container images and dependencies according to policy. Use a supported PostgreSQL image, scan application images, and rebuild dependencies through a controlled update process.
- Run containers with disposable credentials and networks. Do not mount a developer’s Docker socket into an application container or expose database ports publicly.
- Test authorization and tenant isolation at the API boundary. A unit test of an authorization helper is not a substitute for verifying that an actual endpoint rejects a cross-tenant request.
- Keep test logs free of secrets and sensitive payloads. Treat test artifacts, traces, and failure screenshots as potentially sensitive.
- Make migrations additive where practical: expand schema, deploy compatible code, backfill, then remove obsolete schema in a later release. Test both the migration path and the application’s compatibility window.
Compliance requirements vary by sector and jurisdiction. Record which test data is synthetic, where CI logs are retained, and who can access staging. A passing test suite does not establish regulatory compliance; it provides evidence for specific technical controls.
11. Cost
This is an assumption-based monthly estimate, not a cloud quote. Prices depend on region, workload shape, retention, discounting, and service configuration. The table includes a modest production stack and always-on staging allowance appropriate to the stated medium scale; it excludes engineering labor and unusual data-transfer costs.
| Cost area | Assumption-based monthly estimate (USD) | Main cost drivers |
|---|---|---|
| ECS Fargate application and worker | $150–400 | Task count, CPU/memory, running hours, scaling headroom. |
| Application Load Balancer and networking | $80–250 | Load balancer hours, traffic, NAT gateways, data transfer. |
| Multi-AZ RDS PostgreSQL and backups | $250–650 | Instance class, storage, I/O, backup retention. |
| ElastiCache Redis | $60–200 | Node size, replicas, hours, data transfer. |
| SQS, CloudWatch logs and metrics | $50–250 | Message volume, log ingestion, retention, custom metrics. |
| Staging runtime and database | $150–500 | Whether staging is always on, and how production-like it is. |
| GitHub Actions and test execution | $50–250 | Runner minutes, parallel jobs, container image pulls. |
| Illustrative total | $790–2,500 | Assumption-based range; validate with a regional calculator and actual usage. |
Testcontainers is not a separate managed-service charge, but its compute time, image downloads, and CI minutes are real costs. Optimize by sharing a container within a test collection, caching package layers safely, and keeping the end-to-end suite small. Do not cut critical dependency tests just to lower runner minutes; fix redundant coverage and slow fixture setup first.
12. Failure modes, resilience, and disaster recovery
| Failure mode | Detection | Response and test evidence |
|---|---|---|
| Container startup or image pull fails in CI | Test job setup failure and runner logs | Surface container logs and image tag; retry only transient infrastructure failures, not assertions. |
| PostgreSQL migration breaks after release | Migration test from prior schema; staging deploy failure | Block promotion, repair forward or roll back code using the documented migration strategy. |
| Queue consumer is unavailable | Oldest-message age and DLQ growth | Scale or restore consumers; preserve messages and rehearse redrive procedures. |
| Redis fails or evicts heavily | Cache errors, eviction rate, latency | Fall back to the database for cacheable reads with bounded load; do not make the cache the source of truth. |
| A test passes locally but fails in CI | Environment and dependency metadata | Pin supported images, remove hidden machine dependencies, and reproduce using the same test command. |
| Staging passes but production fails | Production RED signals and deployment comparison | Promote the same digest, compare configuration, and use a rollback or forward-fix decision based on data compatibility. |
| Availability Zone or database failure | Cloud health events and API errors | Rely on Multi-AZ deployment and tested recovery procedures; verify recovery objectives through exercises. |
Test resilience deliberately. Unit tests can verify retry and timeout decisions with a controllable clock or injected fault. Integration tests can verify transaction rollback and constraint behavior. Staging exercises can verify permissions, queue redrive, and deployment rollback. Do not make ordinary pull requests depend on an uncontrolled production-like failure injection environment.
Back up the database and verify restoration periodically. Define recovery point and recovery time objectives with the business rather than inferring them from a backup-retention setting. Queue messages and outbox rows need their own retention and replay policy. A backup that has never been restored is an unverified assumption.
13. Evolving to the next scale
Do not split the application into services because the test suite has become inconvenient. First improve the boundaries inside the modular monolith, fixture lifecycle, suite parallelism, and ownership of slow tests.
| Signal | First response | Consider a larger change when |
|---|---|---|
| Pull-request feedback exceeds its budget | Report duration by test class; parallelize independent suites; reuse container fixtures safely. | A sustained increase remains after removing duplicate coverage and fixing fixture bottlenecks. |
| PostgreSQL test container becomes a bottleneck | Use per-worker containers or isolated schemas; keep migrations representative. | Specialized database test infrastructure is required for team throughput or stronger isolation. |
| Queue integration coverage misses provider behavior | Add a small staging test against real SQS and verify IAM and DLQ policy. | The queue is a critical independently owned service with separate lifecycle and release needs. |
| Staging is too different from production | Compare Terraform plans, environment settings, and runtime telemetry. | More production-like ephemeral environments are justified by release risk and cost. |
| Teams need independent release cadence | Define module ownership and stable contracts first. | A module has an independent scaling, security, or deployment requirement that outweighs distributed-systems cost. |
At higher scale, consider dedicated integration-test workers, a shared contract-testing service, and ephemeral cloud environments for selected releases. Those add operational complexity and need ownership. Keep Testcontainers for fast dependency-backed tests even if the deployment architecture later grows; the purpose is repeatable test dependencies, not a particular production topology.
14. Phased implementation plan
- Agree on boundaries and risk. List critical user journeys, database invariants, external contracts, and failure consequences. Assign each behavior to the narrowest useful test type. Establish explicit exclusions, such as tests that require live payment credentials.
- Build a fast unit-test baseline. Add pure domain tests for validation, pricing, state transitions, and event creation. Keep them independent of ASP.NET Core, PostgreSQL, and AWS.
- Introduce one Testcontainers fixture. Start with PostgreSQL and the application’s real migrations. Add tests for a persistence invariant and one HTTP-to-database path. Make fixture lifetime and parallelization policy explicit.
- Test asynchronous boundaries. Validate event schemas and producer/consumer compatibility. Use a containerized queue emulator where it provides useful local feedback, then retain a staging test for AWS-specific behavior.
- Make CI enforce the strategy. Order checks for fast feedback, publish clear test-category results, and require appropriate checks for merge. Track duration and flake rate before expanding the suite.
- Add safe staging promotion. Deploy one immutable image digest, run health checks and a small user-visible flow, then promote that digest through an approval gate.
- Instrument the system and the pipeline. Add structured logs, RED/USE dashboards, traces, SLOs, and actionable alerts. Give test failures enough context to reproduce without leaking secrets or customer data.
- Review evidence, not just coverage. After several releases, identify incidents or escaped defects the tests did not catch. Add the missing boundary test, improve the contract, or change the deployment check—without automatically adding another broad end-to-end test.
The result is a layered strategy with honest claims: unit tests establish domain behavior, Testcontainers-backed tests establish behavior against real dependencies, contract tests establish compatibility, and staging end-to-end tests establish that a deployed release works across its actual boundaries. A test suite is valuable when the team can explain what a green result proves—and what it does not.
This article was generated with AI. #AIGenerated