1. Context and scale
This blueprint describes a multi-tenant SaaS backend on Google Cloud Platform. It serves a browser or mobile client through an HTTP API, stores transactional state in PostgreSQL, caches selected reads in Redis, and handles both event-driven work and scheduled or rate-limited tasks.
The architecture is deliberately sized for one or two product teams. It uses a modular monolith for the public API and a separate worker service, rather than turning each domain module into a deployable service. Pub/Sub distributes business events; Cloud Tasks schedules explicit HTTP work with per-queue dispatch controls. Those are different jobs, and using both is justified only when the product needs both semantics.
Assumptions
These are planning assumptions, not product limits or capacity guarantees. Validate them with workload tests and current regional pricing before committing to a production budget.
| Dimension | Assumption |
|---|---|
| Users | 50,000 daily active users; 10,000–100,000 is the intended range |
| Request rate | 5 requests/s average; 50 requests/s peak, with short bursts above that |
| Data volume | 300 GB of primary PostgreSQL data after one year, plus backups |
| Workload | Mostly interactive reads and writes; asynchronous notifications, exports, and integrations |
| Team size | 1–2 teams, approximately 5–10 engineers total, with a rotating operations owner |
| Availability target | 99.9% monthly API availability for the initial production service |
| Budget | Planning envelope of US$1,000–2,000/month before taxes, support, and unusually high egress or logging |
The design assumes one primary GCP region, private connectivity from Cloud Run to data services, and a staging environment that is smaller than production. It does not assume that every component is multi-region or that a regional failure is automatically transparent to users.
2. Requirements
The first release should optimize for clear ownership, bounded operational work, and correct asynchronous processing. Do not add a service merely because it is available: each additional queue, database, and deployment unit adds failure modes and lifecycle work.
Functional requirements
| Requirement | Design response |
|---|---|
| Authenticate users and resolve tenant context | API validates identity tokens and derives tenant identity from trusted claims or server-side membership lookup |
| Serve tenant-scoped CRUD operations | ASP.NET Core API with PostgreSQL transactions and tenant filters enforced in application queries |
| Cache frequently read, non-authoritative data | Memorystore for Redis; PostgreSQL remains the source of truth |
| Publish business events after commits | Transactional outbox written with domain changes, then relayed to Pub/Sub |
| Run scheduled, retryable, or rate-limited work | Cloud Tasks delivers authenticated HTTP requests to a private worker endpoint |
| Fan out domain events to subscribers | Pub/Sub topics and subscriptions, with independently deployable consumers when needed |
| Provide operational visibility | Cloud Logging, Cloud Monitoring, Cloud Trace, and OpenTelemetry instrumentation |
Non-functional requirements
| Requirement | Target or constraint |
|---|---|
| Availability | 99.9% monthly API SLO; dependencies and deployment configuration must support regional recovery |
| Latency | API p95 under 300 ms for ordinary database-backed requests, excluding client network time |
| Durability | A successful API write is committed to Cloud SQL before returning success |
| Tenant isolation | Every tenant-owned row and cache key is tenant-scoped; authorization is checked server-side |
| Retry safety | Queue delivery is treated as at-least-once; handlers must be idempotent |
| Scaling | API and worker instances scale independently within configured concurrency and maximum-instance bounds |
| Operations | One team can deploy, diagnose, and roll back without coordinating a fleet of independently owned services |
3. High-level design
The public API is a Cloud Run service. It handles synchronous requests and writes domain state to Cloud SQL for PostgreSQL. A transactional outbox row is committed in the same database transaction as the domain change; a relay publishes it to Pub/Sub and records delivery.
Memorystore reduces repeated reads for suitable data, but it is not a correctness dependency. Pub/Sub carries events that can have multiple subscribers. Cloud Tasks carries commands to a specific HTTP handler, especially when dispatch rate, scheduling, or per-task retry behavior matters. Both call a separate Cloud Run worker service using authenticated service identities.
flowchart LR
Client["Web or mobile client"] --> Api["Cloud Run API"]
Api --> Db["Cloud SQL for PostgreSQL"]
Api --> Cache["Memorystore for Redis"]
Api --> Outbox["Transactional outbox"]
Outbox --> Relay["Cloud Run outbox relay"]
Relay --> PubSub["Pub/Sub topic"]
PubSub --> Worker["Cloud Run worker"]
Api --> Tasks["Cloud Tasks queue"]
Tasks --> Worker
Worker --> Db
Worker --> Cache
Api --> Telemetry["Cloud Logging and Monitoring"]
Worker --> TelemetryCaption: The API owns synchronous transactions; Pub/Sub distributes events, while Cloud Tasks dispatches targeted work.
Keep the relay in the worker deployment initially if its lifecycle and scaling needs are similar. Split it into a separate Cloud Run service only when its throughput, permissions, or release cadence requires independent control. Avoid a continuously polling process if a scheduled trigger or bounded relay invocation can meet the outbox latency target.
4. Cloud service mapping
| Component | GCP choice | AWS equivalent | Azure equivalent | Reason for choice |
|---|---|---|---|---|
| HTTP API and worker | Cloud Run | AWS App Runner or ECS on Fargate | Azure Container Apps | Runs containers with independent scaling and managed HTTPS integration |
| Relational database | Cloud SQL for PostgreSQL | Amazon RDS for PostgreSQL | Azure Database for PostgreSQL | Managed PostgreSQL with backups and regional high-availability options |
| Cache | Memorystore for Redis | ElastiCache for Redis OSS | Azure Managed Redis | Low-latency cache accessed over private networking |
| Event distribution | Pub/Sub | Amazon SNS and SQS, depending on fan-out pattern | Azure Service Bus topics | Durable event delivery to independent subscribers |
| Scheduled and rate-controlled HTTP work | Cloud Tasks | Amazon SQS with a worker, or EventBridge Scheduler plus a target | Azure Queue Storage or Service Bus with a worker | Explicit HTTP target, scheduling, retry, and dispatch-rate controls |
| Container image storage | Artifact Registry | Amazon ECR | Azure Container Registry | Private image storage integrated with deployment identities |
| Secrets | Secret Manager | AWS Secrets Manager | Azure Key Vault | Centralized secret storage and access auditing |
| Telemetry | Cloud Logging, Monitoring, Trace | CloudWatch and X-Ray | Azure Monitor and Application Insights | Native GCP collection and alerting |
Cloud Run is the right default here because the workload is request-driven and the team wants to operate containers without maintaining a cluster. Cloud Tasks and Pub/Sub are not interchangeable: Cloud Tasks is a dispatch mechanism for explicit work; Pub/Sub is a message distribution system for events and subscriptions.
5. Data design
Use PostgreSQL for transactional state and enforce tenant ownership in the data model. The API should include tenant_id in every tenant-owned lookup, update, and uniqueness constraint. Application-level scoping is necessary, but it is not a reason to omit database constraints and defensive query design.
The outbox table closes the dual-write gap between committing a business change and publishing its event. A relay claims undelivered rows, publishes them, and marks them delivered. A crash after publish but before marking the row can create a duplicate, so consumers still need idempotency.
erDiagram
TENANT ||--o{ WORK_ITEM : owns
TENANT ||--o{ OUTBOX_EVENT : emits
TENANT ||--o{ IDEMPOTENCY_RECORD : scopes
WORK_ITEM ||--o{ OUTBOX_EVENT : produces
TENANT {
uuid id PK
text name
timestamptz created_at
}
WORK_ITEM {
uuid id PK
uuid tenant_id FK
text status
jsonb payload
timestamptz created_at
}
OUTBOX_EVENT {
uuid id PK
uuid tenant_id FK
uuid work_item_id FK
text event_type
jsonb payload
timestamptz created_at
}
IDEMPOTENCY_RECORD {
uuid id PK
uuid tenant_id FK
text key
text response_body
timestamptz created_at
}Caption: Tenant ownership is explicit, and outbox events are persisted alongside the domain records that produce them.
CREATE TABLE tenant (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE work_item (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id uuid NOT NULL REFERENCES tenant(id),
status text NOT NULL CHECK (status IN ('pending', 'running', 'complete', 'failed')),
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, id)
);
CREATE INDEX work_item_tenant_created_idx
ON work_item (tenant_id, created_at DESC);
CREATE TABLE outbox_event (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id uuid NOT NULL REFERENCES tenant(id),
work_item_id uuid NOT NULL,
event_type text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz,
FOREIGN KEY (tenant_id, work_item_id) REFERENCES work_item (tenant_id, id)
);
CREATE INDEX outbox_unpublished_idx
ON outbox_event (created_at)
WHERE published_at IS NULL;
CREATE TABLE idempotency_record (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id uuid NOT NULL REFERENCES tenant(id),
key text NOT NULL,
response_body jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, key)
);
Do not put large exports, uploaded files, or unbounded event payloads in PostgreSQL rows. Store object content in Cloud Storage and keep references and lifecycle metadata in the relational database. Define retention for outbox history and idempotency records; cleanup should be bounded and observable rather than a single untested bulk delete.
6. Low-level design
The API is a modular monolith organized around business capabilities. Modules can share one process and database while keeping ownership of their tables and application services explicit. The worker shares domain contracts but has separate startup configuration, concurrency limits, and IAM permissions.
classDiagram
class ApiHost {
+MapEndpoints()
}
class WorkItemModule {
+CreateWorkItem()
+GetWorkItem()
}
class WorkItemRepository {
+Insert()
+FindByTenant()
}
class OutboxWriter {
+AddEvent()
}
class TaskScheduler {
+Schedule()
}
class WorkerHost {
+HandleTask()
+HandleEvent()
}
class IdempotencyStore {
+TryBegin()
+Complete()
}
ApiHost --> WorkItemModule
WorkItemModule --> WorkItemRepository
WorkItemModule --> OutboxWriter
WorkItemModule --> TaskScheduler
WorkerHost --> IdempotencyStore
WorkerHost --> WorkItemRepositoryCaption: Keep business boundaries in code while API and worker remain a small number of deployable services.
Minimal runnable API
This is a minimal ASP.NET Core endpoint suitable for validating the container and Cloud Run request path. It deliberately has no persistence or authentication; add those before exposing a production route. Run it with dotnet run in a .NET 10 web project.
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/health", () => Results.Ok(new
{
status = "ok",
service = "saas-api"
}));
app.MapPost("/work-items", (CreateWorkItem request) =>
{
if (string.IsNullOrWhiteSpace(request.Name))
return Results.ValidationProblem(new Dictionary<string, string[]>
{
["name"] = ["Name is required."]
});
return Results.Accepted(value: new
{
id = Guid.NewGuid(),
status = "pending"
});
});
app.Run();
record CreateWorkItem(string Name);
Notice: the example binds to the ASP.NET Core defaults; a Cloud Run container must listen on the port provided by the PORT environment variable.
Realistic task scheduling and idempotent handling
The API should commit the work item and outbox event before scheduling side effects. This example shows the Cloud Tasks client and a worker endpoint; production code should use dependency injection, database transactions, tenant authorization, structured logging, and an explicit task payload schema. Add the Google.Cloud.Tasks.V2 package.
using Google.Cloud.Tasks.V2;
using Google.Protobuf;
using System.Text.Json;
var projectId = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_PROJECT")
?? throw new InvalidOperationException("Missing project ID.");
var location = Environment.GetEnvironmentVariable("TASKS_LOCATION") ?? "us-central1";
var queue = Environment.GetEnvironmentVariable("TASKS_QUEUE") ?? "work-items";
var workerUrl = Environment.GetEnvironmentVariable("WORKER_URL")
?? throw new InvalidOperationException("Missing worker URL.");
var invokerServiceAccount = Environment.GetEnvironmentVariable("TASK_INVOKER_EMAIL")
?? throw new InvalidOperationException("Missing task invoker identity.");
var tasksClient = await CloudTasksClient.CreateAsync();
var parent = QueueName.FromProjectLocationQueue(projectId, location, queue);
app.MapPost("/internal/schedule", async (ScheduleRequest request) =>
{
var taskId = $"{request.TenantId}-{request.WorkItemId}";
var payload = JsonSerializer.SerializeToUtf8Bytes(request);
var task = new Google.Cloud.Tasks.V2.Task
{
Name = $"{parent}/tasks/{taskId}",
HttpRequest = new HttpRequest
{
HttpMethod = HttpMethod.Post,
Url = $"{workerUrl}/internal/process",
Headers = { ["Content-Type"] = "application/json" },
Body = ByteString.CopyFrom(payload),
OidcToken = new OidcToken
{
ServiceAccountEmail = invokerServiceAccount,
Audience = workerUrl
}
}
};
try
{
await tasksClient.CreateTaskAsync(new CreateTaskRequest
{
Parent = parent.ToString(),
Task = task
});
}
catch (Google.Api.Gax.Rpc.AlreadyExistsException)
{
// A deterministic task name makes repeated scheduling requests safe.
}
return Results.Accepted();
});
app.MapPost("/internal/process", async (
ProcessRequest request,
IWorkItemProcessor processor,
CancellationToken cancellationToken) =>
{
var completed = await processor.TryProcessOnceAsync(
request.TenantId,
request.WorkItemId,
cancellationToken);
// Return success for completed or already-completed work.
// Return a retryable error only when a retry can make progress.
return completed ? Results.Ok() : Results.StatusCode(503);
});
record ScheduleRequest(Guid TenantId, Guid WorkItemId);
record ProcessRequest(Guid TenantId, Guid WorkItemId);
Notice: a deterministic task name helps prevent duplicate enqueueing, but it does not make task execution exactly once; the processor still needs a durable idempotency check.
For event consumers, persist a processed-event identifier in the same transaction as the consumer’s state change. Return success only after that transaction commits. On transient failures, allow Pub/Sub or Cloud Tasks to retry; on permanent failures, route to a dead-letter path or record a terminal state that operators can inspect.
7. Key flows
The synchronous flow returns only after the relational transaction commits. Cache invalidation happens after the write; if invalidation fails, the system must tolerate stale cache entries through a short TTL or versioned cache keys.
sequenceDiagram
participant Client
participant API as Cloud Run API
participant Cache as Memorystore
participant DB as Cloud SQL
Client->>API: Create work item
API->>DB: Insert item and outbox row
DB-->>API: Commit
API->>Cache: Invalidate tenant key
API-->>Client: Accepted with item IDCaption: A successful response means the database transaction committed, not that downstream work finished.
The asynchronous flow separates event fan-out from targeted task dispatch. The worker must authenticate each request, validate tenant scope, and make retries harmless. Keep queue payloads small and versioned; large or sensitive data should be fetched by identifier under the worker’s service identity.
sequenceDiagram
participant Relay as Outbox relay
participant PubSub as Pub/Sub
participant Tasks as Cloud Tasks
participant Worker as Cloud Run worker
participant DB as Cloud SQL
Relay->>PubSub: Publish domain event
PubSub->>Worker: Deliver event
Worker->>DB: Apply idempotent event
DB-->>Worker: Commit
Worker-->>PubSub: Acknowledge
API->>Tasks: Enqueue scheduled task
Tasks->>Worker: Authenticated HTTP request
Worker->>DB: Process task idempotently
DB-->>Worker: Commit result
Worker-->>Tasks: Return successCaption: Pub/Sub acknowledges successful event processing; Cloud Tasks retries a targeted HTTP request until its configured policy is exhausted.
8. Build and release
Keep infrastructure in Terraform and application code in the same repository or in repositories with a shared, versioned deployment contract. Use separate projects or at least separate service identities and data resources for staging and production. Pin provider and action versions in the actual repository, and review provider upgrades independently from application changes.
Infrastructure as code
This excerpt illustrates the core resources, not a complete production module. Add Cloud SQL, networking, IAM bindings, secrets, monitoring, and environment-specific variables before applying it. State should be stored remotely with access controls and locking.
resource "google_pubsub_topic" "domain_events" {
name = "domain-events"
}
resource "google_cloud_tasks_queue" "work_items" {
name = "work-items"
location = var.region
rate_limits {
max_dispatches_per_second = 20
max_concurrent_dispatches = 40
}
retry_config {
max_attempts = 8
max_retry_duration = "3600s"
min_backoff = "5s"
max_backoff = "300s"
}
}
resource "google_cloud_run_v2_service" "api" {
name = "saas-api"
location = var.region
template {
service_account = google_service_account.api.email
max_instance_request_concurrency = 40
scaling {
min_instance_count = 0
max_instance_count = 20
}
containers {
image = var.api_image
ports {
container_port = 8080
}
env {
name = "ASPNETCORE_URLS"
value = "http://0.0.0.0:8080"
}
}
}
}
resource "google_service_account" "api" {
account_id = "saas-api"
display_name = "SaaS API runtime"
}
resource "google_cloud_run_v2_service_iam_member" "api_public_invoker" {
name = google_cloud_run_v2_service.api.name
location = google_cloud_run_v2_service.api.location
role = "roles/run.invoker"
member = "allUsers"
}
The public invoker binding is appropriate only for an intentionally public API whose application layer performs authentication. For a private service, remove it and grant invocation only to trusted identities. Give the API and worker separate service accounts; grant each only the database, task, topic, and secret permissions it needs.
CI/CD pipeline
A production workflow should build once, scan and test the artifact, deploy to staging, run smoke tests, then promote the same image digest to production with an approval gate. Workload Identity Federation avoids long-lived service-account keys in GitHub.
name: build-and-deploy
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
id-token: write
jobs:
test-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.0.x"
- run: dotnet restore
- run: dotnet test --configuration Release
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
service_account: ${{ secrets.GCP_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- run: gcloud auth configure-docker ${{ vars.GCP_REGION }}-docker.pkg.dev --quiet
- run: |
IMAGE="${{ vars.GCP_REGION }}-docker.pkg.dev/${{ vars.GCP_PROJECT }}/apps/saas-api:${GITHUB_SHA}"
docker build --tag "$IMAGE" .
docker push "$IMAGE"
echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
- run: |
gcloud run deploy saas-api-staging \
--image "$IMAGE" \
--region "${{ vars.GCP_REGION }}" \
--project "${{ vars.GCP_PROJECT }}" \
--no-allow-unauthenticated
staging-smoke:
needs: test-build
runs-on: ubuntu-latest
steps:
- run: echo "Run authenticated staging smoke tests against the deployed image."
production:
needs: staging-smoke
runs-on: ubuntu-latest
environment: production
steps:
- run: echo "Promote the verified image digest to the production Cloud Run service."
The abbreviated final step is an intentional placeholder: production promotion must use the exact built image digest, not rebuild from source. Add an authenticated staging test, image vulnerability scanning, database migration checks, and a rollback or traffic-reversal procedure before treating this as a release pipeline.
Test strategy
| Test level | What to verify |
|---|---|
| Unit | Domain rules, tenant scoping, idempotency decisions, retry classification, serialization compatibility |
| Integration | PostgreSQL transactions and migrations; Redis behavior; task and Pub/Sub adapters against controlled test infrastructure |
| Contract | Event schema compatibility, worker request authentication, API response and error contracts |
| Load | Peak API traffic, database connection use, cache hit behavior, queue drain time, and recovery after dependency latency |
Load tests should include tenant skew, duplicate delivery, cold starts, database saturation, and queue backlog recovery. Test migrations against representative data volume; a migration that is fast on an empty test database can still be disruptive in production.
9. Observability
Instrument the API, relay, and worker with the same trace and correlation conventions. Propagate trace context through Pub/Sub attributes and task headers where the client and handler support it; otherwise create a consumer span linked to the originating message identifiers. Do not treat a single HTTP request trace as proof that asynchronous processing completed.
Logs and metrics
Use structured logs with stable field names. Log identifiers needed to connect the request, tenant, work item, and message, but do not log access tokens, credentials, or sensitive payloads.
{
"severity": "INFO",
"message": "work_item_processing_completed",
"service": "saas-worker",
"trace_id": "projects/example/traces/0123456789abcdef",
"tenant_id": "tenant-42",
"work_item_id": "f2f3c4d5-1234-4567-8901-123456789abc",
"delivery_id": "task-8a31",
"attempt": 2,
"duration_ms": 184,
"outcome": "success"
}
For services, watch RED: request rate, error rate, and duration. For infrastructure, watch USE: utilization, saturation, and errors. Useful signals include Cloud Run instance count and concurrency, database CPU and connection pressure, Redis memory and evictions, and queue age or backlog. Use bounded-cardinality labels: tenant IDs and item IDs belong in logs and traces, not metric labels.
Distributed tracing
Use OpenTelemetry in the .NET application and export traces through the Google Cloud OpenTelemetry integration or an OpenTelemetry Collector configured for the GCP environment. Keep instrumentation consistent across services and avoid capturing request bodies by default.
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(
serviceName: "saas-api",
serviceVersion: Environment.GetEnvironmentVariable("APP_VERSION") ?? "local"))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource("SaaS.WorkItems")
.AddOtlpExporter());
var app = builder.Build();
app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
app.Run();
Configure the OTLP endpoint and credentials through the deployment environment rather than hard-coding them. Add database client instrumentation where supported, and create explicit spans around outbox publishing and task processing. Verify that trace IDs appear in logs and that asynchronous spans preserve message correlation.
SLI/SLOs, dashboards, and alerts
| SLI | Initial SLO | Measurement |
|---|---|---|
| API availability | 99.9% monthly | Successful eligible requests divided by eligible requests |
| API latency | 95% under 300 ms | Server-side request duration for ordinary API routes |
| Accepted-work completion | 99% within 10 minutes | Work items reaching a terminal success state within the target window |
| Database write success | 99.9% of valid writes | Committed writes excluding client and validation errors |
Create dashboards for API RED, Cloud Run scaling, Cloud SQL health and connections, Redis memory and evictions, Pub/Sub oldest unacked message age, Cloud Tasks queue backlog, and outbox publish delay.
Page on sustained API SLO burn, elevated 5xx rate, database unavailability or near-exhausted connections, and asynchronous work exceeding its completion SLO. Alert without paging for a growing but recoverable queue backlog, cache eviction changes, or a single failed task. Tune thresholds from observed baseline behavior and documented SLOs rather than isolated noisy samples.
10. Security and compliance
Use separate runtime identities for the API, relay, and worker. The public API should not inherit permissions to publish arbitrary events or administer queues unless it needs those capabilities. Use narrowly scoped IAM roles, private connectivity for PostgreSQL and Redis, and Secret Manager for credentials that cannot be replaced with identity-based access.
Apply these controls from the first production release:
- Validate user authentication and tenant authorization independently; a valid token does not imply access to every tenant.
- Use parameterized SQL and bind tenant identity from authenticated server-side context, not an untrusted request field.
- Require authenticated invocation for worker endpoints and verify that task and Pub/Sub identities are the intended callers.
- Set request size limits, timeouts, concurrency bounds, and per-tenant application quotas.
- Encrypt data in transit and at rest using managed defaults; assess customer-managed encryption keys only when compliance or contractual requirements justify their operational cost.
- Keep audit records for privileged actions and define retention for application logs, database backups, and queue dead-letter data.
- Run dependency and container image scanning in CI; rotate credentials and review IAM changes.
- Document data classification, deletion workflows, backup retention, and incident response against the actual regulatory obligations of the product.
Do not claim compliance merely because a service has compliance documentation. The product’s data flows, access controls, contracts, retention, and operating procedures determine whether a particular compliance regime is satisfied.
11. Cost
This is an assumption-based monthly estimate, not a quote. It assumes one production region, the workload and data volumes in Section 1, modest staging usage, Cloud SQL high availability, and ordinary rather than unusually high network egress. Prices vary by region, configuration, discounts, and current GCP pricing.
| Cost area | Monthly planning range (USD) | Main driver |
|---|---|---|
| Cloud Run API and worker | $50–250 | CPU and memory allocation, concurrency, minimum instances, request duration |
| Cloud SQL for PostgreSQL HA and storage | $350–700 | Machine configuration, regional HA, storage, backups |
| Memorystore for Redis | $100–300 | Tier, memory size, availability configuration |
| Pub/Sub and Cloud Tasks | $10–100 | Message volume, retention, retries, queue usage |
| Networking and load balancing | $50–250 | Egress, connector or network configuration, external traffic |
| Logging, monitoring, and trace storage | $50–200 | Ingestion, retention, sampling, and metric volume |
| Artifact Registry, Secret Manager, and misc. | $20–100 | Storage, secret operations, and supporting resources |
| Estimated total | $630–1,900 | Validate with the GCP pricing calculator and measured usage |
The largest predictable cost is often the highly available database, not the serverless request layer. Avoid paying for excess baseline capacity without evidence, but do not remove database HA merely to make a small estimate look attractive. Control logging volume, choose trace sampling deliberately, set queue limits, and review egress before enabling large exports or cross-region traffic.
12. Failure modes, resilience, and disaster recovery
| Failure mode | Expected behavior | Recovery or mitigation |
|---|---|---|
| Cloud Run instance restart | In-flight request may fail; clients may retry | Keep handlers stateless and make writes idempotent |
| Cloud SQL unavailable | Writes fail; API returns a bounded error | Use connection limits, retries with jitter, HA configuration, backups, and tested restore procedures |
| Redis unavailable or stale | Cache reads fail or return old values | Fall back to PostgreSQL for safe reads; use TTLs and invalidate after writes |
| Pub/Sub duplicate or delayed delivery | Consumer sees repeated or late events | Durable consumer idempotency, dead-letter handling, backlog alerts |
| Cloud Tasks retry exhaustion | Target work remains incomplete | Inspect terminal failures and provide an operator-controlled replay path |
| Outbox relay interruption | Events remain unpublished in PostgreSQL | Alert on oldest unpublished row and restart or replay the relay |
| Bad deployment or migration | Increased errors or blocked writes | Progressive deployment, compatible schema changes, and rollback plan |
Use Cloud SQL high availability for zonal failure tolerance within the selected region. Enable backups and point-in-time recovery where supported by the chosen configuration, and prove restoration by running scheduled restore exercises into an isolated environment. Keep schema changes backward-compatible across at least one release: deploy additive changes first, migrate data in bounded batches, then remove obsolete fields later.
Regional disaster recovery is a separate product decision. A regional standby, replicated database, and tested traffic switch add cost and operational complexity. For this scale, first define acceptable RPO and RTO, document a regional rebuild or restore procedure, and measure it. Add warm standby or cross-region replication only when contractual targets or demonstrated business impact justify the additional system.
13. Evolving to the next scale
Scale by following measured pressure, not daily-user count alone. A tenant with a large export workload can dominate a system whose total user count appears modest.
| Signal | First response | Consider a larger change when |
|---|---|---|
| API CPU or request latency rises | Profile code, tune concurrency, bound expensive routes | API and background work interfere despite independent Cloud Run services |
| Database connection pressure grows | Use bounded connection pools, reduce chatty queries, cache suitable reads | Sustained write throughput or workload isolation exceeds the current database design |
| Queue age grows | Tune worker concurrency and dispatch rate; inspect slow handlers | Different workload classes need independent scaling, retention, or operational ownership |
| Tenant skew causes noisy-neighbor effects | Add per-tenant quotas and fair scheduling | A small number of large tenants need isolated data or compute |
| Deployment coordination becomes costly | Preserve module boundaries and ownership | A bounded context has an independent team, scaling profile, and release lifecycle |
Possible next steps include splitting a high-throughput worker from the shared worker service, introducing separate queues for distinct priority classes, adding read replicas for read-heavy workloads, or moving selected large-tenant data into isolated databases. Extract a module into a service only when the boundary is stable and the operational benefit exceeds the cost of distributed transactions, deployment, and on-call ownership.
14. Phased implementation plan
- Establish the baseline. Confirm tenant model, authentication source, availability target, data classification, RPO/RTO, and regional choice. Create separate staging and production environments and a minimal Terraform foundation.
- Ship the synchronous product path. Build the Cloud Run API, Cloud SQL schema, tenant-scoped repository layer, migrations, health checks, and basic structured logging. Add request and database limits before public launch.
- Add caching deliberately. Instrument read paths first, then introduce Memorystore for data with a clear freshness policy. Test cache outage behavior and invalidation; do not cache authorization decisions without an explicit model.
- Introduce reliable asynchronous work. Add the outbox and relay, publish versioned events to Pub/Sub, then add Cloud Tasks only for work that needs scheduling or dispatch-rate control. Implement idempotency and dead-letter operations before increasing traffic.
- Operationalize delivery. Add CI tests, image scanning, staging smoke tests, approval-gated production promotion, dashboards, SLO alerts, and a rollback runbook. Exercise database migration and image rollback procedures.
- Prove recovery and tune capacity. Run peak-load and dependency-failure tests, restore a database backup, replay a controlled queue backlog, and set production scaling bounds from observed results. Review cost and alert noise monthly.
The result is intentionally modest: a small number of services with distinct responsibilities, a transactional source of truth, and asynchronous paths whose retry semantics are explicit. That is enough architecture for this scale—provided the team tests the failure paths as carefully as the happy path.
This article was generated with AI. #AIGenerated