Skip to content

GCP SaaS Backend Blueprint: Cloud Run, Pub/Sub, Memorystore, and Cloud Tasks

A medium-scale GCP SaaS architecture that combines Cloud Run, Cloud SQL, Pub/Sub, Memorystore, and Cloud Tasks across the software delivery lifecycle.

Ekanatha Reddy Urivakili18 min3,405 words

1. Context and scale

This blueprint describes a multi-tenant SaaS backend on Google Cloud Platform. It serves a browser or mobile client through an HTTP API, stores transactional state in PostgreSQL, caches selected reads in Redis, and handles both event-driven work and scheduled or rate-limited tasks.

The architecture is deliberately sized for one or two product teams. It uses a modular monolith for the public API and a separate worker service, rather than turning each domain module into a deployable service. Pub/Sub distributes business events; Cloud Tasks schedules explicit HTTP work with per-queue dispatch controls. Those are different jobs, and using both is justified only when the product needs both semantics.

Assumptions

These are planning assumptions, not product limits or capacity guarantees. Validate them with workload tests and current regional pricing before committing to a production budget.

DimensionAssumption
Users50,000 daily active users; 10,000–100,000 is the intended range
Request rate5 requests/s average; 50 requests/s peak, with short bursts above that
Data volume300 GB of primary PostgreSQL data after one year, plus backups
WorkloadMostly interactive reads and writes; asynchronous notifications, exports, and integrations
Team size1–2 teams, approximately 5–10 engineers total, with a rotating operations owner
Availability target99.9% monthly API availability for the initial production service
BudgetPlanning envelope of US$1,000–2,000/month before taxes, support, and unusually high egress or logging

The design assumes one primary GCP region, private connectivity from Cloud Run to data services, and a staging environment that is smaller than production. It does not assume that every component is multi-region or that a regional failure is automatically transparent to users.

2. Requirements

The first release should optimize for clear ownership, bounded operational work, and correct asynchronous processing. Do not add a service merely because it is available: each additional queue, database, and deployment unit adds failure modes and lifecycle work.

Functional requirements

RequirementDesign response
Authenticate users and resolve tenant contextAPI validates identity tokens and derives tenant identity from trusted claims or server-side membership lookup
Serve tenant-scoped CRUD operationsASP.NET Core API with PostgreSQL transactions and tenant filters enforced in application queries
Cache frequently read, non-authoritative dataMemorystore for Redis; PostgreSQL remains the source of truth
Publish business events after commitsTransactional outbox written with domain changes, then relayed to Pub/Sub
Run scheduled, retryable, or rate-limited workCloud Tasks delivers authenticated HTTP requests to a private worker endpoint
Fan out domain events to subscribersPub/Sub topics and subscriptions, with independently deployable consumers when needed
Provide operational visibilityCloud Logging, Cloud Monitoring, Cloud Trace, and OpenTelemetry instrumentation

Non-functional requirements

RequirementTarget or constraint
Availability99.9% monthly API SLO; dependencies and deployment configuration must support regional recovery
LatencyAPI p95 under 300 ms for ordinary database-backed requests, excluding client network time
DurabilityA successful API write is committed to Cloud SQL before returning success
Tenant isolationEvery tenant-owned row and cache key is tenant-scoped; authorization is checked server-side
Retry safetyQueue delivery is treated as at-least-once; handlers must be idempotent
ScalingAPI and worker instances scale independently within configured concurrency and maximum-instance bounds
OperationsOne team can deploy, diagnose, and roll back without coordinating a fleet of independently owned services

3. High-level design

The public API is a Cloud Run service. It handles synchronous requests and writes domain state to Cloud SQL for PostgreSQL. A transactional outbox row is committed in the same database transaction as the domain change; a relay publishes it to Pub/Sub and records delivery.

Memorystore reduces repeated reads for suitable data, but it is not a correctness dependency. Pub/Sub carries events that can have multiple subscribers. Cloud Tasks carries commands to a specific HTTP handler, especially when dispatch rate, scheduling, or per-task retry behavior matters. Both call a separate Cloud Run worker service using authenticated service identities.

flowchart LR
    Client["Web or mobile client"] --> Api["Cloud Run API"]
    Api --> Db["Cloud SQL for PostgreSQL"]
    Api --> Cache["Memorystore for Redis"]
    Api --> Outbox["Transactional outbox"]
    Outbox --> Relay["Cloud Run outbox relay"]
    Relay --> PubSub["Pub/Sub topic"]
    PubSub --> Worker["Cloud Run worker"]
    Api --> Tasks["Cloud Tasks queue"]
    Tasks --> Worker
    Worker --> Db
    Worker --> Cache
    Api --> Telemetry["Cloud Logging and Monitoring"]
    Worker --> Telemetry

Caption: The API owns synchronous transactions; Pub/Sub distributes events, while Cloud Tasks dispatches targeted work.

Keep the relay in the worker deployment initially if its lifecycle and scaling needs are similar. Split it into a separate Cloud Run service only when its throughput, permissions, or release cadence requires independent control. Avoid a continuously polling process if a scheduled trigger or bounded relay invocation can meet the outbox latency target.

4. Cloud service mapping

ComponentGCP choiceAWS equivalentAzure equivalentReason for choice
HTTP API and workerCloud RunAWS App Runner or ECS on FargateAzure Container AppsRuns containers with independent scaling and managed HTTPS integration
Relational databaseCloud SQL for PostgreSQLAmazon RDS for PostgreSQLAzure Database for PostgreSQLManaged PostgreSQL with backups and regional high-availability options
CacheMemorystore for RedisElastiCache for Redis OSSAzure Managed RedisLow-latency cache accessed over private networking
Event distributionPub/SubAmazon SNS and SQS, depending on fan-out patternAzure Service Bus topicsDurable event delivery to independent subscribers
Scheduled and rate-controlled HTTP workCloud TasksAmazon SQS with a worker, or EventBridge Scheduler plus a targetAzure Queue Storage or Service Bus with a workerExplicit HTTP target, scheduling, retry, and dispatch-rate controls
Container image storageArtifact RegistryAmazon ECRAzure Container RegistryPrivate image storage integrated with deployment identities
SecretsSecret ManagerAWS Secrets ManagerAzure Key VaultCentralized secret storage and access auditing
TelemetryCloud Logging, Monitoring, TraceCloudWatch and X-RayAzure Monitor and Application InsightsNative GCP collection and alerting

Cloud Run is the right default here because the workload is request-driven and the team wants to operate containers without maintaining a cluster. Cloud Tasks and Pub/Sub are not interchangeable: Cloud Tasks is a dispatch mechanism for explicit work; Pub/Sub is a message distribution system for events and subscriptions.

5. Data design

Use PostgreSQL for transactional state and enforce tenant ownership in the data model. The API should include tenant_id in every tenant-owned lookup, update, and uniqueness constraint. Application-level scoping is necessary, but it is not a reason to omit database constraints and defensive query design.

The outbox table closes the dual-write gap between committing a business change and publishing its event. A relay claims undelivered rows, publishes them, and marks them delivered. A crash after publish but before marking the row can create a duplicate, so consumers still need idempotency.

erDiagram
    TENANT ||--o{ WORK_ITEM : owns
    TENANT ||--o{ OUTBOX_EVENT : emits
    TENANT ||--o{ IDEMPOTENCY_RECORD : scopes
    WORK_ITEM ||--o{ OUTBOX_EVENT : produces
    TENANT {
        uuid id PK
        text name
        timestamptz created_at
    }
    WORK_ITEM {
        uuid id PK
        uuid tenant_id FK
        text status
        jsonb payload
        timestamptz created_at
    }
    OUTBOX_EVENT {
        uuid id PK
        uuid tenant_id FK
        uuid work_item_id FK
        text event_type
        jsonb payload
        timestamptz created_at
    }
    IDEMPOTENCY_RECORD {
        uuid id PK
        uuid tenant_id FK
        text key
        text response_body
        timestamptz created_at
    }

Caption: Tenant ownership is explicit, and outbox events are persisted alongside the domain records that produce them.

CREATE TABLE tenant (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name        text NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE work_item (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id   uuid NOT NULL REFERENCES tenant(id),
    status      text NOT NULL CHECK (status IN ('pending', 'running', 'complete', 'failed')),
    payload     jsonb NOT NULL,
    created_at  timestamptz NOT NULL DEFAULT now(),
    updated_at  timestamptz NOT NULL DEFAULT now(),
    UNIQUE (tenant_id, id)
);

CREATE INDEX work_item_tenant_created_idx
    ON work_item (tenant_id, created_at DESC);

CREATE TABLE outbox_event (
    id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id     uuid NOT NULL REFERENCES tenant(id),
    work_item_id  uuid NOT NULL,
    event_type    text NOT NULL,
    payload       jsonb NOT NULL,
    created_at    timestamptz NOT NULL DEFAULT now(),
    published_at  timestamptz,
    FOREIGN KEY (tenant_id, work_item_id) REFERENCES work_item (tenant_id, id)
);

CREATE INDEX outbox_unpublished_idx
    ON outbox_event (created_at)
    WHERE published_at IS NULL;

CREATE TABLE idempotency_record (
    id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id      uuid NOT NULL REFERENCES tenant(id),
    key            text NOT NULL,
    response_body  jsonb NOT NULL,
    created_at     timestamptz NOT NULL DEFAULT now(),
    UNIQUE (tenant_id, key)
);

Do not put large exports, uploaded files, or unbounded event payloads in PostgreSQL rows. Store object content in Cloud Storage and keep references and lifecycle metadata in the relational database. Define retention for outbox history and idempotency records; cleanup should be bounded and observable rather than a single untested bulk delete.

6. Low-level design

The API is a modular monolith organized around business capabilities. Modules can share one process and database while keeping ownership of their tables and application services explicit. The worker shares domain contracts but has separate startup configuration, concurrency limits, and IAM permissions.

classDiagram
    class ApiHost {
        +MapEndpoints()
    }
    class WorkItemModule {
        +CreateWorkItem()
        +GetWorkItem()
    }
    class WorkItemRepository {
        +Insert()
        +FindByTenant()
    }
    class OutboxWriter {
        +AddEvent()
    }
    class TaskScheduler {
        +Schedule()
    }
    class WorkerHost {
        +HandleTask()
        +HandleEvent()
    }
    class IdempotencyStore {
        +TryBegin()
        +Complete()
    }
    ApiHost --> WorkItemModule
    WorkItemModule --> WorkItemRepository
    WorkItemModule --> OutboxWriter
    WorkItemModule --> TaskScheduler
    WorkerHost --> IdempotencyStore
    WorkerHost --> WorkItemRepository

Caption: Keep business boundaries in code while API and worker remain a small number of deployable services.

Minimal runnable API

This is a minimal ASP.NET Core endpoint suitable for validating the container and Cloud Run request path. It deliberately has no persistence or authentication; add those before exposing a production route. Run it with dotnet run in a .NET 10 web project.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/health", () => Results.Ok(new
{
    status = "ok",
    service = "saas-api"
}));

app.MapPost("/work-items", (CreateWorkItem request) =>
{
    if (string.IsNullOrWhiteSpace(request.Name))
        return Results.ValidationProblem(new Dictionary<string, string[]>
        {
            ["name"] = ["Name is required."]
        });

    return Results.Accepted(value: new
    {
        id = Guid.NewGuid(),
        status = "pending"
    });
});

app.Run();

record CreateWorkItem(string Name);

Notice: the example binds to the ASP.NET Core defaults; a Cloud Run container must listen on the port provided by the PORT environment variable.

Realistic task scheduling and idempotent handling

The API should commit the work item and outbox event before scheduling side effects. This example shows the Cloud Tasks client and a worker endpoint; production code should use dependency injection, database transactions, tenant authorization, structured logging, and an explicit task payload schema. Add the Google.Cloud.Tasks.V2 package.

using Google.Cloud.Tasks.V2;
using Google.Protobuf;
using System.Text.Json;

var projectId = Environment.GetEnvironmentVariable("GOOGLE_CLOUD_PROJECT")
    ?? throw new InvalidOperationException("Missing project ID.");
var location = Environment.GetEnvironmentVariable("TASKS_LOCATION") ?? "us-central1";
var queue = Environment.GetEnvironmentVariable("TASKS_QUEUE") ?? "work-items";
var workerUrl = Environment.GetEnvironmentVariable("WORKER_URL")
    ?? throw new InvalidOperationException("Missing worker URL.");
var invokerServiceAccount = Environment.GetEnvironmentVariable("TASK_INVOKER_EMAIL")
    ?? throw new InvalidOperationException("Missing task invoker identity.");

var tasksClient = await CloudTasksClient.CreateAsync();
var parent = QueueName.FromProjectLocationQueue(projectId, location, queue);

app.MapPost("/internal/schedule", async (ScheduleRequest request) =>
{
    var taskId = $"{request.TenantId}-{request.WorkItemId}";
    var payload = JsonSerializer.SerializeToUtf8Bytes(request);

    var task = new Google.Cloud.Tasks.V2.Task
    {
        Name = $"{parent}/tasks/{taskId}",
        HttpRequest = new HttpRequest
        {
            HttpMethod = HttpMethod.Post,
            Url = $"{workerUrl}/internal/process",
            Headers = { ["Content-Type"] = "application/json" },
            Body = ByteString.CopyFrom(payload),
            OidcToken = new OidcToken
            {
                ServiceAccountEmail = invokerServiceAccount,
                Audience = workerUrl
            }
        }
    };

    try
    {
        await tasksClient.CreateTaskAsync(new CreateTaskRequest
        {
            Parent = parent.ToString(),
            Task = task
        });
    }
    catch (Google.Api.Gax.Rpc.AlreadyExistsException)
    {
        // A deterministic task name makes repeated scheduling requests safe.
    }

    return Results.Accepted();
});

app.MapPost("/internal/process", async (
    ProcessRequest request,
    IWorkItemProcessor processor,
    CancellationToken cancellationToken) =>
{
    var completed = await processor.TryProcessOnceAsync(
        request.TenantId,
        request.WorkItemId,
        cancellationToken);

    // Return success for completed or already-completed work.
    // Return a retryable error only when a retry can make progress.
    return completed ? Results.Ok() : Results.StatusCode(503);
});

record ScheduleRequest(Guid TenantId, Guid WorkItemId);
record ProcessRequest(Guid TenantId, Guid WorkItemId);

Notice: a deterministic task name helps prevent duplicate enqueueing, but it does not make task execution exactly once; the processor still needs a durable idempotency check.

For event consumers, persist a processed-event identifier in the same transaction as the consumer’s state change. Return success only after that transaction commits. On transient failures, allow Pub/Sub or Cloud Tasks to retry; on permanent failures, route to a dead-letter path or record a terminal state that operators can inspect.

7. Key flows

The synchronous flow returns only after the relational transaction commits. Cache invalidation happens after the write; if invalidation fails, the system must tolerate stale cache entries through a short TTL or versioned cache keys.

sequenceDiagram
    participant Client
    participant API as Cloud Run API
    participant Cache as Memorystore
    participant DB as Cloud SQL
    Client->>API: Create work item
    API->>DB: Insert item and outbox row
    DB-->>API: Commit
    API->>Cache: Invalidate tenant key
    API-->>Client: Accepted with item ID

Caption: A successful response means the database transaction committed, not that downstream work finished.

The asynchronous flow separates event fan-out from targeted task dispatch. The worker must authenticate each request, validate tenant scope, and make retries harmless. Keep queue payloads small and versioned; large or sensitive data should be fetched by identifier under the worker’s service identity.

sequenceDiagram
    participant Relay as Outbox relay
    participant PubSub as Pub/Sub
    participant Tasks as Cloud Tasks
    participant Worker as Cloud Run worker
    participant DB as Cloud SQL
    Relay->>PubSub: Publish domain event
    PubSub->>Worker: Deliver event
    Worker->>DB: Apply idempotent event
    DB-->>Worker: Commit
    Worker-->>PubSub: Acknowledge
    API->>Tasks: Enqueue scheduled task
    Tasks->>Worker: Authenticated HTTP request
    Worker->>DB: Process task idempotently
    DB-->>Worker: Commit result
    Worker-->>Tasks: Return success

Caption: Pub/Sub acknowledges successful event processing; Cloud Tasks retries a targeted HTTP request until its configured policy is exhausted.

8. Build and release

Keep infrastructure in Terraform and application code in the same repository or in repositories with a shared, versioned deployment contract. Use separate projects or at least separate service identities and data resources for staging and production. Pin provider and action versions in the actual repository, and review provider upgrades independently from application changes.

Infrastructure as code

This excerpt illustrates the core resources, not a complete production module. Add Cloud SQL, networking, IAM bindings, secrets, monitoring, and environment-specific variables before applying it. State should be stored remotely with access controls and locking.

resource "google_pubsub_topic" "domain_events" {
  name = "domain-events"
}

resource "google_cloud_tasks_queue" "work_items" {
  name     = "work-items"
  location = var.region

  rate_limits {
    max_dispatches_per_second = 20
    max_concurrent_dispatches = 40
  }

  retry_config {
    max_attempts       = 8
    max_retry_duration = "3600s"
    min_backoff        = "5s"
    max_backoff        = "300s"
  }
}

resource "google_cloud_run_v2_service" "api" {
  name     = "saas-api"
  location = var.region

  template {
    service_account = google_service_account.api.email
    max_instance_request_concurrency = 40

    scaling {
      min_instance_count = 0
      max_instance_count = 20
    }

    containers {
      image = var.api_image

      ports {
        container_port = 8080
      }

      env {
        name  = "ASPNETCORE_URLS"
        value = "http://0.0.0.0:8080"
      }
    }
  }
}

resource "google_service_account" "api" {
  account_id   = "saas-api"
  display_name = "SaaS API runtime"
}

resource "google_cloud_run_v2_service_iam_member" "api_public_invoker" {
  name     = google_cloud_run_v2_service.api.name
  location = google_cloud_run_v2_service.api.location
  role     = "roles/run.invoker"
  member   = "allUsers"
}

The public invoker binding is appropriate only for an intentionally public API whose application layer performs authentication. For a private service, remove it and grant invocation only to trusted identities. Give the API and worker separate service accounts; grant each only the database, task, topic, and secret permissions it needs.

CI/CD pipeline

A production workflow should build once, scan and test the artifact, deploy to staging, run smoke tests, then promote the same image digest to production with an approval gate. Workload Identity Federation avoids long-lived service-account keys in GitHub.

name: build-and-deploy

on:
  push:
    branches: [main]
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

jobs:
  test-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"
      - run: dotnet restore
      - run: dotnet test --configuration Release
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
          service_account: ${{ secrets.GCP_DEPLOY_SERVICE_ACCOUNT }}
      - uses: google-github-actions/setup-gcloud@v2
      - run: gcloud auth configure-docker ${{ vars.GCP_REGION }}-docker.pkg.dev --quiet
      - run: |
          IMAGE="${{ vars.GCP_REGION }}-docker.pkg.dev/${{ vars.GCP_PROJECT }}/apps/saas-api:${GITHUB_SHA}"
          docker build --tag "$IMAGE" .
          docker push "$IMAGE"
          echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
      - run: |
          gcloud run deploy saas-api-staging \
            --image "$IMAGE" \
            --region "${{ vars.GCP_REGION }}" \
            --project "${{ vars.GCP_PROJECT }}" \
            --no-allow-unauthenticated

  staging-smoke:
    needs: test-build
    runs-on: ubuntu-latest
    steps:
      - run: echo "Run authenticated staging smoke tests against the deployed image."

  production:
    needs: staging-smoke
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: echo "Promote the verified image digest to the production Cloud Run service."

The abbreviated final step is an intentional placeholder: production promotion must use the exact built image digest, not rebuild from source. Add an authenticated staging test, image vulnerability scanning, database migration checks, and a rollback or traffic-reversal procedure before treating this as a release pipeline.

Test strategy

Test levelWhat to verify
UnitDomain rules, tenant scoping, idempotency decisions, retry classification, serialization compatibility
IntegrationPostgreSQL transactions and migrations; Redis behavior; task and Pub/Sub adapters against controlled test infrastructure
ContractEvent schema compatibility, worker request authentication, API response and error contracts
LoadPeak API traffic, database connection use, cache hit behavior, queue drain time, and recovery after dependency latency

Load tests should include tenant skew, duplicate delivery, cold starts, database saturation, and queue backlog recovery. Test migrations against representative data volume; a migration that is fast on an empty test database can still be disruptive in production.

9. Observability

Instrument the API, relay, and worker with the same trace and correlation conventions. Propagate trace context through Pub/Sub attributes and task headers where the client and handler support it; otherwise create a consumer span linked to the originating message identifiers. Do not treat a single HTTP request trace as proof that asynchronous processing completed.

Logs and metrics

Use structured logs with stable field names. Log identifiers needed to connect the request, tenant, work item, and message, but do not log access tokens, credentials, or sensitive payloads.

{
  "severity": "INFO",
  "message": "work_item_processing_completed",
  "service": "saas-worker",
  "trace_id": "projects/example/traces/0123456789abcdef",
  "tenant_id": "tenant-42",
  "work_item_id": "f2f3c4d5-1234-4567-8901-123456789abc",
  "delivery_id": "task-8a31",
  "attempt": 2,
  "duration_ms": 184,
  "outcome": "success"
}

For services, watch RED: request rate, error rate, and duration. For infrastructure, watch USE: utilization, saturation, and errors. Useful signals include Cloud Run instance count and concurrency, database CPU and connection pressure, Redis memory and evictions, and queue age or backlog. Use bounded-cardinality labels: tenant IDs and item IDs belong in logs and traces, not metric labels.

Distributed tracing

Use OpenTelemetry in the .NET application and export traces through the Google Cloud OpenTelemetry integration or an OpenTelemetry Collector configured for the GCP environment. Keep instrumentation consistent across services and avoid capturing request bodies by default.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "saas-api",
            serviceVersion: Environment.GetEnvironmentVariable("APP_VERSION") ?? "local"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("SaaS.WorkItems")
        .AddOtlpExporter());

var app = builder.Build();
app.MapGet("/health", () => Results.Ok(new { status = "ok" }));
app.Run();

Configure the OTLP endpoint and credentials through the deployment environment rather than hard-coding them. Add database client instrumentation where supported, and create explicit spans around outbox publishing and task processing. Verify that trace IDs appear in logs and that asynchronous spans preserve message correlation.

SLI/SLOs, dashboards, and alerts

SLIInitial SLOMeasurement
API availability99.9% monthlySuccessful eligible requests divided by eligible requests
API latency95% under 300 msServer-side request duration for ordinary API routes
Accepted-work completion99% within 10 minutesWork items reaching a terminal success state within the target window
Database write success99.9% of valid writesCommitted writes excluding client and validation errors

Create dashboards for API RED, Cloud Run scaling, Cloud SQL health and connections, Redis memory and evictions, Pub/Sub oldest unacked message age, Cloud Tasks queue backlog, and outbox publish delay.

Page on sustained API SLO burn, elevated 5xx rate, database unavailability or near-exhausted connections, and asynchronous work exceeding its completion SLO. Alert without paging for a growing but recoverable queue backlog, cache eviction changes, or a single failed task. Tune thresholds from observed baseline behavior and documented SLOs rather than isolated noisy samples.

10. Security and compliance

Use separate runtime identities for the API, relay, and worker. The public API should not inherit permissions to publish arbitrary events or administer queues unless it needs those capabilities. Use narrowly scoped IAM roles, private connectivity for PostgreSQL and Redis, and Secret Manager for credentials that cannot be replaced with identity-based access.

Apply these controls from the first production release:

  • Validate user authentication and tenant authorization independently; a valid token does not imply access to every tenant.
  • Use parameterized SQL and bind tenant identity from authenticated server-side context, not an untrusted request field.
  • Require authenticated invocation for worker endpoints and verify that task and Pub/Sub identities are the intended callers.
  • Set request size limits, timeouts, concurrency bounds, and per-tenant application quotas.
  • Encrypt data in transit and at rest using managed defaults; assess customer-managed encryption keys only when compliance or contractual requirements justify their operational cost.
  • Keep audit records for privileged actions and define retention for application logs, database backups, and queue dead-letter data.
  • Run dependency and container image scanning in CI; rotate credentials and review IAM changes.
  • Document data classification, deletion workflows, backup retention, and incident response against the actual regulatory obligations of the product.

Do not claim compliance merely because a service has compliance documentation. The product’s data flows, access controls, contracts, retention, and operating procedures determine whether a particular compliance regime is satisfied.

11. Cost

This is an assumption-based monthly estimate, not a quote. It assumes one production region, the workload and data volumes in Section 1, modest staging usage, Cloud SQL high availability, and ordinary rather than unusually high network egress. Prices vary by region, configuration, discounts, and current GCP pricing.

Cost areaMonthly planning range (USD)Main driver
Cloud Run API and worker$50–250CPU and memory allocation, concurrency, minimum instances, request duration
Cloud SQL for PostgreSQL HA and storage$350–700Machine configuration, regional HA, storage, backups
Memorystore for Redis$100–300Tier, memory size, availability configuration
Pub/Sub and Cloud Tasks$10–100Message volume, retention, retries, queue usage
Networking and load balancing$50–250Egress, connector or network configuration, external traffic
Logging, monitoring, and trace storage$50–200Ingestion, retention, sampling, and metric volume
Artifact Registry, Secret Manager, and misc.$20–100Storage, secret operations, and supporting resources
Estimated total$630–1,900Validate with the GCP pricing calculator and measured usage

The largest predictable cost is often the highly available database, not the serverless request layer. Avoid paying for excess baseline capacity without evidence, but do not remove database HA merely to make a small estimate look attractive. Control logging volume, choose trace sampling deliberately, set queue limits, and review egress before enabling large exports or cross-region traffic.

12. Failure modes, resilience, and disaster recovery

Failure modeExpected behaviorRecovery or mitigation
Cloud Run instance restartIn-flight request may fail; clients may retryKeep handlers stateless and make writes idempotent
Cloud SQL unavailableWrites fail; API returns a bounded errorUse connection limits, retries with jitter, HA configuration, backups, and tested restore procedures
Redis unavailable or staleCache reads fail or return old valuesFall back to PostgreSQL for safe reads; use TTLs and invalidate after writes
Pub/Sub duplicate or delayed deliveryConsumer sees repeated or late eventsDurable consumer idempotency, dead-letter handling, backlog alerts
Cloud Tasks retry exhaustionTarget work remains incompleteInspect terminal failures and provide an operator-controlled replay path
Outbox relay interruptionEvents remain unpublished in PostgreSQLAlert on oldest unpublished row and restart or replay the relay
Bad deployment or migrationIncreased errors or blocked writesProgressive deployment, compatible schema changes, and rollback plan

Use Cloud SQL high availability for zonal failure tolerance within the selected region. Enable backups and point-in-time recovery where supported by the chosen configuration, and prove restoration by running scheduled restore exercises into an isolated environment. Keep schema changes backward-compatible across at least one release: deploy additive changes first, migrate data in bounded batches, then remove obsolete fields later.

Regional disaster recovery is a separate product decision. A regional standby, replicated database, and tested traffic switch add cost and operational complexity. For this scale, first define acceptable RPO and RTO, document a regional rebuild or restore procedure, and measure it. Add warm standby or cross-region replication only when contractual targets or demonstrated business impact justify the additional system.

13. Evolving to the next scale

Scale by following measured pressure, not daily-user count alone. A tenant with a large export workload can dominate a system whose total user count appears modest.

SignalFirst responseConsider a larger change when
API CPU or request latency risesProfile code, tune concurrency, bound expensive routesAPI and background work interfere despite independent Cloud Run services
Database connection pressure growsUse bounded connection pools, reduce chatty queries, cache suitable readsSustained write throughput or workload isolation exceeds the current database design
Queue age growsTune worker concurrency and dispatch rate; inspect slow handlersDifferent workload classes need independent scaling, retention, or operational ownership
Tenant skew causes noisy-neighbor effectsAdd per-tenant quotas and fair schedulingA small number of large tenants need isolated data or compute
Deployment coordination becomes costlyPreserve module boundaries and ownershipA bounded context has an independent team, scaling profile, and release lifecycle

Possible next steps include splitting a high-throughput worker from the shared worker service, introducing separate queues for distinct priority classes, adding read replicas for read-heavy workloads, or moving selected large-tenant data into isolated databases. Extract a module into a service only when the boundary is stable and the operational benefit exceeds the cost of distributed transactions, deployment, and on-call ownership.

14. Phased implementation plan

  1. Establish the baseline. Confirm tenant model, authentication source, availability target, data classification, RPO/RTO, and regional choice. Create separate staging and production environments and a minimal Terraform foundation.
  2. Ship the synchronous product path. Build the Cloud Run API, Cloud SQL schema, tenant-scoped repository layer, migrations, health checks, and basic structured logging. Add request and database limits before public launch.
  3. Add caching deliberately. Instrument read paths first, then introduce Memorystore for data with a clear freshness policy. Test cache outage behavior and invalidation; do not cache authorization decisions without an explicit model.
  4. Introduce reliable asynchronous work. Add the outbox and relay, publish versioned events to Pub/Sub, then add Cloud Tasks only for work that needs scheduling or dispatch-rate control. Implement idempotency and dead-letter operations before increasing traffic.
  5. Operationalize delivery. Add CI tests, image scanning, staging smoke tests, approval-gated production promotion, dashboards, SLO alerts, and a rollback runbook. Exercise database migration and image rollback procedures.
  6. Prove recovery and tune capacity. Run peak-load and dependency-failure tests, restore a database backup, replay a controlled queue backlog, and set production scaling bounds from observed results. Review cost and alert noise monthly.

The result is intentionally modest: a small number of services with distinct responsibilities, a transactional source of truth, and asynchronous paths whose retry semantics are explicit. That is enough architecture for this scale—provided the team tests the failure paths as carefully as the happy path.


This article was generated with AI. #AIGenerated