Context and scale
This blueprint is for a multi-tenant SaaS that accepts customer commands, persists tenant-owned records, and performs some work asynchronously: provisioning, notifications, exports, or integrations. It uses a small number of .NET services rather than decomposing every capability into a separately deployed service.
The core is an ASP.NET Core API and a background worker deployed to Azure Container Apps. Azure Service Bus separates request acceptance from asynchronous processing. Azure Cosmos DB for NoSQL stores tenant-partitioned operational data. Bicep owns the infrastructure, and a staging environment exercises the same deployment path as production.
These are planning assumptions, not measured demand or Azure capacity recommendations:
| Dimension | Assumption |
|---|---|
| Users | 25,000 daily active users across 2,000 tenant accounts |
| Requests | 50 API requests/s average; 250 requests/s short peak; 3 queued messages/s average |
| Data volume | 200 GB initially, growing by roughly 25 GB/month; tenant records and operational events, not large binary files |
| Team size | 6 engineers across one product team, with a second team possible as the product grows |
| Monthly budget | $3,000 planning target for the production region and its shared platform costs |
The system is intentionally single-region. Cosmos DB and Service Bus are configured for zone redundancy where the chosen Azure region and service tier support it. That protects against some availability-zone failures, not a regional outage. A secondary region is a later, explicit investment.
The design assumes tenant identity is established by a trusted authentication system, such as Microsoft Entra External ID or an enterprise identity provider. Application authorization still determines which tenant and records a user may access. A tenant ID supplied in an untrusted request body is never sufficient authorization.
Requirements
| Functional requirement | Design response |
|---|---|
| Authenticate users and scope actions to a tenant | Validate identity and tenant membership at the API boundary; authorize each operation |
| Accept commands and return promptly | Validate, persist the command and an outbox event, then return a resource or operation identifier |
| Process work asynchronously | Dispatch outbox events to Service Bus; run a separately scaled Container Apps worker |
| Read tenant-owned data | Query Cosmos DB using the tenant partition key and enforce tenant scope in repository methods |
| Support retries and duplicate delivery | Make consumers idempotent using event IDs and tenant-scoped records |
| Operate and deploy safely | Use Bicep, immutable image tags, staging, automated tests, and revision-based rollout |
| Non-functional requirement | Target or constraint |
|---|---|
| Availability | Target 99.9% monthly availability for the API, subject to dependencies and a defined SLI |
| Latency | API read/write p95 below 500 ms under agreed production load; asynchronous completion is a separate SLI |
| Durability | Persist accepted commands before acknowledging them; retain failed messages for inspection and replay |
| Isolation | Tenant-scoped access, partitioning, and tests for cross-tenant access; noisy-neighbor controls are explicit |
| Scaling | Scale API on HTTP demand and worker on queue backlog; cap maximum replicas to protect downstream capacity |
| Security | Managed identities for Azure service access; no account keys in application configuration |
| Recovery | Documented restore and replay procedures; regional recovery is not claimed until implemented and tested |
Treat latency and availability targets as product decisions to validate with real traffic. An SLO without a measurement definition and a response plan is just a number in a document.
High-level design
flowchart LR
User["Browser or client"] --> Edge["Azure Front Door or API ingress"]
Edge --> Api["Azure Container Apps API"]
Api --> Cosmos["Azure Cosmos DB for NoSQL"]
Api --> Outbox["Outbox documents in Cosmos DB"]
Outbox --> Dispatcher["Container Apps outbox dispatcher"]
Dispatcher --> Bus["Azure Service Bus queue"]
Bus --> Worker["Container Apps worker"]
Worker --> Cosmos
Api --> Identity["Microsoft Entra ID"]
Worker --> Monitor["Azure Monitor and Log Analytics"]
Api --> MonitorCaption: Commands enter through Container Apps, commit to Cosmos DB, and reach workers through Service Bus.
The API is the synchronous boundary. It authenticates the caller, checks tenant authorization, validates the command, and writes both the business record and an outbox record in a Cosmos DB transactional batch. Those documents must share a logical partition key for the batch to be atomic. The API returns only after that write succeeds.
An outbox dispatcher reads committed events and sends them to Service Bus. It may send an event more than once: the send and the dispatcher checkpoint cannot be made one atomic transaction across Cosmos DB and Service Bus. Consumers therefore deduplicate by event ID. Service Bus provides durable buffering and retry handling; it does not make application side effects exactly once.
The worker uses a managed identity to receive messages and access Cosmos DB. It commits its idempotency marker and business side effect in one Cosmos transactional batch, then completes the message. If it fails before completion, redelivery is safe. If a message repeatedly fails, it eventually reaches the dead-letter queue for investigation.
Azure Container Apps supplies ingress, revisions, scaling, and container execution. It is not a substitute for application-level timeouts, idempotency, capacity limits, or dependency health checks. Put the public edge behind Azure Front Door when the product needs a global entry point, WAF policies, or edge caching; it is optional in this regional baseline.
Cloud service mapping
| Concern | Azure choice | AWS equivalent | GCP equivalent |
|---|---|---|---|
| Containerized API and worker | Azure Container Apps | AWS App Runner or ECS on Fargate | Cloud Run |
| Durable queue | Azure Service Bus | Amazon SQS | Cloud Tasks or Pub/Sub, depending on semantics |
| Tenant operational store | Azure Cosmos DB for NoSQL | Amazon DynamoDB | Firestore or Bigtable, depending on access patterns |
| Container image registry | Azure Container Registry | Amazon ECR | Artifact Registry |
| Workload identity | Managed identities and Microsoft Entra ID | IAM roles for tasks | Workload Identity |
| Infrastructure as code | Bicep and Azure Resource Manager | AWS CDK or CloudFormation | Terraform or Google Cloud deployment tooling |
| Logs and metrics | Azure Monitor and Log Analytics | CloudWatch | Cloud Monitoring and Cloud Logging |
| Secrets, if required | Azure Key Vault | AWS Secrets Manager | Secret Manager |
These are service-level analogies, not interchangeable APIs. In particular, queue ordering, delivery, deduplication, and transaction capabilities differ. Select a cloud-native service based on required semantics rather than assuming a migration between equivalents is a configuration change.
Data design
The primary Cosmos DB container uses /tenantId as its partition key. A tenant’s records can then be queried and, where needed, transactionally updated together. The model is deliberately denormalized: Cosmos DB does not provide relational joins across partitions, so read models should match the product’s access patterns.
erDiagram
TENANT ||--o{ SUBSCRIPTION : owns
TENANT ||--o{ ORDER : owns
TENANT ||--o{ OUTBOX_EVENT : emits
TENANT ||--o{ INBOX_RECORD : processes
ORDER ||--o{ OUTBOX_EVENT : produces
OUTBOX_EVENT ||--o| INBOX_RECORD : deduplicated_byCaption: These logical records are tenant-scoped documents in a Cosmos DB container, not relational tables.
| Document type | ID convention | Partition key | Purpose |
|---|---|---|---|
| Tenant | tenant:{tenantId} | tenantId | Tenant settings and status |
| Subscription | subscription:{subscriptionId} | tenantId | Plan and billing-related application state |
| Order | order:{orderId} | tenantId | Customer-owned resource and current status |
| Outbox event | outbox:{eventId} | tenantId | Durable event to dispatch after a business write |
| Inbox record | inbox:{eventId} | tenantId | Evidence that a consumer applied an event |
Cosmos DB for NoSQL is schemaless and does not use relational SQL DDL to create tables. The operational equivalent of the container definition belongs in Bicep: create one container with partition key /tenantId and an autoscale throughput policy sized from observed request-unit consumption. Do not copy a guessed RU/s setting between environments and call it capacity planning.
A representative order and event might look like this:
{
"id": "order:ord_01J9A7",
"tenantId": "ten_42",
"type": "order",
"status": "accepted",
"createdAt": "2026-11-04T12:00:00Z",
"version": 1
}
{
"id": "outbox:evt_01J9A8",
"tenantId": "ten_42",
"type": "outbox",
"eventId": "evt_01J9A8",
"eventType": "OrderAccepted",
"aggregateId": "ord_01J9A7",
"occurredAt": "2026-11-04T12:00:00Z",
"schemaVersion": 1
}
The API always includes tenantId in its query partition key and derives it from authorized identity context. A composite business identifier is not a security boundary. Establish retention rules for outbox and inbox records: deleting deduplication records too early reopens the possibility that a delayed message repeats a side effect.
Partitioning by tenant is appropriate at this scale only if tenant sizes are reasonably distributed. A single exceptionally large tenant can become a hot logical partition. Monitor per-partition behavior and plan a migration strategy before adopting a partition key that cannot be changed in place.
Low-level design
Keep the codebase modular even if the API and worker initially share a repository. Useful modules include identity and tenant access, orders, subscriptions, outbox dispatch, and message handling. Each module owns its contracts and persistence operations. Avoid a shared “common” layer that becomes a second, implicit domain model.
classDiagram
class OrdersApi
class TenantContext
class OrderService
class CosmosOrderRepository
class OutboxDispatcher
class ServiceBusPublisher
class OrderWorker
class InboxRepository
OrdersApi --> TenantContext
OrdersApi --> OrderService
OrderService --> CosmosOrderRepository
OrderService --> OutboxDispatcher
OutboxDispatcher --> ServiceBusPublisher
OrderWorker --> InboxRepository
OrderWorker --> CosmosOrderRepositoryCaption: Keep tenant authorization at the API boundary and idempotency inside the message-handling path.
A useful boundary is “command accepted” versus “command completed.” The API can return 202 Accepted with an operation or resource URL when work is genuinely asynchronous. Do not return success for a write that has not yet been durably recorded. Use an explicit status model such as accepted, processing, completed, and failed where users need to track progress.
Minimal example: publish a message with managed identity. Add the Azure.Messaging.ServiceBus and Azure.Identity packages to a .NET project. The environment variable contains the Service Bus fully qualified namespace, not a connection string. Locally, DefaultAzureCredential can use a developer identity; in Azure, it uses the app’s managed identity.
using Azure.Identity;
using Azure.Messaging.ServiceBus;
var namespaceName = Environment.GetEnvironmentVariable("SERVICEBUS_NAMESPACE")
?? throw new InvalidOperationException("SERVICEBUS_NAMESPACE is required.");
await using var client = new ServiceBusClient(
namespaceName,
new DefaultAzureCredential());
var sender = client.CreateSender("work");
var message = new ServiceBusMessage("""
{"eventId":"evt-123","tenantId":"ten-42","type":"OrderAccepted"}
""")
{
MessageId = "evt-123",
Subject = "OrderAccepted"
};
await sender.SendMessageAsync(message);
Notice that MessageId is stable across retries; it helps broker-side duplicate detection when enabled, but consumer idempotency is still required.
Realistic example: apply a message idempotently. This worker uses a single tenant partition and a transactional batch to create an inbox marker and update the order projection atomically. A conflict on the inbox marker means the event was already applied. In production, validate the envelope and schema version, bound message processing time, and emit useful failure context.
using System.Net;
using Azure;
using Azure.Identity;
using Azure.Messaging.ServiceBus;
using Microsoft.Azure.Cosmos;
var cosmosEndpoint = Environment.GetEnvironmentVariable("COSMOS_ENDPOINT")
?? throw new InvalidOperationException("COSMOS_ENDPOINT is required.");
var busNamespace = Environment.GetEnvironmentVariable("SERVICEBUS_NAMESPACE")
?? throw new InvalidOperationException("SERVICEBUS_NAMESPACE is required.");
using var cosmos = new CosmosClient(cosmosEndpoint, new DefaultAzureCredential());
var container = cosmos.GetContainer("saas", "records");
await using var client = new ServiceBusClient(busNamespace, new DefaultAzureCredential());
await using var processor = client.CreateProcessor("work", new ServiceBusProcessorOptions
{
MaxConcurrentCalls = 8,
AutoCompleteMessages = false
});
processor.ProcessMessageAsync += async args =>
{
var evt = args.Message.Body.ToObjectFromJson<OrderAccepted>()
?? throw new InvalidOperationException("Invalid event body.");
var partition = new PartitionKey(evt.TenantId);
var batch = container.CreateTransactionalBatch(partition);
batch.CreateItem(new
{
id = $"inbox:{evt.EventId}",
tenantId = evt.TenantId,
type = "inbox",
eventId = evt.EventId
});
batch.UpsertItem(new
{
id = $"order:{evt.OrderId}",
tenantId = evt.TenantId,
type = "order",
status = "processing",
version = evt.Version
});
using var result = await batch.ExecuteAsync(args.CancellationToken);
if (result.IsSuccessStatusCode)
{
await args.CompleteMessageAsync(args.Message, args.CancellationToken);
}
else if (result.StatusCode == HttpStatusCode.Conflict)
{
await args.CompleteMessageAsync(args.Message, args.CancellationToken);
}
else
{
throw new RequestFailedException((int)result.StatusCode, "Cosmos batch failed.");
}
};
processor.ProcessErrorAsync += args =>
{
Console.Error.WriteLine($"Service Bus error: {args.Exception.Message}");
return Task.CompletedTask;
};
await processor.StartProcessingAsync();
await Task.Delay(Timeout.Infinite);
public sealed record OrderAccepted(
string EventId,
string TenantId,
string OrderId,
int Version);
Notice the side effect and inbox marker share a partition and transaction; if the side effect spans other systems, use an additional idempotent workflow rather than claiming this transaction covers them.
The sample uses simple host lifetime and error output to make the flow visible. A production worker should use the .NET Generic Host, health checks, structured logging, dependency injection, cancellation-aware shutdown, and explicit retry and dead-letter policies.
Key flows
The synchronous flow commits the resource and outbox record together. If the commit fails, the API does not acknowledge the command. The outbox dispatcher can retry publishing without losing the committed intent.
sequenceDiagram
actor Client
participant Api as Container Apps API
participant Cosmos as Cosmos DB
participant Dispatcher as Outbox dispatcher
participant Bus as Service Bus
Client->>Api: Submit command
Api->>Api: Authenticate and authorize tenant
Api->>Cosmos: Transactional batch resource and outbox
Cosmos-->>Api: Commit
Api-->>Client: 202 Accepted with operation URL
Dispatcher->>Cosmos: Read pending outbox events
Dispatcher->>Bus: Send event with stable event ID
Bus-->>Dispatcher: Acknowledge send
Dispatcher->>Cosmos: Mark dispatched or advance checkpointCaption: The outbox closes the database-to-queue dual-write gap by persisting intent before dispatch.
The worker flow assumes at-least-once delivery. It acknowledges only after its durable work succeeds or after it recognizes a duplicate. A permanent validation or business-rule failure should be dead-lettered with a reason, not retried indefinitely.
sequenceDiagram
participant Bus as Service Bus
participant Worker as Container Apps worker
participant Cosmos as Cosmos DB
Bus->>Worker: Deliver message
Worker->>Worker: Validate envelope
Worker->>Cosmos: Transactional batch inbox and projection
alt New event
Cosmos-->>Worker: Commit
Worker->>Bus: Complete message
else Duplicate event
Cosmos-->>Worker: Inbox conflict
Worker->>Bus: Complete message
else Transient failure
Cosmos-->>Worker: Failure
Worker->>Bus: Abandon or retry
endCaption: Acknowledge after durable processing, and route repeatedly failing messages to the dead-letter queue.
Build and release
Bicep should create the environment and application dependencies, not bake environment-specific secrets into container images. The following is a compact starting point for one region. It uses managed identities, an Azure Container Registry pull identity, a Premium Service Bus namespace, and a Cosmos DB container partitioned by tenant. The image name and region are parameters; production deployments should also define networking, diagnostic settings, resource locks, and policy-compliant naming.
param location string = resourceGroup().location
param image string
param acrName string
param environmentName string = 'saas-prod'
param cosmosName string
param serviceBusName string
var acrPullRole = '7f951dda-4ed3-4680-a7ca-43fe172d538d'
var cosmosDataContributorRole = '00000000-0000-0000-0000-000000000002'
var serviceBusSenderRole = '69a216fc-b8fb-44d8-bc22-1f3c2cd27a39'
var serviceBusReceiverRole = '4f6d3b9b-027b-4f4c-9141-8e5ca3e9c5a6'
resource acr 'Microsoft.ContainerRegistry/registries@2023-07-01' existing = {
name: acrName
}
resource pullIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
name: '${environmentName}-pull'
location: location
}
resource acrPull 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(acr.id, pullIdentity.id, acrPullRole)
scope: acr
properties: {
principalId: pullIdentity.properties.principalId
principalType: 'ServicePrincipal'
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', acrPullRole)
}
}
resource logs 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
name: '${environmentName}-logs'
location: location
properties: {
sku: { name: 'PerGB2018' }
retentionInDays: 30
}
}
resource env 'Microsoft.App/managedEnvironments@2024-03-01' = {
name: environmentName
location: location
properties: {
appLogsConfiguration: {
destination: 'log-analytics'
logAnalyticsConfiguration: {
customerId: logs.properties.customerId
sharedKey: logs.listKeys().primarySharedKey
}
}
}
}
resource cosmos 'Microsoft.DocumentDB/databaseAccounts@2024-05-15' = {
name: cosmosName
location: location
kind: 'GlobalDocumentDB'
properties: {
databaseAccountOfferType: 'Standard'
consistencyPolicy: {
defaultConsistencyLevel: 'Session'
}
locations: [
{
locationName: location
failoverPriority: 0
isZoneRedundant: true
}
]
}
}
resource db 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases@2024-05-15' = {
parent: cosmos
name: 'saas'
properties: {
resource: { id: 'saas' }
}
}
resource records 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2024-05-15' = {
parent: db
name: 'records'
properties: {
resource: {
id: 'records'
partitionKey: {
paths: ['/tenantId']
kind: 'Hash'
}
indexingPolicy: {
indexingMode: 'consistent'
automatic: true
includedPaths: [{ path: '/*' }]
excludedPaths: [{ path: '/"_etag"/?' }]
}
}
options: {
autoscaleSettings: {
maxThroughput: 4000
}
}
}
}
resource bus 'Microsoft.ServiceBus/namespaces@2022-10-01-preview' = {
name: serviceBusName
location: location
sku: {
name: 'Premium'
tier: 'Premium'
capacity: 1
}
}
resource queue 'Microsoft.ServiceBus/namespaces/queues@2022-10-01-preview' = {
parent: bus
name: 'work'
properties: {
requiresDuplicateDetection: true
duplicateDetectionHistoryTimeWindow: 'PT10M'
deadLetteringOnMessageExpiration: true
maxDeliveryCount: 10
}
}
output cosmosEndpoint string = cosmos.properties.documentEndpoint
output serviceBusNamespace string = bus.properties.serviceBusEndpoint
Notice that the container autoscale maximum, Service Bus messaging unit, retention, and API versions must be validated against the target region, subscription quotas, and workload before rollout; they are starting assumptions, not capacity guarantees.
The two Service Bus role definition IDs in this compact example should be verified against the role names in the target Azure environment before deployment. In production, assign Azure Service Bus Data Sender only to the API or dispatcher that publishes, and Azure Service Bus Data Receiver only to the worker. Assign Cosmos DB Built-in Data Contributor at the narrowest practical scope. A production Bicep module should create those role assignments explicitly and ensure applications tolerate role-assignment propagation delay.
Use a GitHub Actions workflow with workload identity federation for Azure login. Do not store a long-lived Azure client secret in repository secrets.
name: build-and-deploy
on:
push:
branches: [main]
pull_request:
permissions:
contents: read
id-token: write
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- run: dotnet restore
- run: dotnet test --configuration Release --no-restore
deploy-staging:
if: github.event_name == 'push'
needs: test
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/checkout@v4
- uses: azure/login@v2
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- name: Build and push image
run: |
az acr build \
--registry "$ACR_NAME" \
--image "saas-api:${GITHUB_SHA}" \
--file src/Api/Dockerfile .
env:
ACR_NAME: ${{ vars.ACR_NAME }}
- name: Validate and deploy infrastructure
run: |
az deployment group what-if \
--resource-group "$RESOURCE_GROUP" \
--template-file infra/main.bicep \
--parameters image="$IMAGE" acrName="$ACR_NAME" cosmosName="$COSMOS_NAME" serviceBusName="$BUS_NAME"
az deployment group create \
--resource-group "$RESOURCE_GROUP" \
--template-file infra/main.bicep \
--parameters image="$IMAGE" acrName="$ACR_NAME" cosmosName="$COSMOS_NAME" serviceBusName="$BUS_NAME"
env:
RESOURCE_GROUP: ${{ vars.STAGING_RESOURCE_GROUP }}
IMAGE: ${{ vars.STAGING_IMAGE }}
ACR_NAME: ${{ vars.ACR_NAME }}
COSMOS_NAME: ${{ vars.STAGING_COSMOS_NAME }}
BUS_NAME: ${{ vars.STAGING_BUS_NAME }}
- name: Update API image
run: az containerapp update --name saas-api --resource-group "$RESOURCE_GROUP" --image "$IMAGE"
env:
RESOURCE_GROUP: ${{ vars.STAGING_RESOURCE_GROUP }}
IMAGE: ${{ vars.STAGING_IMAGE }}
Notice that the workflow deploys only from main; production should use a separately protected environment, approval gates, and the same immutable commit image.
The workflow is illustrative: define the image parameter as a full ACR login-server path in a real deployment, and split infrastructure creation from application rollout when their permissions or lifecycles differ. Prefer a staged Container Apps revision rollout with health checks and a rollback procedure over an unobserved image replacement.
The test strategy should be layered:
- Unit: domain rules, tenant authorization policies, event serialization, and retry classification.
- Integration: Cosmos transactional batches, partition-key behavior, Service Bus sending/receiving, and managed-identity authorization in a non-production Azure environment.
- Contract: versioned event schemas and compatibility checks between dispatcher and worker.
- Load: representative tenant distribution, burst traffic, queue backlog recovery, and Cosmos RU consumption. Include a deliberately large tenant rather than testing only uniformly small tenants.
Observability
Instrument the API, dispatcher, and worker. Carry a trace context through the outbox event into Service Bus application properties so the worker can continue the distributed trace. Add tenantId only in a privacy-reviewed, appropriately bounded form; never use tenant IDs, user IDs, or message IDs as metric labels because high-cardinality dimensions make metrics expensive and hard to query.
A structured log event should capture enough context to investigate a failure without logging credentials, message bodies, or personal data:
{
"timestamp": "2026-11-04T12:00:00Z",
"level": "Information",
"service": "orders-api",
"operation": "CreateOrder",
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"tenantId": "ten_42",
"orderId": "ord_01J9A7",
"outcome": "accepted",
"durationMs": 82
}
Use .NET OpenTelemetry instrumentation and export using OTLP to the configured collector or Azure Monitor OpenTelemetry distribution. Add the relevant OpenTelemetry packages to the application.
using System.Diagnostics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService("orders-api"))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource("SaaS.Orders")
.AddOtlpExporter());
var app = builder.Build();
var source = new ActivitySource("SaaS.Orders");
app.MapPost("/orders", () =>
{
using var activity = source.StartActivity("orders.accept");
activity?.SetTag("app.operation", "create-order");
return Results.Accepted("/operations/op-123");
});
app.Run();
Notice that the custom activity joins automatic ASP.NET Core and HTTP client instrumentation; configure the OTLP endpoint and authentication for the actual deployment.
Track RED (rate, errors, duration) for HTTP endpoints and message processing. Track USE (utilization, saturation, errors) for the resources that can constrain the service:
| Signal | Useful measurements |
|---|---|
| API RED | Requests/s by route template, 5xx and timeout rate, p50/p95/p99 latency |
| Worker RED | Messages completed/s, processing failures, processing duration |
| Service Bus | Active message count, age of oldest message, dead-letter count, throttling |
| Cosmos DB | RU consumption and throttling responses, request latency, availability |
| Container Apps USE | CPU and memory utilization, replica count, scale limits, restarts |
| Platform USE | Log ingestion volume, storage and network saturation where instrumented |
| SLI | Initial SLO | Measurement |
|---|---|---|
| API availability | 99.9% monthly | Successful eligible requests divided by eligible requests, excluding agreed client errors |
| API latency | 95% under 500 ms | Server-side latency for successful API requests, by route class |
| Accepted-command durability | 99.99% of acknowledged writes remain represented | Reconcile accepted command records and outbox records; validate with fault tests |
| Queue completion | 99% of valid messages complete within 5 minutes | Message enqueue time to successful consumer completion |
| Recovery | Restore service within 4 hours for a regional incident | Tested recovery exercise, not an inferred platform capability |
Page on sustained user-visible API error-budget burn, API availability failure, a rapidly growing oldest-message age that threatens the completion SLO, Cosmos throttling that persists despite backoff, and dead-letter growth above an agreed threshold. Alert on symptoms and customer impact, not every replica or transient retry. Dashboards should separate API health, async backlog, tenant-aware support diagnostics, and deployment/revision health.
Security and compliance
Use separate identities and permissions for the API, dispatcher, and worker. The API should not have receive rights on the queue; the worker should not have send rights unless its role requires them. Use managed identities for Cosmos DB and Service Bus access. ACR pull uses a dedicated identity or a clearly scoped app identity. Avoid connection strings and account keys in application settings.
The public API needs TLS, authentication, tenant authorization, request-size limits, rate limiting, and input validation. Keep worker ingress internal or disabled. For private network access to Cosmos DB and Service Bus, add a VNet-integrated Container Apps environment, private endpoints, private DNS, and an explicit outbound strategy. That is a meaningful infrastructure and operational step, not a checkbox: test name resolution, deployment access, image pulls, and observability paths.
Encrypt data in transit and at rest using the service capabilities and key-management policy appropriate to the product. Use Key Vault only for secrets that cannot be eliminated with identity-based access. Define retention and deletion for tenant data, outbox records, dead-letter messages, logs, and backups. A dead-letter queue can contain sensitive payloads and needs the same access and retention discipline as the primary store.
For compliance, map controls to the actual product obligations and customer contracts. Do not imply that selecting Azure services alone makes an application compliant. Maintain a data inventory, access reviews, incident response procedures, audit trails for sensitive actions, and tested deletion and restore workflows.
Cost
This is an assumption-based monthly estimate, not an Azure price quote. Actual cost depends on region, negotiated pricing, uptime, throughput, selected networking, log volume, retention, and support plan. Recheck the Azure Pricing Calculator and measured usage before committing to a budget.
| Area | Planning allowance (USD/month) | Assumption |
|---|---|---|
| Container Apps | $250–$700 | API and worker scale with demand; modest baseline replicas and burst capacity |
| Cosmos DB | $300–$800 | Single-region, zone-redundant configuration where available, autoscale throughput, 200 GB data assumption |
| Service Bus | $700–$1,100 | One Premium messaging unit, queue traffic, and availability-oriented tier |
| Container Registry | $30–$100 | Private image storage and routine image pulls |
| Azure Monitor and logs | $100–$400 | 30-day retention, moderate structured logs, sampled traces |
| Networking and Key Vault | $100–$400 | Allowance for private connectivity and supporting resources, if enabled |
| Estimated total | $1,480–$3,500 | Excludes taxes, support plan, DR region, and exceptional egress or log volume |
The budget is sensitive to two design choices: Premium Service Bus for the intended availability characteristics and the amount of Cosmos DB throughput actually consumed. A low-traffic system may still pay for a minimum premium tier; a high-churn log strategy can outgrow the application compute bill. Put budget alerts on subscriptions and resource groups, and review cost per tenant and per successfully completed unit of work.
Failure modes, resilience and disaster recovery
| Failure | Expected behavior | Operational response |
|---|---|---|
| API replica crashes | In-flight request may fail; client retries with an idempotency key | Monitor restarts and 5xx; make command submission safe to retry |
| Cosmos throttles | SDK retries within configured bounds; API may return a controlled retryable response | Inspect RU use and hot partitions; change queries or throughput based on evidence |
| Service Bus unavailable | Outbox retains unsent intent; API can continue only while database capacity and product policy allow | Alert on oldest pending outbox event and resume dispatch after recovery |
| Worker crashes after database commit | Message redelivers; inbox marker prevents duplicate side effects | Verify duplicate handling and delivery count |
| Poison message | Repeated delivery eventually dead-letters | Inspect reason and schema; fix or explicitly replay with auditability |
| One tenant is unusually hot | Partition-level contention may affect that tenant and its operations | Identify tenant skew; throttle, batch, or redesign partition strategy |
| Regional outage | Single-region baseline may be unavailable | Follow documented restore/failover plan; do not claim multi-region recovery before testing |
Service Bus Premium availability-zone support and Cosmos DB zone redundancy should be confirmed for the selected region and configuration. Automatic service resiliency does not replace application retry limits, poison-message handling, or restore drills.
For regional disaster recovery, first decide the required recovery point objective (RPO) and recovery time objective (RTO). A backup-and-restore strategy is simpler but usually has a longer recovery time. A second Cosmos DB region and a regional messaging strategy can reduce recovery time, but introduce routing, data consistency, failover, and duplicate-processing concerns. Keep recovery steps versioned, test them on a schedule, and verify that restored identities, networking, DNS, images, and configuration work—not merely that data can be copied.
Evolving to the next scale
Do not split the modular monolith merely because the architecture diagram has several boxes. Split a capability when an independent deployment boundary solves a real constraint: different scaling behavior, ownership, security boundary, release cadence, or failure isolation. At this scale, the API and worker already have separate runtime scaling while sharing a repository and domain contracts.
| Signal | Change to consider |
|---|---|
| Sustained queue age breaches the completion SLO | Increase worker concurrency or replicas after validating Cosmos and downstream capacity; then assess workload-specific queues |
| A tenant dominates partition traffic | Introduce a tenant-specific partitioning strategy or workload isolation; this may require data migration |
| Read traffic drives RU cost | Add carefully invalidated caches for stable, non-sensitive reads; consider dedicated read models for proven hot queries |
| Event consumers need independent release cycles | Split a worker or bounded context into a separate Container Apps app with its own contract and operational ownership |
| Regional recovery objectives tighten | Design and rehearse a multi-region topology with explicit data and messaging failover |
| Log volume obscures useful signals | Apply sampling, structured retention policies, and per-category ingestion controls before increasing retention |
Caching is not a default correctness improvement. Define what can be stale, how invalidation works, and whether tenant scope is part of the cache key. Likewise, a second queue is not isolation if every worker still saturates the same Cosmos partition or downstream API.
Phased implementation plan
- Agree on product and operational contracts. Define tenant authorization, request idempotency, async status semantics, message schema ownership, data retention, SLIs, and recovery objectives. Review access patterns before fixing the Cosmos partition key.
- Build a thin vertical slice. Deploy one API, one worker, one Service Bus queue, and one Cosmos container to a development environment. Use managed identities and verify the full publish, process, retry, and duplicate-delivery path.
- Establish infrastructure and delivery. Create reusable Bicep modules, separate staging and production configuration, image scanning, protected deployment environments, and immutable image tagging. Add integration and contract tests before adding more capabilities.
- Prove operational behavior. Create RED/USE dashboards, queue-age and dead-letter alerts, structured logs, traces, budget alerts, and a runbook. Load test tenant skew, throttling, message bursts, and worker recovery.
- Harden and launch. Add network restrictions and private endpoints if required, validate regional availability features, perform a restore exercise, and conduct a production readiness review with the on-call owner.
- Scale from evidence. Review SLO performance, RU consumption, queue age, tenant distribution, and cost per workload. Change partitioning, caching, service boundaries, or regional topology only when an observed constraint or product requirement justifies the added complexity.
The central trade-off is deliberate: Cosmos DB and Service Bus make asynchronous, tenant-scoped workloads manageable, but they do not remove distributed-systems failure modes. Durable outbox intent, idempotent consumers, explicit tenant boundaries, and observable recovery paths are the architecture—not optional refinements.
This article was generated with AI. #AIGenerated