1. Context and scale
A business application accepts documents from authenticated users, extracts structured data, validates the result, and makes the processing status and extracted fields available through an API. Typical inputs are invoices, forms, and supporting documents. Some documents finish quickly; others wait on downstream services or retry after transient faults.
The architecture uses Azure Durable Functions as the workflow coordinator, not as a place to store documents or perform long-running CPU work. Blob Storage owns the files, Azure SQL owns application state, and Azure AI Document Intelligence performs extraction. Durable Functions coordinates activities and persists orchestration state in its task hub storage.
The design targets a medium product: one modular application and a handful of Azure services, operated by one or two teams. It does not assume active-active multi-region operation or an enterprise data platform.
| Dimension | Assumption |
|---|---|
| Users | 25,000 daily active users |
| Requests | 0.5 average API requests/second, with short peaks around 8 requests/second |
| Documents | 10,000 submitted per day; average 3 pages per document |
| Data volume | About 30,000 processed pages/day; source and result storage sized for retention and reprocessing |
| Team | 5 engineers across application, platform, and test responsibilities |
| Budget | Target of about $3,000/month for the production workload; usage and regional prices must be checked before commitment |
These are assumptions, not capacity guarantees or Azure price quotations. The workload model deliberately separates API traffic from document-processing volume: a modest request rate can still create substantial OCR and storage costs.
2. Requirements
Functional requirements
| ID | Requirement | Design response |
|---|---|---|
| F1 | Accept a document without sending its bytes through the application API | API issues a short-lived, blob-scoped upload URL |
| F2 | Start processing only after a complete upload | Blob-created event is routed through Event Grid and Service Bus |
| F3 | Track status and return a stable submission identifier | Azure SQL stores the business record; the API exposes status |
| F4 | Extract fields and retain source and normalized output | Document Intelligence processes the source; results are stored in Blob Storage and SQL |
| F5 | Retry transient failures without duplicating business effects | Durable activity retries plus idempotent activity implementations |
| F6 | Support reprocessing and operator investigation | Persist model/configuration version, attempt data, and workflow instance ID |
Non-functional requirements
| ID | Requirement | Target or constraint |
|---|---|---|
| N1 | Keep the upload path responsive | Upload directly to Blob Storage; API handles metadata and authorization only |
| N2 | Tolerate process restarts and transient downstream failures | Durable task hub persists orchestration state; activities are retryable |
| N3 | Avoid duplicate submissions caused by event redelivery | Enforce a unique submission key and make state transitions idempotent |
| N4 | Protect sensitive documents | Entra ID, managed identities, scoped upload access, encryption, and least privilege |
| N5 | Make operational health visible | Correlated logs, traces, queue metrics, workflow-age dashboards, and SLOs |
| N6 | Recover from a regional outage with controlled data loss | Documented secondary-region recovery; no claim of automatic cross-region orchestration failover |
A workflow being durable does not make every activity exactly once. Activities may run again after a failure. The application must make their externally visible effects safe to repeat.
3. High-level design
The browser or client obtains an upload authorization from the API, then uploads directly to Blob Storage. Event Grid sends the completed-blob event to a Service Bus queue. A queue-triggered Function validates the event, finds the pre-created SQL submission, and starts a Durable Functions orchestration. Activities validate, extract, normalize, and persist results.
The task hub is backed by a dedicated Azure Storage account. Keep business documents and orchestration state separate: their access patterns, retention requirements, and operational failure modes differ.
flowchart LR
Client["Browser or client"] --> Static["Azure Static Web Apps"]
Static --> APIM["Azure API Management"]
APIM --> Api["Azure Functions API"]
Api --> SQL["Azure SQL Database"]
Api --> Blob["Azure Blob Storage"]
Blob --> Events["Azure Event Grid"]
Events --> Queue["Azure Service Bus queue"]
Queue --> Starter["Azure Functions starter"]
Starter --> Durable["Durable Functions task hub"]
Durable --> Orch["Document orchestration"]
Orch --> Activities["Azure Functions activities"]
Activities --> DI["Azure AI Document Intelligence"]
Activities --> SQL
Activities --> Blob
Orch --> Hub["Azure Storage task hub"]
Api --> KV["Azure Key Vault"]
Activities --> Monitor["Azure Monitor and Application Insights"]The workflow separates the upload path, event transport, orchestration state, document storage, and business state.
The diagram is intentionally a small number of components. Avoid adding a separate microservice for each activity at this scale; Functions can remain one deployable application with clear modules and independently testable activities.
4. Cloud service mapping
| Component | Azure choice | AWS equivalent | GCP equivalent |
|---|---|---|---|
| Web client hosting | Azure Static Web Apps | AWS Amplify Hosting | Firebase Hosting |
| API gateway | Azure API Management | Amazon API Gateway | Apigee or API Gateway |
| API and workflow compute | Azure Functions, .NET isolated | AWS Lambda | Cloud Run functions |
| Durable workflow | Durable Functions | AWS Step Functions | Workflows |
| Event routing | Azure Event Grid | Amazon EventBridge | Eventarc |
| Queue and buffering | Azure Service Bus | Amazon SQS | Cloud Tasks or Pub/Sub, depending on delivery needs |
| Documents and task-hub storage | Azure Blob Storage | Amazon S3 | Cloud Storage |
| OCR and document extraction | Azure AI Document Intelligence | Amazon Textract | Document AI |
| Relational business state | Azure SQL Database | Amazon RDS for SQL Server or Aurora | Cloud SQL |
| Secrets | Azure Key Vault | AWS Secrets Manager | Secret Manager |
| Logs, metrics, traces | Azure Monitor and Application Insights | CloudWatch and AWS X-Ray | Cloud Monitoring and Cloud Trace |
The equivalents are functional analogues, not identical products. In particular, queue ordering, workflow replay, event delivery, and identity integration differ; porting the architecture requires redesigning those contracts.
5. Data design
Use Azure SQL for submission metadata and current status, not for document bytes or Durable Functions history. Blob Storage retains the source document and normalized extraction output. The Durable Functions task hub stores orchestration state and is managed as runtime infrastructure, not queried as the application’s reporting database.
The core relational model needs to support status lookup, idempotent event handling, and reprocessing. Store a stable client submission key and make it unique within the tenant. Store the Durable instance ID for support correlation, but treat the SQL status as the application-facing source of truth.
erDiagram
TENANT ||--o{ SUBMISSION : owns
SUBMISSION ||--o{ PROCESSING_ATTEMPT : has
SUBMISSION {
uniqueidentifier submission_id PK
uniqueidentifier tenant_id FK
nvarchar client_key
nvarchar source_blob_name
nvarchar status
nvarchar orchestration_id
datetime2 created_at
datetime2 updated_at
}
PROCESSING_ATTEMPT {
uniqueidentifier attempt_id PK
uniqueidentifier submission_id FK
int attempt_number
nvarchar model_version
nvarchar outcome
nvarchar result_blob_name
nvarchar error_code
datetime2 started_at
datetime2 completed_at
}
TENANT {
uniqueidentifier tenant_id PK
nvarchar name
}Business records track ownership, current processing status, and each processing attempt.
A simplified SQL Server schema follows. In production, add tenant foreign keys to the tenant table, role-based access policies as required, and retention controls aligned with the product’s data policy.
CREATE TABLE dbo.Submission (
SubmissionId uniqueidentifier NOT NULL PRIMARY KEY,
TenantId uniqueidentifier NOT NULL,
ClientKey nvarchar(120) NOT NULL,
SourceBlobName nvarchar(512) NOT NULL,
ResultBlobName nvarchar(512) NULL,
Status varchar(24) NOT NULL,
OrchestrationId nvarchar(100) NULL,
CreatedAtUtc datetime2 NOT NULL,
UpdatedAtUtc datetime2 NOT NULL,
RowVersion rowversion NOT NULL,
CONSTRAINT UQ_Submission_Tenant_ClientKey UNIQUE (TenantId, ClientKey)
);
CREATE INDEX IX_Submission_Tenant_Status_Created
ON dbo.Submission (TenantId, Status, CreatedAtUtc DESC);
CREATE TABLE dbo.ProcessingAttempt (
AttemptId uniqueidentifier NOT NULL PRIMARY KEY,
SubmissionId uniqueidentifier NOT NULL,
AttemptNumber int NOT NULL,
ModelVersion nvarchar(80) NOT NULL,
Outcome varchar(24) NOT NULL,
ResultBlobName nvarchar(512) NULL,
ErrorCode nvarchar(100) NULL,
StartedAtUtc datetime2 NOT NULL,
CompletedAtUtc datetime2 NULL,
CONSTRAINT FK_Attempt_Submission
FOREIGN KEY (SubmissionId) REFERENCES dbo.Submission(SubmissionId),
CONSTRAINT UQ_Attempt_Number UNIQUE (SubmissionId, AttemptNumber)
);
Keep extracted data in a versioned JSON result blob when the schema varies by document type. Put frequently filtered, product-critical fields in typed columns only when query requirements justify the migration and indexing cost. Never persist access tokens or SAS URLs as document metadata.
6. Low-level design
The Functions application has three small modules:
- Intake: creates the submission and grants narrowly scoped upload access.
- Workflow: starts and coordinates a Durable Functions orchestration.
- Processing: validates input, calls Document Intelligence, normalizes output, and commits results.
The orchestrator must be deterministic. It should coordinate activities and timers, but should not call SQL, Blob Storage, HTTP services, or generate nondeterministic values directly. Put those operations in activities. Keep orchestration inputs small: pass identifiers and blob names, not document contents.
classDiagram
class UploadApi {
+CreateSubmission(tenantId, clientKey)
+CreateUploadUrl(submissionId)
}
class QueueStarter {
+StartFromBlobEvent(event)
}
class DocumentOrchestrator {
+Run(input)
}
class ValidateActivity {
+Run(input)
}
class ExtractActivity {
+Run(input)
}
class PersistActivity {
+Run(input)
}
class SubmissionRepository {
+GetById(id)
+SetStatus(id, status)
+SaveAttempt(result)
}
UploadApi --> SubmissionRepository
QueueStarter --> DocumentOrchestrator
DocumentOrchestrator --> ValidateActivity
DocumentOrchestrator --> ExtractActivity
DocumentOrchestrator --> PersistActivity
ValidateActivity --> SubmissionRepository
PersistActivity --> SubmissionRepositoryActivities own I/O; the orchestrator owns sequencing and retry policy.
Minimal runnable example
This minimal .NET isolated Durable Functions example demonstrates the orchestration boundary. It assumes a Functions project configured with the Durable Task extension and the required storage setting.
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;
public static class HelloWorkflow
{
[Function(nameof(HelloWorkflow))]
public static async Task<string> Run(
[OrchestrationTrigger] TaskOrchestrationContext context)
{
var name = context.GetInput<string>() ?? "document";
return await context.CallActivityAsync<string>(
nameof(Greet), name);
}
[Function(nameof(Greet))]
public static string Greet([ActivityTrigger] string name) =>
$"Ready to process {name}";
}
Notice that the orchestrator calls an activity instead of doing external work itself, which keeps replay deterministic.
Realistic workflow example
The following example shows a starter and a retryable orchestration. The starter receives a queue message after Blob Storage reports a committed upload. Production code should validate the event’s container and tenant ownership before scheduling work, and should record the instance ID in SQL using an idempotent update.
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
public sealed record DocumentJob(
Guid SubmissionId,
string BlobName,
string ModelId);
public sealed record ExtractionResult(
string ResultBlobName,
string ModelVersion);
public static class DocumentWorkflow
{
[Function(nameof(StartDocumentWorkflow))]
public static async Task StartDocumentWorkflow(
[ServiceBusTrigger("document-events", Connection = "ServiceBus")]
DocumentJob job,
[DurableClient] DurableTaskClient durableClient)
{
// Use SubmissionId as the business idempotency key in SQL.
await durableClient.ScheduleNewOrchestrationInstanceAsync(
nameof(ProcessDocument), job);
}
[Function(nameof(ProcessDocument))]
public static async Task<string> ProcessDocument(
[OrchestrationTrigger] TaskOrchestrationContext context)
{
var job = context.GetInput<DocumentJob>()
?? throw new InvalidOperationException("Missing workflow input.");
var retry = new TaskOptions(
new TaskRetryOptions(
firstRetryInterval: TimeSpan.FromSeconds(10),
maxNumberOfAttempts: 4));
await context.CallActivityAsync(
nameof(ValidateDocument), job);
var result = await context.CallActivityAsync<ExtractionResult>(
nameof(ExtractAndStoreResult), job, retry);
await context.CallActivityAsync(
nameof(CommitSubmission), (job.SubmissionId, result));
return result.ResultBlobName;
}
[Function(nameof(ValidateDocument))]
public static Task ValidateDocument(
[ActivityTrigger] DocumentJob job)
{
// Activity implementation checks blob existence, size, and file type.
// It must be safe to execute again.
return Task.CompletedTask;
}
[Function(nameof(ExtractAndStoreResult))]
public static Task<ExtractionResult> ExtractAndStoreResult(
[ActivityTrigger] DocumentJob job)
{
// Production implementation reads the blob, calls Document Intelligence,
// normalizes the response, and writes a versioned result blob.
throw new NotImplementedException(
"Inject Blob and Document Intelligence clients here.");
}
[Function(nameof(CommitSubmission))]
public static Task CommitSubmission(
[ActivityTrigger] (Guid SubmissionId, ExtractionResult Result) input)
{
// Use a transaction/upsert guarded by SubmissionId and attempt number.
throw new NotImplementedException(
"Inject the repository and persist status plus attempt data.");
}
}
Notice that retries apply to an activity, while idempotency belongs in the activity’s persistence contract—not in the retry setting.
The example leaves cloud client construction and persistence behind injected interfaces rather than pretending a short snippet is a complete production application. Register SDK clients once through dependency injection, use managed identity for Azure resources, and write activity tests against those interfaces. For current Durable Functions isolated-worker setup and binding details, see the Durable Functions overview and the isolated worker guide.
7. Key flows
Upload and workflow start
The upload path is asynchronous by design. The API creates the submission before returning upload authorization, so the later event can be associated with a known tenant and submission. Event Grid and Service Bus can redeliver; the starter must handle duplicate events safely.
sequenceDiagram
participant C as Client
participant A as API Management and Functions API
participant S as Azure SQL
participant B as Blob Storage
participant E as Event Grid
participant Q as Service Bus
participant F as Queue-triggered Function
participant D as Durable Functions
C->>A: Create submission
A->>S: Insert pending record
A-->>C: Submission ID and scoped upload URL
C->>B: Upload document
B->>E: Blob created event
E->>Q: Enqueue event
Q->>F: Deliver event
F->>S: Validate submission and deduplicate
F->>D: Schedule orchestration
F-->>Q: Complete messageThe queue buffers event bursts and decouples upload completion from workflow scheduling.
Process, persist, and report
The workflow advances business status through explicit transitions. If extraction fails permanently, persist a failure outcome and expose a supportable error code; do not leave the submission indefinitely in a generic “processing” state.
sequenceDiagram
participant D as Durable orchestration
participant V as Validation activity
participant B as Blob Storage
participant X as Extraction activity
participant I as Document Intelligence
participant P as Persistence activity
participant S as Azure SQL
participant C as Client
D->>V: Validate document reference
V->>B: Read metadata and content
V-->>D: Valid
D->>X: Extract fields
X->>B: Read source document
X->>I: Analyze document
I-->>X: Extraction response
X->>B: Write normalized result
X-->>D: Result reference
D->>P: Commit outcome
P->>S: Save attempt and completed status
C->>S: Query submission status
S-->>C: Completed status and result referenceThe workflow stores references and results, not document bytes, in its durable history.
8. Build and release
Deploy one Functions application with separate staging and production environments. Use infrastructure as code for the service boundary, identity assignments, monitoring, and network settings. Keep secrets out of deployment parameters; prefer managed identity and Key Vault references where a secret is unavoidable.
Bicep infrastructure excerpt
This excerpt provisions a Functions Premium plan, a storage account for the Functions runtime and Durable task hub, and an Application Insights resource. It is a starting point, not a complete environment: production deployments also need the application, SQL, Service Bus, Blob Storage, Key Vault, identities, role assignments, and network rules.
param location string = resourceGroup().location
param appName string
param storageName string
param planName string
param appInsightsName string
resource taskStorage 'Microsoft.Storage/storageAccounts@2023-05-01' = {
name: storageName
location: location
sku: {
name: 'Standard_LRS'
}
kind: 'StorageV2'
properties: {
minimumTlsVersion: 'TLS1_2'
supportsHttpsTrafficOnly: true
allowBlobPublicAccess: false
}
}
resource plan 'Microsoft.Web/serverfarms@2023-12-01' = {
name: planName
location: location
kind: 'elastic'
sku: {
name: 'EP1'
tier: 'ElasticPremium'
}
properties: {
reserved: true
}
}
resource insights 'Microsoft.Insights/components@2020-02-02' = {
name: appInsightsName
location: location
kind: 'web'
properties: {
Application_Type: 'web'
}
}
resource functionApp 'Microsoft.Web/sites@2023-12-01' = {
name: appName
location: location
kind: 'functionapp,linux'
identity: {
type: 'SystemAssigned'
}
properties: {
serverFarmId: plan.id
httpsOnly: true
siteConfig: {
linuxFxVersion: 'DOTNET-ISOLATED|10.0'
appSettings: [
{
name: 'FUNCTIONS_WORKER_RUNTIME'
value: 'dotnet-isolated'
}
{
name: 'FUNCTIONS_EXTENSION_VERSION'
value: '~4'
}
{
name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
value: insights.properties.ConnectionString
}
{
name: 'AzureWebJobsStorage'
value: 'Use managed identity in the deployed app configuration'
}
]
}
}
}
The storage setting above is intentionally a deployment placeholder: configure the supported identity-based connection settings for the Functions host and Durable extension, then grant only the required data-plane roles. Do not replace it with a storage account key in source control. Validate the selected Functions runtime and .NET target against Azure’s current support matrix before release.
CI/CD pipeline
A small team benefits more from repeatable promotion and rollback than from a large release platform. Build once, run tests and static checks, deploy to staging, perform a smoke test, then promote the same artifact to production with approval.
name: document-workflow
on:
pull_request:
push:
branches: [main]
jobs:
test-and-package:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: "10.x"
- run: dotnet restore
- run: dotnet build --configuration Release --no-restore
- run: dotnet test --configuration Release --no-build
- run: dotnet publish src/DocumentFunctions -c Release -o publish
- uses: actions/upload-artifact@v4
with:
name: functions-package
path: publish/
deploy-staging:
if: github.ref == 'refs/heads/main'
needs: test-and-package
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/download-artifact@v4
with:
name: functions-package
path: publish/
- run: echo "Deploy package and Bicep to the staging environment"
- run: echo "Run authenticated upload and workflow smoke test"
promote-production:
if: github.ref == 'refs/heads/main'
needs: deploy-staging
runs-on: ubuntu-latest
environment: production
steps:
- run: echo "Promote the tested artifact after environment approval"
Replace the deployment placeholders with Azure login through workload identity federation, Bicep validation and deployment, and the chosen Functions deployment action or CLI command. Pin third-party actions to reviewed versions or commit SHAs according to organizational policy.
Test strategy
| Test level | What to verify | Typical approach |
|---|---|---|
| Unit | Normalization, validation, status transitions, retry classification | Pure .NET tests with fake repositories and service clients |
| Integration | SQL constraints, Blob access, Service Bus trigger behavior, Durable task-hub configuration | Ephemeral Azure test resources or isolated environment |
| Contract | Event payload compatibility, API request/response shapes, extraction result schema | Versioned JSON fixtures and consumer/provider checks |
| Load | Queue growth and drain time, SQL contention, extraction concurrency, API throttling | Replay representative document sizes and page counts in a cost-controlled environment |
Test duplicate events, activity replay, poison messages, and failed model calls explicitly. A happy-path test that only checks “completed” does not validate durable behavior.
9. Observability
Instrument the application around business operations, not only function invocations. A useful correlation chain contains the submission ID, Durable instance ID, queue message ID, tenant identifier in a privacy-safe form, activity name, and model version. Do not log document contents, SAS tokens, personal data, or raw extraction payloads.
Structured logging
Use stable field names and bounded-cardinality values. Do not put submission IDs or tenant IDs into metric labels; they belong in logs and traces.
{
"timestamp": "2026-11-16T12:00:00Z",
"level": "Information",
"event": "document_processing_completed",
"submission_id": "8d7c9c6a-2d73-4abc-a719-2a041f69d2e7",
"durable_instance_id": "document-workflow-7f5f...",
"activity": "ExtractAndStoreResult",
"model_version": "invoice-v3",
"duration_ms": 1840,
"outcome": "completed",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}
RED and USE signals
RED tracks service behavior: Rate, Errors, Duration. USE tracks resource behavior: Utilization, Saturation, Errors. Monitor both the Functions app and its dependencies; queue age can reveal a backlog before API users report stale results.
| Scope | Signals |
|---|---|
| API and Functions (RED) | Request and activity rate; failures by dependency and outcome; duration percentiles; throttling responses |
| Service Bus (USE and backlog) | Active messages, oldest-message age, delivery count, dead-letter count, namespace capacity |
| SQL (USE) | CPU, storage, connections, I/O, deadlocks, throttling, connection-pool pressure |
| Storage and Document Intelligence | Request failures, latency, throttling, capacity, extraction failures by model and document type |
OpenTelemetry tracing
In .NET, create custom spans inside activities and use the Azure Monitor OpenTelemetry exporter for export. Do not create custom spans inside orchestrator code: orchestrator replay can make instrumentation misleading. The SDK and Functions host instrumentation should be configured and verified in the deployed environment.
using System.Diagnostics;
using OpenTelemetry.Trace;
public static class DocumentTelemetry
{
public static readonly ActivitySource Source =
new("Ekanatha.DocumentProcessing");
public static void Configure(IServiceCollection services)
{
services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddSource(Source.Name)
.AddAzureMonitorTraceExporter());
}
}
public static async Task RunExtractionActivity(
Guid submissionId,
Func<CancellationToken, Task> extract,
CancellationToken cancellationToken)
{
using var activity = DocumentTelemetry.Source.StartActivity(
"document.extract",
ActivityKind.Internal);
activity?.SetTag("submission.id", submissionId.ToString());
activity?.SetTag("document.operation", "extract");
await extract(cancellationToken);
}
Add exporter configuration and package references in the application’s startup and project files. Validate end-to-end trace propagation through the queue and Durable activity boundaries rather than assuming all dependencies preserve context identically.
SLI/SLOs, dashboards, and paging
| SLI | Initial SLO | Measurement |
|---|---|---|
| API availability | 99.9% monthly | Successful authorized API requests excluding client errors |
| Upload authorization latency | 95% under 1 second | API request duration |
| Processing completion | 95% of supported documents complete within 10 minutes | Submission creation to terminal state, excluding declared client-side upload time |
| Workflow correctness | At least 99.5% reach a terminal state without operator intervention | Completed or explicitly failed submissions over a rolling window |
These are starting targets, not observed performance claims. Tune the processing target by document class and extraction service behavior.
Dashboards should show API RED signals, queue age and dead letters, workflow age by status, extraction outcomes by model version, SQL health, and monthly cost by service. Page on sustained API availability or latency SLO burn, a growing oldest-message age that threatens the completion SLO, SQL unavailability, or a sudden rise in dead-letter messages. Send isolated document failures and non-urgent retries to a team queue rather than paging on every failed activity.
10. Security and compliance
Documents are sensitive by default. Treat access to source blobs, extraction output, and workflow status as tenant-scoped authorization decisions.
- Authenticate users with Microsoft Entra ID; authorize tenant and submission access in the API.
- Issue short-lived, single-blob upload authorization with the minimum required permissions. Do not let the client choose arbitrary container paths.
- Use managed identities from Functions to access SQL, Blob Storage, Service Bus, Key Vault, and monitoring where supported.
- Disable public blob access. Use private endpoints and network restrictions for data services where the environment’s network design supports them.
- Grant the event-routing identity permission to send only to the target queue; grant the Functions identity only the data-plane roles it needs.
- Enforce file size and type limits, validate content independently of the filename, and quarantine suspicious uploads before processing.
- Apply retention and deletion policies to originals, normalized outputs, SQL rows, and Durable task-hub data. Define which records must be retained for audit and which must be erased.
- Keep secrets and connection strings out of logs, deployment artifacts, and application settings wherever identity-based access is available.
Compliance depends on the customer’s jurisdiction, data classification, configuration, and operating procedures. Select the applicable Azure compliance commitments and validate the specific services and regions with the organization’s security and legal teams; the architecture alone is not a compliance certification.
11. Cost
The table is an assumption-based monthly estimate, not a quote. It models the workload in section 1, a single production region, moderate retention, and one staging environment. Regional rates, redundancy settings, network egress, and actual extraction features can materially change the result.
| Service or cost area | Monthly assumption |
|---|---|
| Functions Premium plan and execution headroom | $350 |
| Azure SQL Database with zone-redundant configuration | $500 |
| Azure AI Document Intelligence, modeled page volume | $900 |
| Blob Storage for source and result retention | $120 |
| Durable task-hub and Functions storage transactions | $90 |
| Service Bus Standard and Event Grid operations | $80 |
| API Management | $250 |
| Application Insights and Log Analytics ingestion/retention | $250 |
| Key Vault, networking, and private endpoint allowance | $250 |
| Static Web Apps and staging overhead | $100 |
| Illustrative total | $2,890 |
The largest uncertainty is document extraction pricing and the amount of diagnostic data retained. Measure pages per document, extraction features, reprocessing frequency, and log ingestion during a pilot. Use the Azure pricing calculator with the actual region and service configuration before setting a production budget.
Cost controls should include per-tenant submission limits, queue back-pressure, bounded retries, log sampling or retention rules where appropriate, and alerts for spend anomalies. Avoid reducing retries so aggressively that transient faults create manual work; optimize from measured failure modes.
12. Failure modes, resilience, and disaster recovery
| Failure | Expected behavior | Operator action |
|---|---|---|
| Duplicate Blob Storage event | Starter recognizes the existing submission and avoids a second business workflow | Inspect duplicate rate if it rises materially |
| Service Bus delivery or Function outage | Messages remain queued and are retried; poison messages move to the dead-letter queue | Diagnose, repair, then replay with an audited procedure |
| Activity timeout or transient dependency error | Durable retry policy reruns the activity | Check attempt count and dependency throttling |
| Permanent invalid document | Workflow records a terminal failure with a stable error code | Notify the user or route to manual review |
| SQL unavailable | Commit activity retries; status may remain pending or processing temporarily | Restore SQL, then verify workflow completion and reconciliation |
| Document Intelligence throttling | Bounded retries and queue buffering prevent unbounded concurrent pressure | Reduce concurrency or adjust capacity after reviewing service limits |
| Region unavailable | Traffic and processing stop or move according to the regional recovery runbook | Fail over infrastructure and reconcile in-flight submissions |
Enable availability-zone support for eligible production services where the selected region and service tier support it. Back up SQL with an appropriate retention policy. Select Blob redundancy based on recovery objectives and data residency constraints.
For regional recovery, deploy a tested secondary environment from infrastructure as code and maintain a documented decision process for routing traffic, restoring or promoting data, and reconciling in-flight work. Durable task-hub state is not a substitute for a cross-region recovery plan. Do not run two regions against the same task hub as an improvised active-active design. After failover, compare SQL submissions with orchestration state and queue messages, then restart eligible work through an idempotent recovery command.
Set explicit recovery objectives with the product owner: maximum acceptable data loss for metadata, maximum processing interruption, and whether uploaded documents can be reprocessed. Test restoration and replay, including the effect of duplicate extraction charges.
13. Evolving to the next scale
Do not split this design into more services merely because the queue grows. First identify the constrained resource and whether the constraint is sustained or a short burst.
| Signal from production | Change to consider |
|---|---|
| Queue age rises while Functions has headroom | Inspect queue-trigger concurrency, poison messages, and downstream throttling before adding compute |
| Extraction capacity or cost dominates | Apply document-type routing, page limits, batching where supported, and explicit per-tenant quotas |
| SQL is the bottleneck | Review indexes and query patterns; separate read-heavy status queries or partition archival data if justified |
| One workflow becomes large or frequently revised | Version orchestration inputs and activity contracts; split only when independent ownership or scaling is real |
| A team needs independent deployment and fault boundaries | Extract a bounded processing service with a stable queue contract and ownership model |
| Recovery objectives require regional continuity | Build and regularly test a regional recovery strategy; consider multi-region routing only with explicit consistency and cost trade-offs |
Durable Functions history grows with orchestration activity. Keep workflows bounded, avoid returning large payloads from activities, and consider sub-orchestrations or ContinueAsNew for genuinely long-running or high-event workflows. Use monitoring and measured history growth to decide when; do not introduce those mechanisms pre-emptively into a short linear workflow.
14. Phased implementation plan
- Prove the contract. Define submission states, idempotency keys, upload limits, tenant authorization, and the result schema. Build a small local workflow with fake extraction and persistence clients.
- Build the secure intake path. Provision Blob Storage, SQL, identity, and the upload API. Verify that clients cannot read other tenants’ documents or upload to arbitrary paths.
- Add event transport and orchestration. Connect Event Grid to Service Bus, implement the starter, and add deterministic orchestration with idempotent activities. Test duplicate delivery and replay.
- Integrate extraction. Add Document Intelligence behind an interface, capture model/version metadata, classify transient versus permanent errors, and establish extraction-cost controls.
- Make operations release-ready. Add structured logs, OpenTelemetry, dashboards, alerts, staging deployment, smoke tests, and a dead-letter replay runbook.
- Validate failure and recovery. Load test representative files and bursts; test SQL, queue, and extraction interruptions; restore data and rehearse regional recovery before committing production SLOs.
The architectural center is a simple boundary: Durable Functions coordinates work; business data and side effects remain in services designed to own them. That separation makes retries understandable, status queryable, costs measurable, and recovery a deliberate engineering practice rather than an assumption.
This article was generated with AI. #AIGenerated