Skip to content

Internal Admin Tool with Django, HTMX, and Celery: A Small-Team Blueprint

A practical AWS architecture for a small internal admin tool, from Django and HTMX request handling through Celery jobs, safe releases, observability, and recovery.

Ekanatha Reddy Urivakili22 min4,205 words

1. Context and scale

An internal admin tool is often treated as a handful of forms around an existing database. That underestimates its risk. Staff actions can change customer access, issue refunds, export sensitive data, or trigger work that affects thousands of records. The application needs a clear permission model, durable job state, and an audit trail even when its interface is deliberately small.

This blueprint builds a single Django application with server-rendered pages and HTMX interactions. Celery runs work that should not hold an HTTP request open. PostgreSQL is the source of truth for both business data and job state; Amazon SQS transports tasks. The system runs in one AWS region on ECS Fargate. It is intentionally a modular monolith, not a collection of services.

Assumptions — sizing inputs, not measured requirements:

DimensionAssumption
UsersUnder 1,000 daily users; 20–50 staff accounts, with fewer than 10 concurrent operators
Traffic1 request/s typical, peaks around 10 requests/s during business hours
Data volumeUp to 100 GB PostgreSQL data; up to 50 GB of generated exports in S3
Background workTens to hundreds of jobs daily; a few jobs may process thousands of records
Team size1–3 engineers, with shared operations responsibility
AvailabilitySingle-region service; business-hours support; a short recovery window is acceptable
BudgetAssumed infrastructure budget of roughly $250–$500/month, excluding engineering time, support plans, and unusually high data transfer

These assumptions favor managed services and a small number of deployable units. The web process and worker process use the same image and codebase, but run as separate ECS services because their scaling and failure behavior differ. PostgreSQL is managed; there is no Kubernetes cluster, cache tier, service mesh, or dedicated data platform.

The admin tool should not become a second source of truth. It calls the application’s domain services and writes through the same database rules as the rest of the system. If it must update another product’s database directly, that is a separate integration risk that should be addressed explicitly rather than hidden behind an admin screen.

2. Requirements

Functional requirements

RequirementDesign response
Authenticate staffIntegrate with the organization’s OIDC identity provider; retain Django sessions and local authorization
Restrict actions by roleDjango groups and explicit permissions; sensitive actions require a permission check in the server-side view
Search and inspect recordsServer-rendered Django views with bounded, paginated queries
Perform bulk or slow actionsCreate a persisted job record, enqueue work, and show status with HTMX polling
Export dataGenerate in a worker, store privately in S3, and provide a short-lived authorized download
Explain who did whatAppend-only audit records for privileged actions, including actor, target, reason, and outcome
Retry operational workRetry transient failures safely; provide a staff-controlled retry for eligible failed jobs
Preserve a usable interfaceReturn HTML fragments for HTMX requests and complete HTML pages for normal navigation

Non-functional requirements

RequirementTarget or constraint
Request responsivenessMost interactive requests should complete within 500 ms at the assumed load; long work must not run in the request
DurabilityA committed job request must be discoverable after a web-process restart
CorrectnessRetried tasks must be idempotent or detect that their effect already occurred
SecurityLeast-privilege access, CSRF protection, private exports, encrypted transport and storage
OperabilityOne deployable codebase, structured logs, useful service and queue metrics, documented recovery
MaintainabilityKeep business rules in Django services/models, not in templates or Celery task wrappers
DeploymentBuild once, promote the same immutable image, and use migrations that tolerate a rolling release
RecoverySingle-region recovery from database backups; no promise of automatic regional failover

These targets are deliberately modest. A staff console does not need a complex availability topology to be responsible. It does need to avoid silently losing a requested action or showing an operator a success state when the work has not even been accepted.

3. High-level design

The browser talks only to Django through an Application Load Balancer. Django renders pages and HTMX fragments, checks authorization, writes the audit and job records, and commits an outbox record in the same PostgreSQL transaction. A Celery worker polls the outbox and publishes to SQS; it then consumes the queued task and updates the job record as work progresses.

The outbox closes an important gap: committing a job row and publishing a message are two independent operations. Without an outbox, the process could die after the database commit but before enqueueing. With it, the work request remains visible in the database and a dispatcher can retry publication. Duplicate delivery is still possible, so task handlers remain idempotent.

flowchart LR
    Browser --> ALB["AWS Application Load Balancer"]
    ALB --> Web["ECS Fargate Django web"]
    Web --> DB["Amazon RDS PostgreSQL"]
    Web --> Outbox["Outbox rows in PostgreSQL"]
    Worker["ECS Fargate Celery worker and beat"] --> Outbox
    Worker --> Queue["Amazon SQS task queue"]
    Queue --> Worker
    Worker --> DB
    Worker --> S3["Amazon S3 private exports"]
    Web --> IdP["Organization OIDC provider"]
    Web --> Logs["Amazon CloudWatch"]
    Worker --> Logs
    DB --> Backup["RDS automated backups"]

The browser-facing web service, worker, database, queue, and export storage remain separate operational components within one application.

The worker runs Celery Beat alongside the worker process at a fixed desired count of one. Beat schedules the outbox dispatcher at a short interval; only one Beat instance should run. If worker availability later requires multiple replicas, separate Beat into a singleton task or replace the periodic dispatcher with a dedicated event-driven design. Do not scale the worker service blindly while embedding multiple Beat instances.

SQS is the broker, not the job database. Job status, actor, timestamps, and failure summaries live in PostgreSQL so the interface remains useful after queue messages are consumed. Celery’s result backend is disabled. The queue’s visibility timeout must exceed the maximum expected task runtime, and tasks that may run longer should extend visibility or be split into bounded chunks.

4. Cloud service mapping

AWS is the chosen cloud because its managed queue, database, container, and object-storage services fit this small deployment without introducing a platform team.

ConcernAWS choiceAzure equivalentGoogle Cloud equivalent
Container executionECS on AWS FargateAzure Container AppsCloud Run services and jobs
HTTP ingress and TLSApplication Load Balancer and AWS Certificate ManagerContainer Apps ingress and managed certificatesCloud Run ingress and Google-managed certificates
Relational databaseAmazon RDS for PostgreSQLAzure Database for PostgreSQLCloud SQL for PostgreSQL
Celery brokerAmazon SQSAzure Service Bus queuesCloud Tasks or Pub/Sub, depending on delivery model
Export storageAmazon S3Azure Blob StorageCloud Storage
Container registryAmazon ECRAzure Container RegistryArtifact Registry
SecretsAWS Secrets ManagerAzure Key VaultSecret Manager
Logs and metricsAmazon CloudWatchAzure MonitorCloud Logging and Cloud Monitoring
TracesADOT Collector with AWS X-Ray exportAzure Monitor OpenTelemetryGoogle Cloud Trace via OpenTelemetry
Staff identityExisting OIDC provider; AWS does not own application identityMicrosoft Entra ID or another OIDC providerCloud Identity or another OIDC provider

SQS is not interchangeable with Redis in every Celery deployment. It avoids operating a broker cache, but has different visibility and delivery semantics. Keep tasks short, make retries explicit, and do not depend on Celery features that require arbitrary broker-side scheduling. For scheduled business work, write a durable schedule or use a managed scheduler to create a task at its due time.

ECS Fargate is a better fit here than a serverless function per operation: Django remains a normal web application, and Celery workers can run continuously. The trade-off is that even a lightly used service has baseline container and database costs.

5. Data design

Use PostgreSQL for domain records, job lifecycle, audit history, and the transactional outbox. This avoids inventing a second persistence layer for job status. Keep exports in S3 because they can be large, should not inflate database backups, and need independent lifecycle and access controls.

A job record is a user-visible operational object. It is not merely a Celery task ID. Its state transitions are controlled by application code: queued, running, succeeded, or failed. The audit row records the operator’s intent and result; the job row tracks execution. For especially sensitive domains, audit records should be exported to a separately controlled retention system because a database administrator can modify records in the application database.

erDiagram
    USER ||--o{ JOB_RUN : requests
    USER ||--o{ AUDIT_EVENT : performs
    JOB_RUN ||--o{ OUTBOX_EVENT : dispatches
    JOB_RUN {
        uuid id PK
        bigint requested_by_id FK
        varchar kind
        varchar status
        varchar idempotency_key UK
        jsonb parameters
        timestamptz created_at
        timestamptz started_at
        timestamptz finished_at
    }
    AUDIT_EVENT {
        bigint id PK
        bigint actor_id FK
        varchar action
        varchar target_type
        varchar target_id
        jsonb details
        timestamptz created_at
    }
    OUTBOX_EVENT {
        uuid id PK
        uuid job_run_id FK
        varchar event_type
        timestamptz published_at
        timestamptz created_at
    }

The job and outbox records are committed together; audit records describe the operator action.

A compact schema for the operational tables follows. Django migrations should be the source of truth in the application; this DDL makes the intended constraints explicit.

CREATE TABLE job_run (
    id uuid PRIMARY KEY,
    requested_by_id bigint NOT NULL REFERENCES auth_user(id),
    kind varchar(80) NOT NULL,
    status varchar(16) NOT NULL
        CHECK (status IN ('queued', 'running', 'succeeded', 'failed')),
    idempotency_key varchar(160) NOT NULL UNIQUE,
    parameters jsonb NOT NULL DEFAULT '{}'::jsonb,
    result_summary jsonb NOT NULL DEFAULT '{}'::jsonb,
    error_code varchar(80),
    created_at timestamptz NOT NULL DEFAULT now(),
    started_at timestamptz,
    finished_at timestamptz
);

CREATE INDEX job_run_recent_idx ON job_run (created_at DESC);
CREATE INDEX job_run_queued_idx ON job_run (created_at) WHERE status = 'queued';

CREATE TABLE audit_event (
    id bigserial PRIMARY KEY,
    actor_id bigint NOT NULL REFERENCES auth_user(id),
    action varchar(100) NOT NULL,
    target_type varchar(100) NOT NULL,
    target_id varchar(160) NOT NULL,
    details jsonb NOT NULL DEFAULT '{}'::jsonb,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX audit_event_target_idx
    ON audit_event (target_type, target_id, created_at DESC);

CREATE TABLE outbox_event (
    id uuid PRIMARY KEY,
    job_run_id uuid NOT NULL REFERENCES job_run(id),
    event_type varchar(100) NOT NULL,
    published_at timestamptz,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX outbox_unpublished_idx
    ON outbox_event (created_at) WHERE published_at IS NULL;

In production, reference the actual configured user table rather than assuming Django’s default auth_user table. Add data-retention rules before the tables grow indefinitely: for example, retain operational job detail for a defined period and archive audit history according to policy. Avoid storing credentials, payment data, or full request payloads in parameters, logs, or audit details.

6. Low-level design

The application has four useful layers: Django views and forms for HTTP contracts, domain services for authorization-sensitive operations, Celery tasks for execution boundaries, and models/repositories for persistence. Do not put core business rules in a template, a task decorator, or an HTMX-specific endpoint. Both synchronous and asynchronous paths should call the same service functions.

The class diagram focuses on the bulk operation path. The interface is deliberately server-rendered: HTMX improves the interaction without introducing a client-side state store or a separate API stack.

classDiagram
    class AdminJobView {
        +post(request)
        +get(request)
    }
    class BulkActionForm {
        +clean()
        +clean_reason()
    }
    class JobService {
        +request_job(actor, kind, params, key)
        +authorize(actor, kind)
    }
    class JobRun {
        +status
        +mark_running()
        +mark_succeeded(summary)
        +mark_failed(code)
    }
    class OutboxEvent {
        +mark_published()
    }
    class OutboxDispatcher {
        +publish_pending()
    }
    class BulkActionTask {
        +run(job_id)
    }
    AdminJobView --> BulkActionForm
    AdminJobView --> JobService
    JobService --> JobRun
    JobService --> OutboxEvent
    OutboxDispatcher --> OutboxEvent
    OutboxDispatcher --> BulkActionTask
    BulkActionTask --> JobRun

Views validate requests, services enforce policy, and task handlers own retry-safe execution.

Minimal runnable example

This minimal example illustrates the boundary: the view accepts a small action and the task performs work outside the request. It assumes a Django project with Celery configured and a send_digest function supplied by the application. In a real internal tool, persist the job and audit record before enqueueing, as in the next example.

# views.py
from django.contrib.auth.decorators import permission_required
from django.http import HttpResponse
from django.shortcuts import render
from django.views.decorators.http import require_POST

from .tasks import rebuild_summary


@permission_required("reports.change_summary", raise_exception=True)
@require_POST
def rebuild(request):
    task = rebuild_summary.delay(request.POST["account_id"])
    return render(request, "admin/job_started.html", {"task_id": task.id})


# tasks.py
from celery import shared_task


@shared_task
def rebuild_summary(account_id: str) -> None:
    # Replace with a bounded, idempotent domain operation.
    print(f"Rebuilding summary for account {account_id}")

Notice that the task is queued only after the request’s authorization check; this sketch intentionally omits durable job tracking.

Realistic job request and HTMX status

The realistic path commits a job and outbox event atomically. The dispatcher publishes the event to Celery and marks it published. A crash after publishing but before marking creates a duplicate, so the task uses the job ID as its idempotency key and checks the stored state before applying effects.

# models.py
import uuid

from django.conf import settings
from django.db import models


class JobRun(models.Model):
    class Status(models.TextChoices):
        QUEUED = "queued"
        RUNNING = "running"
        SUCCEEDED = "succeeded"
        FAILED = "failed"

    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    requested_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
    kind = models.CharField(max_length=80)
    status = models.CharField(max_length=16, choices=Status.choices, default=Status.QUEUED)
    idempotency_key = models.CharField(max_length=160, unique=True)
    parameters = models.JSONField(default=dict)
    result_summary = models.JSONField(default=dict)
    error_code = models.CharField(max_length=80, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)
    started_at = models.DateTimeField(null=True, blank=True)
    finished_at = models.DateTimeField(null=True, blank=True)


class OutboxEvent(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    job = models.ForeignKey(JobRun, on_delete=models.CASCADE)
    event_type = models.CharField(max_length=100)
    published_at = models.DateTimeField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)


class AuditEvent(models.Model):
    actor = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
    action = models.CharField(max_length=100)
    target_type = models.CharField(max_length=100)
    target_id = models.CharField(max_length=160)
    details = models.JSONField(default=dict)
    created_at = models.DateTimeField(auto_now_add=True)
# services.py
import uuid

from django.core.exceptions import PermissionDenied
from django.db import transaction

from .models import AuditEvent, JobRun, OutboxEvent


@transaction.atomic
def request_export(*, actor, account_id: str, reason: str) -> JobRun:
    if not actor.has_perm("exports.add_jobrun"):
        raise PermissionDenied

    key = str(uuid.uuid4())
    job = JobRun.objects.create(
        requested_by=actor,
        kind="account_export",
        idempotency_key=key,
        parameters={"account_id": account_id, "reason": reason},
    )
    OutboxEvent.objects.create(job=job, event_type="job.requested")
    AuditEvent.objects.create(
        actor=actor,
        action="account_export.requested",
        target_type="account",
        target_id=account_id,
        details={"job_id": str(job.id), "reason": reason},
    )
    return job
# views.py
from django.contrib.auth.decorators import permission_required
from django.shortcuts import get_object_or_404, render
from django.views.decorators.http import require_GET, require_POST

from .models import JobRun
from .services import request_export


@permission_required("exports.add_jobrun", raise_exception=True)
@require_POST
def create_export(request):
    job = request_export(
        actor=request.user,
        account_id=request.POST["account_id"],
        reason=request.POST["reason"],
    )
    return render(request, "exports/_job_status.html", {"job": job})


@permission_required("exports.view_jobrun", raise_exception=True)
@require_GET
def job_status(request, job_id):
    job = get_object_or_404(JobRun, pk=job_id)
    return render(request, "exports/_job_status.html", {"job": job})
<form method="post" hx-post="/admin/exports/" hx-target="#job-status">
  {% csrf_token %}
  <label>Account ID <input name="account_id" required></label>
  <label>Reason <input name="reason" required maxlength="500"></label>
  <button type="submit">Request export</button>
</form>
<div id="job-status"></div>
<!-- templates/exports/_job_status.html -->
<div
  id="job-status"
  {% if job.status == "queued" or job.status == "running" %}
  hx-get="/admin/jobs/{{ job.id }}/"
  hx-trigger="every 3s"
  hx-swap="outerHTML"
  {% endif %}
>
  Job {{ job.id }}: {{ job.status }}
</div>

Notice that the database transaction creates the job, audit event, and outbox event together; HTMX polls the application’s persisted state, not Celery’s transient result store.

The omitted dispatcher should claim a small batch of unpublished rows, publish task IDs, and mark successful publications. Use bounded transactions and a safe concurrent-claim strategy such as select_for_update(skip_locked=True) where supported. Never keep a database lock open while waiting on SQS. If publication fails, leave the row unpublished and record a retryable error or rely on the next dispatch pass.

The worker should process a job in chunks, update progress only at useful intervals, and make external side effects idempotent. For example, an export task writes to a temporary S3 object key and only marks the job successful after upload verification. A retry must not email an operator twice or create multiple irreversible changes.

7. Key flows

The first sequence is a privileged operator request. A success response means that the request has been durably recorded, not that the background work has finished.

sequenceDiagram
    actor Staff
    participant Browser
    participant Web as Django web on ECS
    participant DB as RDS PostgreSQL
    participant Worker as Celery worker on ECS
    participant Queue as Amazon SQS
    Staff->>Browser: Submit bulk action
    Browser->>Web: POST with session and CSRF token
    Web->>Web: Check permission and validate input
    Web->>DB: Commit job, audit, and outbox
    Web-->>Browser: Return queued status fragment
    Worker->>DB: Read unpublished outbox rows
    Worker->>Queue: Publish job message
    Worker->>DB: Mark outbox row published
    Worker->>Queue: Receive job message
    Worker->>DB: Mark job running and perform work
    Worker->>DB: Store final status
    Browser->>Web: HTMX GET job status
    Web->>DB: Read job status
    Web-->>Browser: Return status fragment

The HTTP request acknowledges durable acceptance; the operator observes execution through database-backed status.

The second sequence covers failure and retry. A queue retry is not a substitute for task design: duplicate delivery is normal in distributed systems, so the worker must verify the job state and avoid repeating completed effects.

sequenceDiagram
    participant Worker as Celery worker
    participant Queue as Amazon SQS
    participant DB as RDS PostgreSQL
    participant S3 as Amazon S3
    Worker->>Queue: Receive job message
    Worker->>DB: Claim queued job
    Worker->>S3: Write export object
    S3-->>Worker: Temporary service error
    Worker->>DB: Record retryable attempt outcome
    Worker->>Queue: Retry task with backoff
    Queue-->>Worker: Deliver message again
    Worker->>DB: Check job state and idempotency
    Worker->>S3: Retry upload safely
    S3-->>Worker: Object stored
    Worker->>DB: Mark job succeeded

Retryable failures preserve the job’s identity and avoid presenting an intermediate error as a final result.

8. Build and release

Build one container image containing Django, Celery, templates, static assets, and migrations. ECS task definitions choose the process command for the web or worker role. Keep settings and secrets outside the image. Apply database migrations as a controlled release step before shifting web traffic, using expand-and-contract changes: add compatible schema first, deploy code that uses it, and remove obsolete schema only in a later release.

For a small team, Terraform provides reviewable infrastructure changes and a clear state boundary. The following fragment illustrates the execution model; production code should also define networking, security groups, task roles, log groups, alarms, and encryption settings. State belongs in a locked, encrypted remote backend, not in a developer’s local directory.

resource "aws_sqs_queue" "jobs_dlq" {
  name                      = "admin-jobs-dlq"
  message_retention_seconds = 1209600
  sqs_managed_sse_enabled   = true
}

resource "aws_sqs_queue" "jobs" {
  name                       = "admin-jobs"
  visibility_timeout_seconds = 1800
  message_retention_seconds  = 345600
  sqs_managed_sse_enabled    = true

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.jobs_dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_ecs_service" "web" {
  name            = "admin-web"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.web.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  load_balancer {
    target_group_arn = aws_lb_target_group.web.arn
    container_name   = "web"
    container_port   = 8000
  }

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.web.id]
    assign_public_ip = false
  }
}

resource "aws_ecs_service" "worker" {
  name            = "admin-worker"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.worker.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.worker.id]
    assign_public_ip = false
  }
}

The worker task role should receive only the permissions it needs: consume and delete messages from the intended SQS queue, write to the designated S3 prefix, and emit telemetry. The web task role should not inherit worker privileges. The database security group should accept connections only from the ECS task security groups.

A GitHub Actions pipeline should run tests before building and pushing. Production deployment should use a protected environment with approval where the organization requires it. The essential property is promotion of the same image digest, not a rebuild with potentially different dependencies.

name: admin-tool
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.x"
      - run: pip install -r requirements-dev.txt
      - run: ruff check .
      - run: python manage.py check
      - run: pytest

  build:
    if: github.event_name == 'push'
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}
      - uses: aws-actions/amazon-ecr-login@v2
        id: ecr
      - run: |
          docker build -t "${{ steps.ecr.outputs.registry }}/admin:${GITHUB_SHA}" .
          docker push "${{ steps.ecr.outputs.registry }}/admin:${GITHUB_SHA}"

Pin action versions to reviewed commit SHAs in a hardened production workflow. The abbreviated pipeline above demonstrates the stages; add image scanning, migration execution, ECS deployment, health verification, and rollback automation before relying on it for production.

Test strategy

Test layerWhat to verifyPractical approach
UnitAuthorization decisions, form validation, state transitions, idempotencyFast tests with mocked external effects
IntegrationPostgreSQL transaction behavior, outbox dispatch, S3 and SQS adaptersDisposable PostgreSQL and local or test AWS resources
ContractOIDC claims, queue message shape, upstream APIsValidate stable schemas and failure responses
LoadSearch pages, job creation, polling, and worker throughputTest assumed peak traffic and a burst of job submissions
ReleaseMigration compatibility, task health, static assets, rollbackDeploy to a production-like environment using the candidate image

Load tests should include queue backlog and worker restart scenarios, not just HTTP throughput. The critical question is how quickly the system drains a realistic backlog while keeping interactive database queries responsive.

9. Observability

Use one correlation vocabulary across Django, Celery, PostgreSQL logs, and SQS messages: request ID, job ID, actor ID, task name, and trace ID. Do not log session cookies, authorization headers, export contents, or raw form payloads. For staff actions, record enough context to explain the event without duplicating sensitive data.

A structured log record should be machine-queryable and should distinguish an operator request from task execution:

{
  "timestamp": "2026-11-12T14:03:22Z",
  "level": "INFO",
  "event": "admin_job.requested",
  "service": "admin-web",
  "request_id": "req-7d6f",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "job_id": "9a0c1d8e-0f52-4b0e-9912-232b9ebd3a43",
  "actor_id": "184",
  "job_kind": "account_export",
  "outcome": "accepted"
}

Track RED for the web and worker services: request/task rate, errors, and duration. Track USE for resources: utilization, saturation, and errors for ECS CPU and memory, PostgreSQL connections and storage, and queue depth. Queue age is more actionable than message count alone: it tells operators whether work is waiting longer than the product can tolerate.

For traces, instrument Django and Celery with OpenTelemetry’s Python libraries and export OTLP to an AWS Distro for OpenTelemetry Collector sidecar, which forwards traces to X-Ray. Keep the queue and job identifiers in span attributes, but avoid putting personal or sensitive values into attributes.

# observability.py, loaded during Django/Celery startup
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.celery import CeleryInstrumentor
from opentelemetry.instrumentation.django import DjangoInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create({"service.name": "internal-admin"})
)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)

DjangoInstrumentor().instrument()
CeleryInstrumentor().instrument()

Notice that the application exports through a local collector endpoint; configure the collector’s AWS exporter and network permissions in the ECS task definition.

Define SLIs around operator-visible outcomes rather than infrastructure vanity metrics:

SLISLOMeasurement window
Successful authenticated page and action requests99.5% excluding planned maintenance and client errorsRolling 30 days
Interactive request latency95% under 500 ms for non-export requestsRolling 30 days
Job acceptance durability99.9% of accepted requests have a persisted job and outbox recordRolling 30 days
Job start delay95% of eligible jobs start within 2 minutesRolling 30 days
Export completion95% of normal-sized exports complete within 15 minutesRolling 30 days

These are starting objectives, not guarantees. Refine “normal-sized” and eligible-job definitions from observed workload. Page on sustained user-impacting failures: web availability or error-budget burn, database unavailability, a growing oldest-message age that breaches the start-delay objective, or repeated dead-letter queue growth. Send lower-severity notifications for worker CPU pressure, near-full storage, certificate expiry, and isolated task failures that are retrying successfully.

Maintain a web dashboard with request rate, error rate, latency, ECS task health, database connections, and deployment markers. A worker dashboard should show queue depth and oldest message age, task duration and outcomes, retries, outbox backlog age, and dead-letter count. The deployment dashboard should correlate application errors with image versions and migration events.

10. Security and compliance

An internal URL is not an authorization boundary. Require staff authentication through the organization’s OIDC provider, enforce permissions on every server-side action, and require MFA through the identity provider. Django groups are a reasonable starting point for a small team; use explicit permissions for high-impact actions rather than a single all-powerful “admin” role.

Apply these controls from the first release:

  • Use HTTPS at the load balancer; secure Django cookies, enable CSRF middleware, and validate trusted hosts and origins.
  • Keep ECS tasks in private subnets. Permit ingress to web tasks only from the load balancer and database access only from the application security group.
  • Use separate ECS task roles for web and worker. Store database credentials and signing keys in Secrets Manager and rotate them under an operational procedure.
  • Encrypt RDS, SQS, S3, backups, and logs at rest. Keep export buckets private, block public access, and issue short-lived pre-signed downloads only after an authorization check.
  • Record actor, action, target, reason, and outcome for sensitive operations. Do not treat Django’s general application log as an audit system.
  • Limit export scope, expire generated files automatically, and verify that downloaded files are not accessible after the intended retention period.
  • Review dependency updates and container images. Restrict production deployment roles through GitHub’s OIDC federation rather than long-lived AWS keys.

Compliance is not achieved by selecting managed services. Define data classification, retention, access reviews, incident handling, and evidence requirements with the organization’s security and legal owners. If the tool handles regulated data, confirm the applicable AWS services and configuration against the relevant compliance program before deployment.

11. Cost

The following is an assumption-based monthly estimate, not a vendor quote. It assumes one AWS region, a modest always-on service, low request volume, private networking, a small PostgreSQL database, limited exports, and no unusually high egress. Actual charges depend on region, current pricing, storage, backup retention, network design, and telemetry volume.

ComponentAssumed monthly rangeCost drivers
ECS Fargate web and worker tasks$50–$110Two small web tasks and one small worker task, running continuously
Application Load Balancer$20–$40Hours, capacity units, and traffic
RDS PostgreSQL$45–$100Small instance, storage, backups, and deployment configuration
NAT and network transfer$35–$90NAT gateway hours and outbound traffic; endpoint choices change this
SQS, S3, and ECR$5–$25Queue requests, export volume, image storage, and retention
CloudWatch and tracing$15–$60Log volume, retention, metrics, and trace sampling
Secrets and certificates$0–$20Secret count and service-specific charges
Estimated total$170–$445Excludes support plan, labor, taxes, and unusually high egress

The lower end assumes disciplined log retention and small workload. The upper end allows for more telemetry, network transfer, and database capacity. Treat a budget near $250–$500 as a planning envelope, not a guarantee. Check the current AWS pricing calculator before committing. Review actual cost per completed job and cost per active staff user monthly; those reveal waste better than a single total.

Cost optimizations should not remove the properties the tool depends on. Do not make the database public or remove backups to save a small amount. Instead, right-size task resources from measurements, cap log retention, expire exports, and decide deliberately whether private-subnet NAT costs justify VPC endpoints for the services in use.

12. Failure modes, resilience, and disaster recovery

The design accepts single-region risk in exchange for simpler operations. It still needs to handle ordinary process, queue, and database failures predictably.

FailureExpected behaviorRecovery
Web task exitsLoad balancer routes to healthy task; ECS replaces the failed taskInspect logs and deployment; no manual data repair normally needed
Worker exitsSQS retains messages; ECS restarts the workerCheck queue age and worker health; verify tasks are idempotent
SQS is unavailableOutbox rows remain unpublished; requests can still be durably acceptedRestore publishing; monitor oldest outbox age
PostgreSQL is unavailableWeb actions fail clearly; no false “accepted” responseRestore RDS availability; review backups if storage or data is damaged
Task fails repeatedlyMessage eventually reaches the DLQ; job remains visible as failed or needs interventionInspect error, correct cause, and replay through a controlled action
S3 upload failsJob remains retryable; do not expose a partial export as completeRetry safely and verify the object before success
Bad release or migrationHealth checks fail or requests error after rolloutRoll back compatible application code; repair forward for destructive schema changes
Accidental data changeAudit may identify actor and target, but does not reverse the mutationRestore or reconcile from a tested backup procedure

Use RDS automated backups and point-in-time recovery appropriate to the data’s recovery point objective. Set a recovery time objective with the business; do not imply that snapshots create an instant restore. Test restoration into a separate instance, verify application access, and document how to repoint the ECS service. Keep infrastructure definitions and deployment artifacts available independently of the production account.

A regional outage requires a manual recovery plan unless a second region is explicitly funded and tested. For this scale, document how to recreate infrastructure in another region, restore a database backup, restore or replicate required S3 data, and update DNS. The plan should state the expected data-loss window and recovery duration based on a rehearsal, not aspiration.

For queue recovery, distinguish messages waiting in SQS from jobs awaiting dispatch in the outbox. A reconciliation command should identify old queued jobs without a corresponding live message, stale running jobs, and published outbox rows whose jobs never started. Provide a dry-run mode and an audit trail before any automated replay.

13. Evolving to the next scale

Do not split the monolith because “scale” has arrived in the abstract. Change a component when a measured constraint or ownership boundary justifies its additional deployment and failure modes.

SignalFirst changeAvoid doing prematurely
Queue age regularly breaches the job-start SLOIncrease worker concurrency or task size; separate queues by workload classCreating a worker service for every task type
One workload starves interactive readsUse bounded batches, query optimization, and a dedicated worker capacity poolMoving all writes to a new database
Database CPU, connections, or storage becomes the bottleneckTune queries and indexes; add connection pooling or a larger RDS instanceSharding before understanding query patterns
Web releases and worker releases need different cadenceKeep the shared codebase, but version compatible task contracts and deploy roles independentlySplitting into microservices solely to deploy separately
Audit retention or access controls outgrow the application databaseExport immutable audit events to a governed archive or security accountTreating ordinary application logs as tamper-proof audit records
Regional recovery becomes a business requirementDesign and rehearse cross-region database and object recoveryClaiming multi-region readiness from replicated artifacts alone
Admin capability becomes a product surface used by customersRevisit tenancy, authorization boundaries, API contracts, and threat modelExposing internal views by simply adding a public route

The architecture’s scaling boundary is initially the worker pool and PostgreSQL. Both can grow independently without changing the user-facing application. If the application becomes a critical operational console, invest in stronger deployment isolation, tested recovery objectives, and explicit service ownership before adding architectural components.

14. Phased implementation plan

A small team should build the operational path early, then add high-risk actions only after its controls are in place.

  1. Define the action and data boundary. List the first administrative workflows, the data each can read or change, the authorization roles, and the required audit fields. Agree on retention and recovery objectives with the owner of the data.
  2. Build the Django foundation. Add OIDC login, Django permissions, CSRF-safe forms, pagination, and a consistent server-rendered layout. Implement the first read-only workflow before enabling mutations.
  3. Establish the job contract. Add JobRun, OutboxEvent, and AuditEvent; define allowed state transitions and idempotency behavior. Build the HTMX status fragment and test it without Celery first.
  4. Deploy the smallest production-shaped environment. Provision ECS, ALB, RDS, SQS, S3, secrets, and logs through Terraform. Use separate web and worker task roles and private networking.
  5. Add the worker and recovery hooks. Implement outbox dispatch, bounded tasks, retry policies, a dead-letter queue, and a dry-run reconciliation command. Confirm that duplicate messages do not duplicate effects.
  6. Instrument and test operational behavior. Add structured logs, traces, dashboards, alarms, and SLO measurement. Test database outage, worker restart, queue delay, and an export retry.
  7. Release progressively. Deploy read-only use first, then one low-risk write action, then higher-impact bulk actions. Review authorization and audit evidence with operators after each phase.
  8. Rehearse recovery and review cost. Restore a database backup into an isolated environment, validate the recovery notes, and compare actual service costs with the assumptions. Adjust task sizing and retention from evidence.

The key design choice is not Django versus a separate API, or HTMX versus a JavaScript framework. It is whether the tool treats staff actions as durable, authorized operations. A small Django monolith, HTMX fragments, Celery workers, PostgreSQL, and SQS are enough to do that well—provided the job lifecycle, audit trail, retries, and recovery plan are part of the design from the beginning.


This article was generated with AI. #AIGenerated