1. Context and scale
An internal admin tool is often treated as a handful of forms around an existing database. That underestimates its risk. Staff actions can change customer access, issue refunds, export sensitive data, or trigger work that affects thousands of records. The application needs a clear permission model, durable job state, and an audit trail even when its interface is deliberately small.
This blueprint builds a single Django application with server-rendered pages and HTMX interactions. Celery runs work that should not hold an HTTP request open. PostgreSQL is the source of truth for both business data and job state; Amazon SQS transports tasks. The system runs in one AWS region on ECS Fargate. It is intentionally a modular monolith, not a collection of services.
Assumptions — sizing inputs, not measured requirements:
| Dimension | Assumption |
|---|---|
| Users | Under 1,000 daily users; 20–50 staff accounts, with fewer than 10 concurrent operators |
| Traffic | 1 request/s typical, peaks around 10 requests/s during business hours |
| Data volume | Up to 100 GB PostgreSQL data; up to 50 GB of generated exports in S3 |
| Background work | Tens to hundreds of jobs daily; a few jobs may process thousands of records |
| Team size | 1–3 engineers, with shared operations responsibility |
| Availability | Single-region service; business-hours support; a short recovery window is acceptable |
| Budget | Assumed infrastructure budget of roughly $250–$500/month, excluding engineering time, support plans, and unusually high data transfer |
These assumptions favor managed services and a small number of deployable units. The web process and worker process use the same image and codebase, but run as separate ECS services because their scaling and failure behavior differ. PostgreSQL is managed; there is no Kubernetes cluster, cache tier, service mesh, or dedicated data platform.
The admin tool should not become a second source of truth. It calls the application’s domain services and writes through the same database rules as the rest of the system. If it must update another product’s database directly, that is a separate integration risk that should be addressed explicitly rather than hidden behind an admin screen.
2. Requirements
Functional requirements
| Requirement | Design response |
|---|---|
| Authenticate staff | Integrate with the organization’s OIDC identity provider; retain Django sessions and local authorization |
| Restrict actions by role | Django groups and explicit permissions; sensitive actions require a permission check in the server-side view |
| Search and inspect records | Server-rendered Django views with bounded, paginated queries |
| Perform bulk or slow actions | Create a persisted job record, enqueue work, and show status with HTMX polling |
| Export data | Generate in a worker, store privately in S3, and provide a short-lived authorized download |
| Explain who did what | Append-only audit records for privileged actions, including actor, target, reason, and outcome |
| Retry operational work | Retry transient failures safely; provide a staff-controlled retry for eligible failed jobs |
| Preserve a usable interface | Return HTML fragments for HTMX requests and complete HTML pages for normal navigation |
Non-functional requirements
| Requirement | Target or constraint |
|---|---|
| Request responsiveness | Most interactive requests should complete within 500 ms at the assumed load; long work must not run in the request |
| Durability | A committed job request must be discoverable after a web-process restart |
| Correctness | Retried tasks must be idempotent or detect that their effect already occurred |
| Security | Least-privilege access, CSRF protection, private exports, encrypted transport and storage |
| Operability | One deployable codebase, structured logs, useful service and queue metrics, documented recovery |
| Maintainability | Keep business rules in Django services/models, not in templates or Celery task wrappers |
| Deployment | Build once, promote the same immutable image, and use migrations that tolerate a rolling release |
| Recovery | Single-region recovery from database backups; no promise of automatic regional failover |
These targets are deliberately modest. A staff console does not need a complex availability topology to be responsible. It does need to avoid silently losing a requested action or showing an operator a success state when the work has not even been accepted.
3. High-level design
The browser talks only to Django through an Application Load Balancer. Django renders pages and HTMX fragments, checks authorization, writes the audit and job records, and commits an outbox record in the same PostgreSQL transaction. A Celery worker polls the outbox and publishes to SQS; it then consumes the queued task and updates the job record as work progresses.
The outbox closes an important gap: committing a job row and publishing a message are two independent operations. Without an outbox, the process could die after the database commit but before enqueueing. With it, the work request remains visible in the database and a dispatcher can retry publication. Duplicate delivery is still possible, so task handlers remain idempotent.
flowchart LR
Browser --> ALB["AWS Application Load Balancer"]
ALB --> Web["ECS Fargate Django web"]
Web --> DB["Amazon RDS PostgreSQL"]
Web --> Outbox["Outbox rows in PostgreSQL"]
Worker["ECS Fargate Celery worker and beat"] --> Outbox
Worker --> Queue["Amazon SQS task queue"]
Queue --> Worker
Worker --> DB
Worker --> S3["Amazon S3 private exports"]
Web --> IdP["Organization OIDC provider"]
Web --> Logs["Amazon CloudWatch"]
Worker --> Logs
DB --> Backup["RDS automated backups"]The browser-facing web service, worker, database, queue, and export storage remain separate operational components within one application.
The worker runs Celery Beat alongside the worker process at a fixed desired count of one. Beat schedules the outbox dispatcher at a short interval; only one Beat instance should run. If worker availability later requires multiple replicas, separate Beat into a singleton task or replace the periodic dispatcher with a dedicated event-driven design. Do not scale the worker service blindly while embedding multiple Beat instances.
SQS is the broker, not the job database. Job status, actor, timestamps, and failure summaries live in PostgreSQL so the interface remains useful after queue messages are consumed. Celery’s result backend is disabled. The queue’s visibility timeout must exceed the maximum expected task runtime, and tasks that may run longer should extend visibility or be split into bounded chunks.
4. Cloud service mapping
AWS is the chosen cloud because its managed queue, database, container, and object-storage services fit this small deployment without introducing a platform team.
| Concern | AWS choice | Azure equivalent | Google Cloud equivalent |
|---|---|---|---|
| Container execution | ECS on AWS Fargate | Azure Container Apps | Cloud Run services and jobs |
| HTTP ingress and TLS | Application Load Balancer and AWS Certificate Manager | Container Apps ingress and managed certificates | Cloud Run ingress and Google-managed certificates |
| Relational database | Amazon RDS for PostgreSQL | Azure Database for PostgreSQL | Cloud SQL for PostgreSQL |
| Celery broker | Amazon SQS | Azure Service Bus queues | Cloud Tasks or Pub/Sub, depending on delivery model |
| Export storage | Amazon S3 | Azure Blob Storage | Cloud Storage |
| Container registry | Amazon ECR | Azure Container Registry | Artifact Registry |
| Secrets | AWS Secrets Manager | Azure Key Vault | Secret Manager |
| Logs and metrics | Amazon CloudWatch | Azure Monitor | Cloud Logging and Cloud Monitoring |
| Traces | ADOT Collector with AWS X-Ray export | Azure Monitor OpenTelemetry | Google Cloud Trace via OpenTelemetry |
| Staff identity | Existing OIDC provider; AWS does not own application identity | Microsoft Entra ID or another OIDC provider | Cloud Identity or another OIDC provider |
SQS is not interchangeable with Redis in every Celery deployment. It avoids operating a broker cache, but has different visibility and delivery semantics. Keep tasks short, make retries explicit, and do not depend on Celery features that require arbitrary broker-side scheduling. For scheduled business work, write a durable schedule or use a managed scheduler to create a task at its due time.
ECS Fargate is a better fit here than a serverless function per operation: Django remains a normal web application, and Celery workers can run continuously. The trade-off is that even a lightly used service has baseline container and database costs.
5. Data design
Use PostgreSQL for domain records, job lifecycle, audit history, and the transactional outbox. This avoids inventing a second persistence layer for job status. Keep exports in S3 because they can be large, should not inflate database backups, and need independent lifecycle and access controls.
A job record is a user-visible operational object. It is not merely a Celery task ID. Its state transitions are controlled by application code: queued, running, succeeded, or failed. The audit row records the operator’s intent and result; the job row tracks execution. For especially sensitive domains, audit records should be exported to a separately controlled retention system because a database administrator can modify records in the application database.
erDiagram
USER ||--o{ JOB_RUN : requests
USER ||--o{ AUDIT_EVENT : performs
JOB_RUN ||--o{ OUTBOX_EVENT : dispatches
JOB_RUN {
uuid id PK
bigint requested_by_id FK
varchar kind
varchar status
varchar idempotency_key UK
jsonb parameters
timestamptz created_at
timestamptz started_at
timestamptz finished_at
}
AUDIT_EVENT {
bigint id PK
bigint actor_id FK
varchar action
varchar target_type
varchar target_id
jsonb details
timestamptz created_at
}
OUTBOX_EVENT {
uuid id PK
uuid job_run_id FK
varchar event_type
timestamptz published_at
timestamptz created_at
}The job and outbox records are committed together; audit records describe the operator action.
A compact schema for the operational tables follows. Django migrations should be the source of truth in the application; this DDL makes the intended constraints explicit.
CREATE TABLE job_run (
id uuid PRIMARY KEY,
requested_by_id bigint NOT NULL REFERENCES auth_user(id),
kind varchar(80) NOT NULL,
status varchar(16) NOT NULL
CHECK (status IN ('queued', 'running', 'succeeded', 'failed')),
idempotency_key varchar(160) NOT NULL UNIQUE,
parameters jsonb NOT NULL DEFAULT '{}'::jsonb,
result_summary jsonb NOT NULL DEFAULT '{}'::jsonb,
error_code varchar(80),
created_at timestamptz NOT NULL DEFAULT now(),
started_at timestamptz,
finished_at timestamptz
);
CREATE INDEX job_run_recent_idx ON job_run (created_at DESC);
CREATE INDEX job_run_queued_idx ON job_run (created_at) WHERE status = 'queued';
CREATE TABLE audit_event (
id bigserial PRIMARY KEY,
actor_id bigint NOT NULL REFERENCES auth_user(id),
action varchar(100) NOT NULL,
target_type varchar(100) NOT NULL,
target_id varchar(160) NOT NULL,
details jsonb NOT NULL DEFAULT '{}'::jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX audit_event_target_idx
ON audit_event (target_type, target_id, created_at DESC);
CREATE TABLE outbox_event (
id uuid PRIMARY KEY,
job_run_id uuid NOT NULL REFERENCES job_run(id),
event_type varchar(100) NOT NULL,
published_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX outbox_unpublished_idx
ON outbox_event (created_at) WHERE published_at IS NULL;
In production, reference the actual configured user table rather than assuming Django’s default auth_user table. Add data-retention rules before the tables grow indefinitely: for example, retain operational job detail for a defined period and archive audit history according to policy. Avoid storing credentials, payment data, or full request payloads in parameters, logs, or audit details.
6. Low-level design
The application has four useful layers: Django views and forms for HTTP contracts, domain services for authorization-sensitive operations, Celery tasks for execution boundaries, and models/repositories for persistence. Do not put core business rules in a template, a task decorator, or an HTMX-specific endpoint. Both synchronous and asynchronous paths should call the same service functions.
The class diagram focuses on the bulk operation path. The interface is deliberately server-rendered: HTMX improves the interaction without introducing a client-side state store or a separate API stack.
classDiagram
class AdminJobView {
+post(request)
+get(request)
}
class BulkActionForm {
+clean()
+clean_reason()
}
class JobService {
+request_job(actor, kind, params, key)
+authorize(actor, kind)
}
class JobRun {
+status
+mark_running()
+mark_succeeded(summary)
+mark_failed(code)
}
class OutboxEvent {
+mark_published()
}
class OutboxDispatcher {
+publish_pending()
}
class BulkActionTask {
+run(job_id)
}
AdminJobView --> BulkActionForm
AdminJobView --> JobService
JobService --> JobRun
JobService --> OutboxEvent
OutboxDispatcher --> OutboxEvent
OutboxDispatcher --> BulkActionTask
BulkActionTask --> JobRunViews validate requests, services enforce policy, and task handlers own retry-safe execution.
Minimal runnable example
This minimal example illustrates the boundary: the view accepts a small action and the task performs work outside the request. It assumes a Django project with Celery configured and a send_digest function supplied by the application. In a real internal tool, persist the job and audit record before enqueueing, as in the next example.
# views.py
from django.contrib.auth.decorators import permission_required
from django.http import HttpResponse
from django.shortcuts import render
from django.views.decorators.http import require_POST
from .tasks import rebuild_summary
@permission_required("reports.change_summary", raise_exception=True)
@require_POST
def rebuild(request):
task = rebuild_summary.delay(request.POST["account_id"])
return render(request, "admin/job_started.html", {"task_id": task.id})
# tasks.py
from celery import shared_task
@shared_task
def rebuild_summary(account_id: str) -> None:
# Replace with a bounded, idempotent domain operation.
print(f"Rebuilding summary for account {account_id}")
Notice that the task is queued only after the request’s authorization check; this sketch intentionally omits durable job tracking.
Realistic job request and HTMX status
The realistic path commits a job and outbox event atomically. The dispatcher publishes the event to Celery and marks it published. A crash after publishing but before marking creates a duplicate, so the task uses the job ID as its idempotency key and checks the stored state before applying effects.
# models.py
import uuid
from django.conf import settings
from django.db import models
class JobRun(models.Model):
class Status(models.TextChoices):
QUEUED = "queued"
RUNNING = "running"
SUCCEEDED = "succeeded"
FAILED = "failed"
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
requested_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
kind = models.CharField(max_length=80)
status = models.CharField(max_length=16, choices=Status.choices, default=Status.QUEUED)
idempotency_key = models.CharField(max_length=160, unique=True)
parameters = models.JSONField(default=dict)
result_summary = models.JSONField(default=dict)
error_code = models.CharField(max_length=80, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
started_at = models.DateTimeField(null=True, blank=True)
finished_at = models.DateTimeField(null=True, blank=True)
class OutboxEvent(models.Model):
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
job = models.ForeignKey(JobRun, on_delete=models.CASCADE)
event_type = models.CharField(max_length=100)
published_at = models.DateTimeField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class AuditEvent(models.Model):
actor = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
action = models.CharField(max_length=100)
target_type = models.CharField(max_length=100)
target_id = models.CharField(max_length=160)
details = models.JSONField(default=dict)
created_at = models.DateTimeField(auto_now_add=True)
# services.py
import uuid
from django.core.exceptions import PermissionDenied
from django.db import transaction
from .models import AuditEvent, JobRun, OutboxEvent
@transaction.atomic
def request_export(*, actor, account_id: str, reason: str) -> JobRun:
if not actor.has_perm("exports.add_jobrun"):
raise PermissionDenied
key = str(uuid.uuid4())
job = JobRun.objects.create(
requested_by=actor,
kind="account_export",
idempotency_key=key,
parameters={"account_id": account_id, "reason": reason},
)
OutboxEvent.objects.create(job=job, event_type="job.requested")
AuditEvent.objects.create(
actor=actor,
action="account_export.requested",
target_type="account",
target_id=account_id,
details={"job_id": str(job.id), "reason": reason},
)
return job
# views.py
from django.contrib.auth.decorators import permission_required
from django.shortcuts import get_object_or_404, render
from django.views.decorators.http import require_GET, require_POST
from .models import JobRun
from .services import request_export
@permission_required("exports.add_jobrun", raise_exception=True)
@require_POST
def create_export(request):
job = request_export(
actor=request.user,
account_id=request.POST["account_id"],
reason=request.POST["reason"],
)
return render(request, "exports/_job_status.html", {"job": job})
@permission_required("exports.view_jobrun", raise_exception=True)
@require_GET
def job_status(request, job_id):
job = get_object_or_404(JobRun, pk=job_id)
return render(request, "exports/_job_status.html", {"job": job})
<form method="post" hx-post="/admin/exports/" hx-target="#job-status">
{% csrf_token %}
<label>Account ID <input name="account_id" required></label>
<label>Reason <input name="reason" required maxlength="500"></label>
<button type="submit">Request export</button>
</form>
<div id="job-status"></div>
<!-- templates/exports/_job_status.html -->
<div
id="job-status"
{% if job.status == "queued" or job.status == "running" %}
hx-get="/admin/jobs/{{ job.id }}/"
hx-trigger="every 3s"
hx-swap="outerHTML"
{% endif %}
>
Job {{ job.id }}: {{ job.status }}
</div>
Notice that the database transaction creates the job, audit event, and outbox event together; HTMX polls the application’s persisted state, not Celery’s transient result store.
The omitted dispatcher should claim a small batch of unpublished rows, publish task IDs, and mark successful publications. Use bounded transactions and a safe concurrent-claim strategy such as select_for_update(skip_locked=True) where supported. Never keep a database lock open while waiting on SQS. If publication fails, leave the row unpublished and record a retryable error or rely on the next dispatch pass.
The worker should process a job in chunks, update progress only at useful intervals, and make external side effects idempotent. For example, an export task writes to a temporary S3 object key and only marks the job successful after upload verification. A retry must not email an operator twice or create multiple irreversible changes.
7. Key flows
The first sequence is a privileged operator request. A success response means that the request has been durably recorded, not that the background work has finished.
sequenceDiagram
actor Staff
participant Browser
participant Web as Django web on ECS
participant DB as RDS PostgreSQL
participant Worker as Celery worker on ECS
participant Queue as Amazon SQS
Staff->>Browser: Submit bulk action
Browser->>Web: POST with session and CSRF token
Web->>Web: Check permission and validate input
Web->>DB: Commit job, audit, and outbox
Web-->>Browser: Return queued status fragment
Worker->>DB: Read unpublished outbox rows
Worker->>Queue: Publish job message
Worker->>DB: Mark outbox row published
Worker->>Queue: Receive job message
Worker->>DB: Mark job running and perform work
Worker->>DB: Store final status
Browser->>Web: HTMX GET job status
Web->>DB: Read job status
Web-->>Browser: Return status fragmentThe HTTP request acknowledges durable acceptance; the operator observes execution through database-backed status.
The second sequence covers failure and retry. A queue retry is not a substitute for task design: duplicate delivery is normal in distributed systems, so the worker must verify the job state and avoid repeating completed effects.
sequenceDiagram
participant Worker as Celery worker
participant Queue as Amazon SQS
participant DB as RDS PostgreSQL
participant S3 as Amazon S3
Worker->>Queue: Receive job message
Worker->>DB: Claim queued job
Worker->>S3: Write export object
S3-->>Worker: Temporary service error
Worker->>DB: Record retryable attempt outcome
Worker->>Queue: Retry task with backoff
Queue-->>Worker: Deliver message again
Worker->>DB: Check job state and idempotency
Worker->>S3: Retry upload safely
S3-->>Worker: Object stored
Worker->>DB: Mark job succeededRetryable failures preserve the job’s identity and avoid presenting an intermediate error as a final result.
8. Build and release
Build one container image containing Django, Celery, templates, static assets, and migrations. ECS task definitions choose the process command for the web or worker role. Keep settings and secrets outside the image. Apply database migrations as a controlled release step before shifting web traffic, using expand-and-contract changes: add compatible schema first, deploy code that uses it, and remove obsolete schema only in a later release.
For a small team, Terraform provides reviewable infrastructure changes and a clear state boundary. The following fragment illustrates the execution model; production code should also define networking, security groups, task roles, log groups, alarms, and encryption settings. State belongs in a locked, encrypted remote backend, not in a developer’s local directory.
resource "aws_sqs_queue" "jobs_dlq" {
name = "admin-jobs-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sqs_queue" "jobs" {
name = "admin-jobs"
visibility_timeout_seconds = 1800
message_retention_seconds = 345600
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.jobs_dlq.arn
maxReceiveCount = 5
})
}
resource "aws_ecs_service" "web" {
name = "admin-web"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.web.arn
desired_count = 2
launch_type = "FARGATE"
load_balancer {
target_group_arn = aws_lb_target_group.web.arn
container_name = "web"
container_port = 8000
}
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.web.id]
assign_public_ip = false
}
}
resource "aws_ecs_service" "worker" {
name = "admin-worker"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.worker.arn
desired_count = 1
launch_type = "FARGATE"
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.worker.id]
assign_public_ip = false
}
}
The worker task role should receive only the permissions it needs: consume and delete messages from the intended SQS queue, write to the designated S3 prefix, and emit telemetry. The web task role should not inherit worker privileges. The database security group should accept connections only from the ECS task security groups.
A GitHub Actions pipeline should run tests before building and pushing. Production deployment should use a protected environment with approval where the organization requires it. The essential property is promotion of the same image digest, not a rebuild with potentially different dependencies.
name: admin-tool
on:
pull_request:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- run: pip install -r requirements-dev.txt
- run: ruff check .
- run: python manage.py check
- run: pytest
build:
if: github.event_name == 'push'
needs: test
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}
- uses: aws-actions/amazon-ecr-login@v2
id: ecr
- run: |
docker build -t "${{ steps.ecr.outputs.registry }}/admin:${GITHUB_SHA}" .
docker push "${{ steps.ecr.outputs.registry }}/admin:${GITHUB_SHA}"
Pin action versions to reviewed commit SHAs in a hardened production workflow. The abbreviated pipeline above demonstrates the stages; add image scanning, migration execution, ECS deployment, health verification, and rollback automation before relying on it for production.
Test strategy
| Test layer | What to verify | Practical approach |
|---|---|---|
| Unit | Authorization decisions, form validation, state transitions, idempotency | Fast tests with mocked external effects |
| Integration | PostgreSQL transaction behavior, outbox dispatch, S3 and SQS adapters | Disposable PostgreSQL and local or test AWS resources |
| Contract | OIDC claims, queue message shape, upstream APIs | Validate stable schemas and failure responses |
| Load | Search pages, job creation, polling, and worker throughput | Test assumed peak traffic and a burst of job submissions |
| Release | Migration compatibility, task health, static assets, rollback | Deploy to a production-like environment using the candidate image |
Load tests should include queue backlog and worker restart scenarios, not just HTTP throughput. The critical question is how quickly the system drains a realistic backlog while keeping interactive database queries responsive.
9. Observability
Use one correlation vocabulary across Django, Celery, PostgreSQL logs, and SQS messages: request ID, job ID, actor ID, task name, and trace ID. Do not log session cookies, authorization headers, export contents, or raw form payloads. For staff actions, record enough context to explain the event without duplicating sensitive data.
A structured log record should be machine-queryable and should distinguish an operator request from task execution:
{
"timestamp": "2026-11-12T14:03:22Z",
"level": "INFO",
"event": "admin_job.requested",
"service": "admin-web",
"request_id": "req-7d6f",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"job_id": "9a0c1d8e-0f52-4b0e-9912-232b9ebd3a43",
"actor_id": "184",
"job_kind": "account_export",
"outcome": "accepted"
}
Track RED for the web and worker services: request/task rate, errors, and duration. Track USE for resources: utilization, saturation, and errors for ECS CPU and memory, PostgreSQL connections and storage, and queue depth. Queue age is more actionable than message count alone: it tells operators whether work is waiting longer than the product can tolerate.
For traces, instrument Django and Celery with OpenTelemetry’s Python libraries and export OTLP to an AWS Distro for OpenTelemetry Collector sidecar, which forwards traces to X-Ray. Keep the queue and job identifiers in span attributes, but avoid putting personal or sensitive values into attributes.
# observability.py, loaded during Django/Celery startup
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.celery import CeleryInstrumentor
from opentelemetry.instrumentation.django import DjangoInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider(
resource=Resource.create({"service.name": "internal-admin"})
)
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)
DjangoInstrumentor().instrument()
CeleryInstrumentor().instrument()
Notice that the application exports through a local collector endpoint; configure the collector’s AWS exporter and network permissions in the ECS task definition.
Define SLIs around operator-visible outcomes rather than infrastructure vanity metrics:
| SLI | SLO | Measurement window |
|---|---|---|
| Successful authenticated page and action requests | 99.5% excluding planned maintenance and client errors | Rolling 30 days |
| Interactive request latency | 95% under 500 ms for non-export requests | Rolling 30 days |
| Job acceptance durability | 99.9% of accepted requests have a persisted job and outbox record | Rolling 30 days |
| Job start delay | 95% of eligible jobs start within 2 minutes | Rolling 30 days |
| Export completion | 95% of normal-sized exports complete within 15 minutes | Rolling 30 days |
These are starting objectives, not guarantees. Refine “normal-sized” and eligible-job definitions from observed workload. Page on sustained user-impacting failures: web availability or error-budget burn, database unavailability, a growing oldest-message age that breaches the start-delay objective, or repeated dead-letter queue growth. Send lower-severity notifications for worker CPU pressure, near-full storage, certificate expiry, and isolated task failures that are retrying successfully.
Maintain a web dashboard with request rate, error rate, latency, ECS task health, database connections, and deployment markers. A worker dashboard should show queue depth and oldest message age, task duration and outcomes, retries, outbox backlog age, and dead-letter count. The deployment dashboard should correlate application errors with image versions and migration events.
10. Security and compliance
An internal URL is not an authorization boundary. Require staff authentication through the organization’s OIDC provider, enforce permissions on every server-side action, and require MFA through the identity provider. Django groups are a reasonable starting point for a small team; use explicit permissions for high-impact actions rather than a single all-powerful “admin” role.
Apply these controls from the first release:
- Use HTTPS at the load balancer; secure Django cookies, enable CSRF middleware, and validate trusted hosts and origins.
- Keep ECS tasks in private subnets. Permit ingress to web tasks only from the load balancer and database access only from the application security group.
- Use separate ECS task roles for web and worker. Store database credentials and signing keys in Secrets Manager and rotate them under an operational procedure.
- Encrypt RDS, SQS, S3, backups, and logs at rest. Keep export buckets private, block public access, and issue short-lived pre-signed downloads only after an authorization check.
- Record actor, action, target, reason, and outcome for sensitive operations. Do not treat Django’s general application log as an audit system.
- Limit export scope, expire generated files automatically, and verify that downloaded files are not accessible after the intended retention period.
- Review dependency updates and container images. Restrict production deployment roles through GitHub’s OIDC federation rather than long-lived AWS keys.
Compliance is not achieved by selecting managed services. Define data classification, retention, access reviews, incident handling, and evidence requirements with the organization’s security and legal owners. If the tool handles regulated data, confirm the applicable AWS services and configuration against the relevant compliance program before deployment.
11. Cost
The following is an assumption-based monthly estimate, not a vendor quote. It assumes one AWS region, a modest always-on service, low request volume, private networking, a small PostgreSQL database, limited exports, and no unusually high egress. Actual charges depend on region, current pricing, storage, backup retention, network design, and telemetry volume.
| Component | Assumed monthly range | Cost drivers |
|---|---|---|
| ECS Fargate web and worker tasks | $50–$110 | Two small web tasks and one small worker task, running continuously |
| Application Load Balancer | $20–$40 | Hours, capacity units, and traffic |
| RDS PostgreSQL | $45–$100 | Small instance, storage, backups, and deployment configuration |
| NAT and network transfer | $35–$90 | NAT gateway hours and outbound traffic; endpoint choices change this |
| SQS, S3, and ECR | $5–$25 | Queue requests, export volume, image storage, and retention |
| CloudWatch and tracing | $15–$60 | Log volume, retention, metrics, and trace sampling |
| Secrets and certificates | $0–$20 | Secret count and service-specific charges |
| Estimated total | $170–$445 | Excludes support plan, labor, taxes, and unusually high egress |
The lower end assumes disciplined log retention and small workload. The upper end allows for more telemetry, network transfer, and database capacity. Treat a budget near $250–$500 as a planning envelope, not a guarantee. Check the current AWS pricing calculator before committing. Review actual cost per completed job and cost per active staff user monthly; those reveal waste better than a single total.
Cost optimizations should not remove the properties the tool depends on. Do not make the database public or remove backups to save a small amount. Instead, right-size task resources from measurements, cap log retention, expire exports, and decide deliberately whether private-subnet NAT costs justify VPC endpoints for the services in use.
12. Failure modes, resilience, and disaster recovery
The design accepts single-region risk in exchange for simpler operations. It still needs to handle ordinary process, queue, and database failures predictably.
| Failure | Expected behavior | Recovery |
|---|---|---|
| Web task exits | Load balancer routes to healthy task; ECS replaces the failed task | Inspect logs and deployment; no manual data repair normally needed |
| Worker exits | SQS retains messages; ECS restarts the worker | Check queue age and worker health; verify tasks are idempotent |
| SQS is unavailable | Outbox rows remain unpublished; requests can still be durably accepted | Restore publishing; monitor oldest outbox age |
| PostgreSQL is unavailable | Web actions fail clearly; no false “accepted” response | Restore RDS availability; review backups if storage or data is damaged |
| Task fails repeatedly | Message eventually reaches the DLQ; job remains visible as failed or needs intervention | Inspect error, correct cause, and replay through a controlled action |
| S3 upload fails | Job remains retryable; do not expose a partial export as complete | Retry safely and verify the object before success |
| Bad release or migration | Health checks fail or requests error after rollout | Roll back compatible application code; repair forward for destructive schema changes |
| Accidental data change | Audit may identify actor and target, but does not reverse the mutation | Restore or reconcile from a tested backup procedure |
Use RDS automated backups and point-in-time recovery appropriate to the data’s recovery point objective. Set a recovery time objective with the business; do not imply that snapshots create an instant restore. Test restoration into a separate instance, verify application access, and document how to repoint the ECS service. Keep infrastructure definitions and deployment artifacts available independently of the production account.
A regional outage requires a manual recovery plan unless a second region is explicitly funded and tested. For this scale, document how to recreate infrastructure in another region, restore a database backup, restore or replicate required S3 data, and update DNS. The plan should state the expected data-loss window and recovery duration based on a rehearsal, not aspiration.
For queue recovery, distinguish messages waiting in SQS from jobs awaiting dispatch in the outbox. A reconciliation command should identify old queued jobs without a corresponding live message, stale running jobs, and published outbox rows whose jobs never started. Provide a dry-run mode and an audit trail before any automated replay.
13. Evolving to the next scale
Do not split the monolith because “scale” has arrived in the abstract. Change a component when a measured constraint or ownership boundary justifies its additional deployment and failure modes.
| Signal | First change | Avoid doing prematurely |
|---|---|---|
| Queue age regularly breaches the job-start SLO | Increase worker concurrency or task size; separate queues by workload class | Creating a worker service for every task type |
| One workload starves interactive reads | Use bounded batches, query optimization, and a dedicated worker capacity pool | Moving all writes to a new database |
| Database CPU, connections, or storage becomes the bottleneck | Tune queries and indexes; add connection pooling or a larger RDS instance | Sharding before understanding query patterns |
| Web releases and worker releases need different cadence | Keep the shared codebase, but version compatible task contracts and deploy roles independently | Splitting into microservices solely to deploy separately |
| Audit retention or access controls outgrow the application database | Export immutable audit events to a governed archive or security account | Treating ordinary application logs as tamper-proof audit records |
| Regional recovery becomes a business requirement | Design and rehearse cross-region database and object recovery | Claiming multi-region readiness from replicated artifacts alone |
| Admin capability becomes a product surface used by customers | Revisit tenancy, authorization boundaries, API contracts, and threat model | Exposing internal views by simply adding a public route |
The architecture’s scaling boundary is initially the worker pool and PostgreSQL. Both can grow independently without changing the user-facing application. If the application becomes a critical operational console, invest in stronger deployment isolation, tested recovery objectives, and explicit service ownership before adding architectural components.
14. Phased implementation plan
A small team should build the operational path early, then add high-risk actions only after its controls are in place.
- Define the action and data boundary. List the first administrative workflows, the data each can read or change, the authorization roles, and the required audit fields. Agree on retention and recovery objectives with the owner of the data.
- Build the Django foundation. Add OIDC login, Django permissions, CSRF-safe forms, pagination, and a consistent server-rendered layout. Implement the first read-only workflow before enabling mutations.
- Establish the job contract. Add
JobRun,OutboxEvent, andAuditEvent; define allowed state transitions and idempotency behavior. Build the HTMX status fragment and test it without Celery first. - Deploy the smallest production-shaped environment. Provision ECS, ALB, RDS, SQS, S3, secrets, and logs through Terraform. Use separate web and worker task roles and private networking.
- Add the worker and recovery hooks. Implement outbox dispatch, bounded tasks, retry policies, a dead-letter queue, and a dry-run reconciliation command. Confirm that duplicate messages do not duplicate effects.
- Instrument and test operational behavior. Add structured logs, traces, dashboards, alarms, and SLO measurement. Test database outage, worker restart, queue delay, and an export retry.
- Release progressively. Deploy read-only use first, then one low-risk write action, then higher-impact bulk actions. Review authorization and audit evidence with operators after each phase.
- Rehearse recovery and review cost. Restore a database backup into an isolated environment, validate the recovery notes, and compare actual service costs with the assumptions. Adjust task sizing and retention from evidence.
The key design choice is not Django versus a separate API, or HTMX versus a JavaScript framework. It is whether the tool treats staff actions as durable, authorized operations. A small Django monolith, HTMX fragments, Celery workers, PostgreSQL, and SQS are enough to do that well—provided the job lifecycle, audit trail, retries, and recovery plan are part of the design from the beginning.
This article was generated with AI. #AIGenerated