Overview
Distributed systems often require multi-service transactions that need to maintain consistency across different bounded contexts or microservices. Traditional ACID transactions are infeasible across services due to latency, autonomy, and availability concerns. Sagas offer a pragmatic alternative by breaking down a distributed transaction into a sequence of local transactions, each managing its own database and state.
Each local transaction performs its work and emits events or commands to trigger the next step. If a step fails, sagas execute compensating transactions that rollback or undo prior successful steps to restore consistency, typically in an eventually consistent manner.
This article:
- Explains the two main patterns for saga coordination: choreography and orchestration.
- Compares their trade-offs, observability, and scalability.
- Provides sequence and architectural diagrams for better understanding.
- Shows minimal and realistic code examples using widely adopted technologies.
- Details observability, monitoring, and alerting strategies for sagas.
Saga Patterns: Choreography vs Orchestration
Choreography (Event-driven)
In the choreography pattern, there is no centralized coordinator. Each service autonomously reacts to events emitted by other services and executes the local transaction or compensating action.
- Each local transaction publishes domain events when completed.
- Subscribers pick up events and execute their part of the saga, emitting further events.
- Service coupling is low because communication is asynchronous and via events.
- The saga's global state is implicit in the event log.
Pros:
- Simple architecture with few moving parts.
- High scalability and loose coupling.
- No single point of failure or bottleneck.
Cons:
- The distributed flow can become opaque and difficult to trace.
- Observability challenges for long-running processes.
- Hard to implement complex conditional logic or retries.
- Error handling requires careful event design.
flowchart LR A[Create Order] --> B[Publish OrderCreated] B --> C[Reserve Payment] C --> D[Publish PaymentReserved] D --> E[Reserve Inventory] E --> F[Publish InventoryReserved] F --> G[Confirm Order] E -->|Fail| H[Compensate Payment] C -->|Fail| I[Compensate Order]
Saga choreography sequence flow with compensations for failure
Orchestration (Central Coordinator)
Orchestration introduces a central saga orchestrator or coordinator service that controls the entire saga lifecycle.
- The orchestrator sends command messages to participating services.
- Services execute local transactions and reply with success/failure.
- The orchestrator tracks state transitions and decides the next step (including compensations).
- The saga's state is explicit in the orchestrator's persistent store.
Pros:
- Clear global state and flow control.
- Easier to implement complex flows with branching or retries.
- Better observability and monitoring.
- Simplified error handling.
Cons:
- Orchestrator is a single component responsible for correctness and availability.
- Potential to become a bottleneck or single point of failure if not designed scalable.
- Somewhat tighter coupling between services and orchestrator API.
sequenceDiagram participant Orchestrator participant OrderService participant PaymentService participant InventoryService Orchestrator->>OrderService: createOrder() OrderService-->>Orchestrator: orderCreated() Orchestrator->>PaymentService: reserveFunds() PaymentService-->>Orchestrator: fundsReserved() Orchestrator->>InventoryService: reserveStock() InventoryService-->>Orchestrator: stockReserved() Orchestrator->>OrderService: confirmOrder() OrderService-->>Orchestrator: orderConfirmed()
Sequence of interactions in saga orchestration
When to Choose Which Pattern?
| Factor | Choreography | Orchestration |
|---|---|---|
| Complexity of workflow | Simple linear flows | Complex, conditional, or branching flows |
| Observability | Hard; distributed event tracing needed | Easier; single point controlling progress |
| Coupling | Looser; event-driven | Tighter; commands & replies |
| Scalability | Highly scalable | Dependent on orchestrator scalability |
| Error handling | Complex, needs idempotent compensations | Centralized with retry and compensation logic |
| Development effort | Lower for simple flows | Higher; requires orchestrator design |
Practical Checklist for Implementing Sagas
- Design local transactions to be idempotent since retries and compensations may occur.
- Implement compensating actions that semantically revert or mitigate effects of prior local transactions.
- Persist saga state durably (either event log or orchestrator DB) to handle process restarts and manual recovery.
- Expose APIs (REST, GraphQL, or subscription) to query saga status, enabling clients to poll or receive webhook callbacks.
- Use correlation IDs in events or commands to link messages to saga instances.
- Implement timeouts and retries for each step with at-least-once delivery semantics.
- Plan for partial failure, manual intervention and reconciliation in rare cases.
- Create automated tests simulating failure paths and compensations.
Example Saga Flow: Reservation Process
A common real-world example is a product reservation flow involving an order system, payments, and inventory.
Successful Scenario
- OrderService creates an order in
PENDINGstate. - PaymentService reserves the customer funds.
- InventoryService reserves stock.
- On success, OrderService updates order status to
CONFIRMED.
Failure Scenario
If inventory reservation fails at step 3:
- PaymentService triggers compensation by releasing reserved funds.
- OrderService marks order as
FAILEDor reverts to an earlier status.
sequenceDiagram participant Order as OrderService participant Payment as PaymentService participant Inventory as InventoryService Order->>Order: createOrder(PENDING) Order->>Payment: reserveFunds() Payment-->>Order: fundsReserved() Order->>Inventory: reserveStock() Inventory--xOrder: reserveFailed() Order->>Payment: releaseFunds() -- Compensation Payment-->>Order: fundsReleased() Order->>Order: markFailed()
Failure compensation sequence in reservation saga
Code Example 1: Orchestration with Node.js & TypeScript
A minimal orchestrator example using an express app that sends commands and tracks saga state.
import express from 'express';
interface SagaState {
orderId: string;
status: 'PENDING' | 'PAYMENT_RESERVED' | 'STOCK_RESERVED' | 'CONFIRMED' | 'FAILED';
}
const sagaStates = new Map<string, SagaState>();
const app = express();
app.use(express.json());
// Endpoint to start the saga
app.post('/start-saga', (req, res) => {
const orderId = req.body.orderId;
sagaStates.set(orderId, { orderId, status: 'PENDING' });
// Kickoff saga steps asynchronously
reservePayment(orderId);
res.status(202).send(`Saga started for order ${orderId}`);
});
async function reservePayment(orderId: string) {
// Simulate async call to PaymentService
await delay(100);
sagaStates.get(orderId)!.status = 'PAYMENT_RESERVED';
reserveStock(orderId);
}
async function reserveStock(orderId: string) {
await delay(100);
// Simulate inventory failure for demo
const fail = Math.random() < 0.2;
if (fail) {
sagaStates.get(orderId)!.status = 'FAILED';
compensationReleasePayment(orderId);
} else {
sagaStates.get(orderId)!.status = 'STOCK_RESERVED';
confirmOrder(orderId);
}
}
async function compensationReleasePayment(orderId: string) {
await delay(100);
// Compensation done
console.log(`Compensated payment for order ${orderId}`);
}
async function confirmOrder(orderId: string) {
await delay(100);
sagaStates.get(orderId)!.status = 'CONFIRMED';
console.log(`Order ${orderId} confirmed`);
}
function delay(ms: number) {
return new Promise(resolve => setTimeout(resolve, ms));
}
app.listen(3000, () => console.log('Saga orchestrator running on port 3000'));
Notice how the orchestrator manages saga state and coordinates async calls with compensations.
Code Example 2: Choreography with Apache Kafka & Spring Boot
A realistic event-driven example illustrates services reacting to domain events using Kafka topics and Spring Boot.
@Service
public class OrderService {
private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
public OrderService(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void createOrder(Order order) {
// Persist local order with status PENDING
orderRepository.save(order.withStatus(OrderStatus.PENDING));
// Publish OrderCreated event
kafkaTemplate.send("order-events", new OrderCreatedEvent(order.getId()));
}
}
@Service
public class PaymentService {
@KafkaListener(topics = "order-events")
public void onOrderCreated(OrderCreatedEvent event) {
try {
reserveFunds(event.getOrderId());
kafkaTemplate.send("payment-events", new PaymentReservedEvent(event.getOrderId()));
} catch (Exception e) {
// Handle reserve failed, may emit compensation event
}
}
private void reserveFunds(String orderId) {
// Business logic to reserve funds locally
}
}
@Service
public class InventoryService {
@KafkaListener(topics = "payment-events")
public void onPaymentReserved(PaymentReservedEvent event) {
try {
reserveStock(event.getOrderId());
kafkaTemplate.send("inventory-events", new InventoryReservedEvent(event.getOrderId()));
} catch (Exception e) {
kafkaTemplate.send("payment-events", new PaymentReleaseEvent(event.getOrderId())); // compensation
}
}
private void reserveStock(String orderId) {
// Business logic to reserve inventory
}
}
Observe the loosely coupled asynchronous event listeners managing each step and emitting events for the next.
Observability & Monitoring for Sagas
Managing sagas in production demands robust observability for debugging, troubleshooting, and SLA adherence.
What to Log?
- Saga lifecycle events: Start, step success, failures, compensations.
- Correlation IDs: Attach saga IDs to all related logs and traces.
- Local transaction states: Each service logs state transitions, retries, errors.
- External system calls: Request/response logs for commands and event publishes.
- Timing and duration metrics: Time taken per step and overall saga completion.
Example structured log (JSON):
{
"timestamp": "2024-05-14T14:35:07.123Z",
"level": "INFO",
"service": "PaymentService",
"sagaId": "order-12345",
"event": "PaymentReserved",
"durationMs": 120,
"status": "success"
}
Key Metrics
Using RED (Requests, Errors, Duration) and USE (Utilization, Saturation, Errors) frameworks:
| Metric | Description | Importance |
|---|---|---|
| Saga started rate | Number of saga initiations | Load, capacity planning |
| Step success and failure rates | Per service step success/failure | Reliability tracking |
| Compensation executions | Frequency of rollbacks | Quality signals |
| End-to-end saga duration | Total time to complete transactions | User experience |
| Errors per step | Step-specific errors | Troubleshooting focal point |
| Orchestrator CPU/memory | Resource saturation if orchestrated | Bottleneck detection |
Distributed Tracing
Use OpenTelemetry to instrument saga services and orchestrators:
- Attach
traceIdas correlation context for saga instances. - Create spans for each local transaction step.
- Propagate context over messaging middleware.
- Use backends like Jaeger or Lightstep for trace visualization.
Example OpenTelemetry span creation (TypeScript):
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('saga-tracer');
async function reservePayment(orderId: string) {
await tracer.startActiveSpan('reservePayment', async (span) => {
try {
// simulate payment reservation
span.setAttribute('order.id', orderId);
// ...
} catch (err) {
span.recordException(err);
throw err;
} finally {
span.end();
}
});
}
SLI/SLO Examples
| SLI | SLO |
|---|---|
| Saga success rate | ≥ 99.9% within 24h |
| Average saga completion time | < 5 seconds for common flows |
| Compensation failure rate | < 0.01% |
| Orchestrator availability | > 99.99% |
Recommended Alerts
- High saga failure or compensation rates sustained over 5 minutes.
- Increased saga step latency indicating bottlenecks.
- Orchestrator or broker connectivity issues.
- Sudden spike in retries or dead-lettered messages.
Internal Mechanics of Saga Orchestration
Understanding the internal workings of a saga orchestrator sheds light on how it manages distributed transactions reliably.
State Machine Model
At its core, the orchestrator implements a persistent state machine representing the saga’s lifecycle:
- States reflect the cumulative progress of the saga, e.g.,
PENDING,PAYMENT_RESERVED,STOCK_RESERVED,CONFIRMED,FAILED, or compensating states. - Transitions are triggered by service responses (success or failure) and compensate steps if needed.
- The orchestrator's state machine persists all state transitions atomically, typically in a durable store like PostgreSQL or a distributed key-value store (Redis/etcd).
stateDiagram-v2 [*] --> Pending Pending --> PaymentReserved : reservePaymentSuccess() PaymentReserved --> StockReserved : reserveStockSuccess() StockReserved --> Confirmed : confirmOrderSuccess() Pending --> Failed : reservePaymentFailed() PaymentReserved --> Failed : reserveStockFailed() Failed --> CompensationPending : startCompensation() CompensationPending --> [*] : compensationComplete()
A simplified state machine lifecycle of a sample saga
Maintaining the state machine consistently is crucial as lost or out-of-order messages can corrupt the workflow. Optimistic concurrency control or event sourcing patterns are often used.
Persistence Layer
The orchestrator maintains durable storage for:
- Saga instance state: Including current state, timestamps of transitions, and metadata.
- Events log or event sourcing: To replay or recover the saga state after failure.
- Commands and compensation commands queues: Pending messages to participants with retry counts.
A typical schema table for saga metadata:
| Column | Type | Description |
|---|---|---|
| saga_id | UUID | Unique saga instance identifier |
| saga_type | VARCHAR | Type of saga (e.g., ReservationSaga) |
| current_state | VARCHAR | Current saga state |
| created_at | TIMESTAMP | Time of saga instantiation |
| updated_at | TIMESTAMP | Last state change timestamp |
| data | JSONB | Context data for saga instance |
This durable state storage enables restarting the orchestration after crashes or server restarts without losing consistency.
Message Handling and Idempotency
The orchestrator sends commands to services and awaits responses. Important internal mechanisms include:
- Message correlation: Each command or event includes a correlation ID linking it to the saga instance.
- Idempotent processing: Duplicate message deliveries require idempotent handlers on the service and orchestrator sides—state transitions or commands must be safely re-applied.
- Retries with exponential backoff: On transient failures, commands are retried before marking the step as failed.
- Dead letter queues: Persistent failures are routed for manual investigation or alternative compensation strategies.
Timeout and Compensations
Timeouts guard against indefinite waits on downstream services:
- When a step is not completed within configured timeout, the orchestrator marks the saga as failed or triggers compensations.
- Timeouts can be implemented with background workers or scheduled jobs querying sagas in intermediate states.
Compensation commands rollback prior successful actions in reverse order:
- The orchestrator tracks the execution path to know which compensations to invoke.
- Compensation failure handling is more complex: partial compensations may require manual recovery or escalation policies.
Production Scenario: Handling Partial Failures and Recovery
Here is a concrete end-to-end scenario demonstrating saga resilience in production, including failure detection and recovery.
Business Context
A travel booking system reserves a flight, hotel, and car rental. All three must succeed to confirm the booking, or all must be compensated.
Step-by-step Flow with Failures
- The orchestrator starts the saga, sets booking state to
STARTED. - Sends
reserveFlight()command; receives success; updates state toFLIGHT_RESERVED. - Sends
reserveHotel()command; receives success; updates state toHOTEL_RESERVED. - Sends
reserveCar()command; due to external downtime, the call times out (no acknowledgment within 20 seconds). - Orchestrator marks saga as
FAILED_RESERVATION, sends compensation commands:cancelHotel()cancelFlight()
- Hotel and flight services acknowledge compensation completions.
- Saga transitions to
COMPENSATION_DONEand reports failure to client.
Implementation Considerations
- Timeout Detection: Use a scheduler or cron job querying sagas stuck in intermediate states past timeouts.
- Manual Intervention: If compensation commands fail repeatedly, alerts prompt manual reconciliation.
- Idempotency: All reserve/cancel commands on services handle retries and duplicate requests gracefully.
- Visibility: The orchestrator exposes API endpoints to inspect saga states and history.
- Error Classification: Distinguish between transient errors (retryable) and permanent errors (fail fast).
sequenceDiagram participant Orchestrator participant FlightService participant HotelService participant CarService Orchestrator->>FlightService: reserveFlight() FlightService-->>Orchestrator: flightReserved() Orchestrator->>HotelService: reserveHotel() HotelService-->>Orchestrator: hotelReserved() Orchestrator->>CarService: reserveCar() Note right of CarService: Times out waiting for confirmation Orchestrator->>HotelService: cancelHotel() -- compensation HotelService-->>Orchestrator: hotelCancelled() Orchestrator->>FlightService: cancelFlight() -- compensation FlightService-->>Orchestrator: flightCancelled() Orchestrator->>Orchestrator: markFailed()
Failed travel booking saga with timeouts and compensations
What if Compensation Fails?
- The orchestrator can escalate to a dead letter queue for manual review.
- Alerting systems notify support teams for intervention.
- Compensation steps can be retried with backoff policies.
- Logging and trace metadata are crucial to diagnose partial failure root causes.
Observability Deep Dive: Logging, Metrics, and Tracing Patterns
Robust observability is paramount for sagas, which are inherently asynchronous and long-lived.
Structured Logging Best Practices
- Include correlation IDs and saga instance IDs in every log line.
- Log step start, step success, step failure, compensation start, and compensation outcome events.
- Log contextual metadata like input parameters, error stack traces, elapsed time.
- Use structured JSON logs for easy parsing and aggregation.
Example log snippet:
{
"timestamp": "2024-06-10T11:30:45.678Z",
"service": "OrderService",
"saga_id": "123e4567-e89b-12d3-a456-426614174000",
"trace_id": "abcd1234efgh5678",
"event": "ReservePaymentStarted",
"orderId": "order-789",
"userId": "user-555",
"step": "reservePayment",
"status": "started"
}
Metrics Collection
Track metrics at multiple granularity levels:
| Metric | Description | Type |
|---|---|---|
saga_started_total | Count of saga instances started | Counter |
saga_completed_total | Count of successful saga completions | Counter |
saga_failed_total | Count of saga failures | Counter |
step_duration_seconds | Latency histogram of individual saga steps | Histogram |
compensation_executed_total | Count of compensation actions executed | Counter |
retry_attempts_total | Number of retries per step | Counter |
orchestrator_cpu_usage | CPU usage percentage for orchestrator | Gauge |
orchestrator_memory_usage | Memory usage for orchestrator | Gauge |
Use Prometheus or a similar time series database for scraping and alerting on these metrics.
Distributed Tracing Patterns
Distributed tracing traces the flow across services and the orchestrator:
- Assign a trace-id and propagate it in commands/events.
- Create a span per saga step, labeled clearly with operation name and service name.
- Span tags add context: saga id, step name, outcome, error details.
- Use baggage items for additional context propagation if needed.
Example OpenTelemetry usage in a Java Spring Boot orchestrator:
@Autowired
Tracer tracer;
public void reserveFlight(String sagaId, String flightId) {
Span span = tracer.spanBuilder("reserveFlight").startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute("saga.id", sagaId);
span.setAttribute("flight.id", flightId);
// Call FlightService API
flightClient.reserve(flightId);
span.setStatus(StatusCode.OK);
} catch (Exception ex) {
span.recordException(ex);
span.setStatus(StatusCode.ERROR, "Failed to reserve flight");
throw ex;
} finally {
span.end();
}
}
These traces surface to tools like Jaeger or Zipkin and provide root cause analysis and latency breakdowns.
Decision Checklist for Choosing Sagas and Patterns
Before implementing saga patterns, consider this checklist to align trade-offs with your system requirements:
| Decision Criteria | Key Considerations | Preferred Saga Pattern |
|---|---|---|
| Workflow complexity | Simple linear vs complex branching and retries | Choreography (linear) or Orchestration (complex) |
| Strong consistency requirement | Need for immediate consistency vs eventual consistency | Orchestration for closer control |
| Operational complexity | Need tight central visibility and control | Orchestration is easier to monitor and debug |
| Service coupling tolerance | Desire to keep services loosely coupled and autonomous | Choreography promotes loose coupling |
| Scalability demands | Handling thousands of concurrent sagas | Choreography scales well due to decentralization |
| Failure recovery and compensation | Need for centralized failure handling and compensation management | Orchestration simplifies failure recovery |
| Latency Sensitivity | Timeouts and retries must be tuned precisely | Orchestration allows explicit timeout controls |
| Testing and Dev resources | Available dev time and ability to implement complex patterns | Choreography has a quicker initial dev cycle |
These deep dives into saga orchestration internals, recoveries in production scenarios, observability patterns, and decision checklists aim to empower engineers designing resilient distributed transactions with confidence and precision.
Summary
Sagas are an indispensable pattern for achieving distributed transaction consistency in microservice architectures without locking resources globally.
- Choreography is best for simple event-driven workflows prioritizing loose coupling and scalability.
- Orchestration suits complex flows requiring explicit control, retry, and observability.
- Idempotency, compensation logic, and durable state tracking are foundational best practices.
- Observability using structured logs, tracing, and metrics is critical for operating sagas reliably in production.
Use the flow diagrams, example implementations, and observability guidelines here as practical references when building your sagas.