Skip to content

Sagas & Long‑Running Workflows: Patterns for Distributed Transactions

Sagas provide a pragmatic way to manage multi‑service transactions using local transactions plus compensations. This article compares choreography and orchestration and gives examples.

Ekanatha Reddy Urivakili12 min2,229 words

Overview

Distributed systems often require multi-service transactions that need to maintain consistency across different bounded contexts or microservices. Traditional ACID transactions are infeasible across services due to latency, autonomy, and availability concerns. Sagas offer a pragmatic alternative by breaking down a distributed transaction into a sequence of local transactions, each managing its own database and state.

Each local transaction performs its work and emits events or commands to trigger the next step. If a step fails, sagas execute compensating transactions that rollback or undo prior successful steps to restore consistency, typically in an eventually consistent manner.

This article:

  • Explains the two main patterns for saga coordination: choreography and orchestration.
  • Compares their trade-offs, observability, and scalability.
  • Provides sequence and architectural diagrams for better understanding.
  • Shows minimal and realistic code examples using widely adopted technologies.
  • Details observability, monitoring, and alerting strategies for sagas.

Saga Patterns: Choreography vs Orchestration

Choreography (Event-driven)

In the choreography pattern, there is no centralized coordinator. Each service autonomously reacts to events emitted by other services and executes the local transaction or compensating action.

  • Each local transaction publishes domain events when completed.
  • Subscribers pick up events and execute their part of the saga, emitting further events.
  • Service coupling is low because communication is asynchronous and via events.
  • The saga's global state is implicit in the event log.

Pros:

  • Simple architecture with few moving parts.
  • High scalability and loose coupling.
  • No single point of failure or bottleneck.

Cons:

  • The distributed flow can become opaque and difficult to trace.
  • Observability challenges for long-running processes.
  • Hard to implement complex conditional logic or retries.
  • Error handling requires careful event design.
flowchart LR
  A[Create Order] --> B[Publish OrderCreated]
  B --> C[Reserve Payment]
  C --> D[Publish PaymentReserved]
  D --> E[Reserve Inventory]
  E --> F[Publish InventoryReserved]
  F --> G[Confirm Order]
  E -->|Fail| H[Compensate Payment]
  C -->|Fail| I[Compensate Order]

Saga choreography sequence flow with compensations for failure

Orchestration (Central Coordinator)

Orchestration introduces a central saga orchestrator or coordinator service that controls the entire saga lifecycle.

  • The orchestrator sends command messages to participating services.
  • Services execute local transactions and reply with success/failure.
  • The orchestrator tracks state transitions and decides the next step (including compensations).
  • The saga's state is explicit in the orchestrator's persistent store.

Pros:

  • Clear global state and flow control.
  • Easier to implement complex flows with branching or retries.
  • Better observability and monitoring.
  • Simplified error handling.

Cons:

  • Orchestrator is a single component responsible for correctness and availability.
  • Potential to become a bottleneck or single point of failure if not designed scalable.
  • Somewhat tighter coupling between services and orchestrator API.
sequenceDiagram
  participant Orchestrator
  participant OrderService
  participant PaymentService
  participant InventoryService

  Orchestrator->>OrderService: createOrder()
  OrderService-->>Orchestrator: orderCreated()
  Orchestrator->>PaymentService: reserveFunds()
  PaymentService-->>Orchestrator: fundsReserved()
  Orchestrator->>InventoryService: reserveStock()
  InventoryService-->>Orchestrator: stockReserved()
  Orchestrator->>OrderService: confirmOrder()
  OrderService-->>Orchestrator: orderConfirmed()

Sequence of interactions in saga orchestration


When to Choose Which Pattern?

FactorChoreographyOrchestration
Complexity of workflowSimple linear flowsComplex, conditional, or branching flows
ObservabilityHard; distributed event tracing neededEasier; single point controlling progress
CouplingLooser; event-drivenTighter; commands & replies
ScalabilityHighly scalableDependent on orchestrator scalability
Error handlingComplex, needs idempotent compensationsCentralized with retry and compensation logic
Development effortLower for simple flowsHigher; requires orchestrator design

Practical Checklist for Implementing Sagas

  • Design local transactions to be idempotent since retries and compensations may occur.
  • Implement compensating actions that semantically revert or mitigate effects of prior local transactions.
  • Persist saga state durably (either event log or orchestrator DB) to handle process restarts and manual recovery.
  • Expose APIs (REST, GraphQL, or subscription) to query saga status, enabling clients to poll or receive webhook callbacks.
  • Use correlation IDs in events or commands to link messages to saga instances.
  • Implement timeouts and retries for each step with at-least-once delivery semantics.
  • Plan for partial failure, manual intervention and reconciliation in rare cases.
  • Create automated tests simulating failure paths and compensations.

Example Saga Flow: Reservation Process

A common real-world example is a product reservation flow involving an order system, payments, and inventory.

Successful Scenario

  1. OrderService creates an order in PENDING state.
  2. PaymentService reserves the customer funds.
  3. InventoryService reserves stock.
  4. On success, OrderService updates order status to CONFIRMED.

Failure Scenario

If inventory reservation fails at step 3:

  • PaymentService triggers compensation by releasing reserved funds.
  • OrderService marks order as FAILED or reverts to an earlier status.
sequenceDiagram
  participant Order as OrderService
  participant Payment as PaymentService
  participant Inventory as InventoryService

  Order->>Order: createOrder(PENDING)
  Order->>Payment: reserveFunds()
  Payment-->>Order: fundsReserved()
  Order->>Inventory: reserveStock()
  Inventory--xOrder: reserveFailed()
  Order->>Payment: releaseFunds()   -- Compensation
  Payment-->>Order: fundsReleased()
  Order->>Order: markFailed()

Failure compensation sequence in reservation saga


Code Example 1: Orchestration with Node.js & TypeScript

A minimal orchestrator example using an express app that sends commands and tracks saga state.

import express from 'express';

interface SagaState {
  orderId: string;
  status: 'PENDING' | 'PAYMENT_RESERVED' | 'STOCK_RESERVED' | 'CONFIRMED' | 'FAILED';
}

const sagaStates = new Map<string, SagaState>();
const app = express();
app.use(express.json());

// Endpoint to start the saga
app.post('/start-saga', (req, res) => {
  const orderId = req.body.orderId;
  sagaStates.set(orderId, { orderId, status: 'PENDING' });
  // Kickoff saga steps asynchronously
  reservePayment(orderId);
  res.status(202).send(`Saga started for order ${orderId}`);
});

async function reservePayment(orderId: string) {
  // Simulate async call to PaymentService
  await delay(100);
  sagaStates.get(orderId)!.status = 'PAYMENT_RESERVED';
  reserveStock(orderId);
}

async function reserveStock(orderId: string) {
  await delay(100);
  // Simulate inventory failure for demo
  const fail = Math.random() < 0.2;
  if (fail) {
    sagaStates.get(orderId)!.status = 'FAILED';
    compensationReleasePayment(orderId);
  } else {
    sagaStates.get(orderId)!.status = 'STOCK_RESERVED';
    confirmOrder(orderId);
  }
}

async function compensationReleasePayment(orderId: string) {
  await delay(100);
  // Compensation done
  console.log(`Compensated payment for order ${orderId}`);
}

async function confirmOrder(orderId: string) {
  await delay(100);
  sagaStates.get(orderId)!.status = 'CONFIRMED';
  console.log(`Order ${orderId} confirmed`);
}

function delay(ms: number) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

app.listen(3000, () => console.log('Saga orchestrator running on port 3000'));

Notice how the orchestrator manages saga state and coordinates async calls with compensations.


Code Example 2: Choreography with Apache Kafka & Spring Boot

A realistic event-driven example illustrates services reacting to domain events using Kafka topics and Spring Boot.

@Service
public class OrderService {

  private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

  public OrderService(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
    this.kafkaTemplate = kafkaTemplate;
  }

  public void createOrder(Order order) {
    // Persist local order with status PENDING
    orderRepository.save(order.withStatus(OrderStatus.PENDING));
    // Publish OrderCreated event
    kafkaTemplate.send("order-events", new OrderCreatedEvent(order.getId()));
  }
}

@Service
public class PaymentService {

  @KafkaListener(topics = "order-events")  
  public void onOrderCreated(OrderCreatedEvent event) {
    try {
      reserveFunds(event.getOrderId());
      kafkaTemplate.send("payment-events", new PaymentReservedEvent(event.getOrderId()));
    } catch (Exception e) {
      // Handle reserve failed, may emit compensation event
    }
  }

  private void reserveFunds(String orderId) {
    // Business logic to reserve funds locally
  }
} 

@Service
public class InventoryService {

  @KafkaListener(topics = "payment-events")
  public void onPaymentReserved(PaymentReservedEvent event) {
    try {
      reserveStock(event.getOrderId());
      kafkaTemplate.send("inventory-events", new InventoryReservedEvent(event.getOrderId()));
    } catch (Exception e) {
      kafkaTemplate.send("payment-events", new PaymentReleaseEvent(event.getOrderId())); // compensation
    }
  }

  private void reserveStock(String orderId) {
    // Business logic to reserve inventory
  }
}

Observe the loosely coupled asynchronous event listeners managing each step and emitting events for the next.


Observability & Monitoring for Sagas

Managing sagas in production demands robust observability for debugging, troubleshooting, and SLA adherence.

What to Log?

  • Saga lifecycle events: Start, step success, failures, compensations.
  • Correlation IDs: Attach saga IDs to all related logs and traces.
  • Local transaction states: Each service logs state transitions, retries, errors.
  • External system calls: Request/response logs for commands and event publishes.
  • Timing and duration metrics: Time taken per step and overall saga completion.

Example structured log (JSON):

{
  "timestamp": "2024-05-14T14:35:07.123Z",
  "level": "INFO",
  "service": "PaymentService",
  "sagaId": "order-12345",
  "event": "PaymentReserved",
  "durationMs": 120,
  "status": "success"
}

Key Metrics

Using RED (Requests, Errors, Duration) and USE (Utilization, Saturation, Errors) frameworks:

MetricDescriptionImportance
Saga started rateNumber of saga initiationsLoad, capacity planning
Step success and failure ratesPer service step success/failureReliability tracking
Compensation executionsFrequency of rollbacksQuality signals
End-to-end saga durationTotal time to complete transactionsUser experience
Errors per stepStep-specific errorsTroubleshooting focal point
Orchestrator CPU/memoryResource saturation if orchestratedBottleneck detection

Distributed Tracing

Use OpenTelemetry to instrument saga services and orchestrators:

  • Attach traceId as correlation context for saga instances.
  • Create spans for each local transaction step.
  • Propagate context over messaging middleware.
  • Use backends like Jaeger or Lightstep for trace visualization.

Example OpenTelemetry span creation (TypeScript):

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('saga-tracer');

async function reservePayment(orderId: string) {
  await tracer.startActiveSpan('reservePayment', async (span) => {
    try {
      // simulate payment reservation
      span.setAttribute('order.id', orderId);
      // ...
    } catch (err) {
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

SLI/SLO Examples

SLISLO
Saga success rate≥ 99.9% within 24h
Average saga completion time< 5 seconds for common flows
Compensation failure rate< 0.01%
Orchestrator availability> 99.99%
  • High saga failure or compensation rates sustained over 5 minutes.
  • Increased saga step latency indicating bottlenecks.
  • Orchestrator or broker connectivity issues.
  • Sudden spike in retries or dead-lettered messages.

Internal Mechanics of Saga Orchestration

Understanding the internal workings of a saga orchestrator sheds light on how it manages distributed transactions reliably.

State Machine Model

At its core, the orchestrator implements a persistent state machine representing the saga’s lifecycle:

  • States reflect the cumulative progress of the saga, e.g., PENDING, PAYMENT_RESERVED, STOCK_RESERVED, CONFIRMED, FAILED, or compensating states.
  • Transitions are triggered by service responses (success or failure) and compensate steps if needed.
  • The orchestrator's state machine persists all state transitions atomically, typically in a durable store like PostgreSQL or a distributed key-value store (Redis/etcd).
stateDiagram-v2
  [*] --> Pending
  Pending --> PaymentReserved : reservePaymentSuccess()
  PaymentReserved --> StockReserved : reserveStockSuccess()
  StockReserved --> Confirmed : confirmOrderSuccess()
  Pending --> Failed : reservePaymentFailed()
  PaymentReserved --> Failed : reserveStockFailed()
  Failed --> CompensationPending : startCompensation()
  CompensationPending --> [*] : compensationComplete()

A simplified state machine lifecycle of a sample saga

Maintaining the state machine consistently is crucial as lost or out-of-order messages can corrupt the workflow. Optimistic concurrency control or event sourcing patterns are often used.

Persistence Layer

The orchestrator maintains durable storage for:

  • Saga instance state: Including current state, timestamps of transitions, and metadata.
  • Events log or event sourcing: To replay or recover the saga state after failure.
  • Commands and compensation commands queues: Pending messages to participants with retry counts.

A typical schema table for saga metadata:

ColumnTypeDescription
saga_idUUIDUnique saga instance identifier
saga_typeVARCHARType of saga (e.g., ReservationSaga)
current_stateVARCHARCurrent saga state
created_atTIMESTAMPTime of saga instantiation
updated_atTIMESTAMPLast state change timestamp
dataJSONBContext data for saga instance

This durable state storage enables restarting the orchestration after crashes or server restarts without losing consistency.

Message Handling and Idempotency

The orchestrator sends commands to services and awaits responses. Important internal mechanisms include:

  • Message correlation: Each command or event includes a correlation ID linking it to the saga instance.
  • Idempotent processing: Duplicate message deliveries require idempotent handlers on the service and orchestrator sides—state transitions or commands must be safely re-applied.
  • Retries with exponential backoff: On transient failures, commands are retried before marking the step as failed.
  • Dead letter queues: Persistent failures are routed for manual investigation or alternative compensation strategies.

Timeout and Compensations

Timeouts guard against indefinite waits on downstream services:

  • When a step is not completed within configured timeout, the orchestrator marks the saga as failed or triggers compensations.
  • Timeouts can be implemented with background workers or scheduled jobs querying sagas in intermediate states.

Compensation commands rollback prior successful actions in reverse order:

  • The orchestrator tracks the execution path to know which compensations to invoke.
  • Compensation failure handling is more complex: partial compensations may require manual recovery or escalation policies.

Production Scenario: Handling Partial Failures and Recovery

Here is a concrete end-to-end scenario demonstrating saga resilience in production, including failure detection and recovery.

Business Context

A travel booking system reserves a flight, hotel, and car rental. All three must succeed to confirm the booking, or all must be compensated.

Step-by-step Flow with Failures

  1. The orchestrator starts the saga, sets booking state to STARTED.
  2. Sends reserveFlight() command; receives success; updates state to FLIGHT_RESERVED.
  3. Sends reserveHotel() command; receives success; updates state to HOTEL_RESERVED.
  4. Sends reserveCar() command; due to external downtime, the call times out (no acknowledgment within 20 seconds).
  5. Orchestrator marks saga as FAILED_RESERVATION, sends compensation commands:
    • cancelHotel()
    • cancelFlight()
  6. Hotel and flight services acknowledge compensation completions.
  7. Saga transitions to COMPENSATION_DONE and reports failure to client.

Implementation Considerations

  • Timeout Detection: Use a scheduler or cron job querying sagas stuck in intermediate states past timeouts.
  • Manual Intervention: If compensation commands fail repeatedly, alerts prompt manual reconciliation.
  • Idempotency: All reserve/cancel commands on services handle retries and duplicate requests gracefully.
  • Visibility: The orchestrator exposes API endpoints to inspect saga states and history.
  • Error Classification: Distinguish between transient errors (retryable) and permanent errors (fail fast).
sequenceDiagram
  participant Orchestrator
  participant FlightService
  participant HotelService
  participant CarService

  Orchestrator->>FlightService: reserveFlight()
  FlightService-->>Orchestrator: flightReserved()
  Orchestrator->>HotelService: reserveHotel()
  HotelService-->>Orchestrator: hotelReserved()
  Orchestrator->>CarService: reserveCar()
  Note right of CarService: Times out waiting for confirmation
  Orchestrator->>HotelService: cancelHotel()  -- compensation
  HotelService-->>Orchestrator: hotelCancelled()
  Orchestrator->>FlightService: cancelFlight() -- compensation
  FlightService-->>Orchestrator: flightCancelled()
  Orchestrator->>Orchestrator: markFailed()

Failed travel booking saga with timeouts and compensations

What if Compensation Fails?

  • The orchestrator can escalate to a dead letter queue for manual review.
  • Alerting systems notify support teams for intervention.
  • Compensation steps can be retried with backoff policies.
  • Logging and trace metadata are crucial to diagnose partial failure root causes.

Observability Deep Dive: Logging, Metrics, and Tracing Patterns

Robust observability is paramount for sagas, which are inherently asynchronous and long-lived.

Structured Logging Best Practices

  • Include correlation IDs and saga instance IDs in every log line.
  • Log step start, step success, step failure, compensation start, and compensation outcome events.
  • Log contextual metadata like input parameters, error stack traces, elapsed time.
  • Use structured JSON logs for easy parsing and aggregation.

Example log snippet:

{
  "timestamp": "2024-06-10T11:30:45.678Z",
  "service": "OrderService",
  "saga_id": "123e4567-e89b-12d3-a456-426614174000",
  "trace_id": "abcd1234efgh5678",
  "event": "ReservePaymentStarted",
  "orderId": "order-789",
  "userId": "user-555",
  "step": "reservePayment",
  "status": "started"
}

Metrics Collection

Track metrics at multiple granularity levels:

MetricDescriptionType
saga_started_totalCount of saga instances startedCounter
saga_completed_totalCount of successful saga completionsCounter
saga_failed_totalCount of saga failuresCounter
step_duration_secondsLatency histogram of individual saga stepsHistogram
compensation_executed_totalCount of compensation actions executedCounter
retry_attempts_totalNumber of retries per stepCounter
orchestrator_cpu_usageCPU usage percentage for orchestratorGauge
orchestrator_memory_usageMemory usage for orchestratorGauge

Use Prometheus or a similar time series database for scraping and alerting on these metrics.

Distributed Tracing Patterns

Distributed tracing traces the flow across services and the orchestrator:

  • Assign a trace-id and propagate it in commands/events.
  • Create a span per saga step, labeled clearly with operation name and service name.
  • Span tags add context: saga id, step name, outcome, error details.
  • Use baggage items for additional context propagation if needed.

Example OpenTelemetry usage in a Java Spring Boot orchestrator:

@Autowired
Tracer tracer;

public void reserveFlight(String sagaId, String flightId) {
  Span span = tracer.spanBuilder("reserveFlight").startSpan();
  try (Scope scope = span.makeCurrent()) {
    span.setAttribute("saga.id", sagaId);
    span.setAttribute("flight.id", flightId);
    // Call FlightService API
    flightClient.reserve(flightId);
    span.setStatus(StatusCode.OK);
  } catch (Exception ex) {
    span.recordException(ex);
    span.setStatus(StatusCode.ERROR, "Failed to reserve flight");
    throw ex;
  } finally {
    span.end();
  }
}

These traces surface to tools like Jaeger or Zipkin and provide root cause analysis and latency breakdowns.


Decision Checklist for Choosing Sagas and Patterns

Before implementing saga patterns, consider this checklist to align trade-offs with your system requirements:

Decision CriteriaKey ConsiderationsPreferred Saga Pattern
Workflow complexitySimple linear vs complex branching and retriesChoreography (linear) or Orchestration (complex)
Strong consistency requirementNeed for immediate consistency vs eventual consistencyOrchestration for closer control
Operational complexityNeed tight central visibility and controlOrchestration is easier to monitor and debug
Service coupling toleranceDesire to keep services loosely coupled and autonomousChoreography promotes loose coupling
Scalability demandsHandling thousands of concurrent sagasChoreography scales well due to decentralization
Failure recovery and compensationNeed for centralized failure handling and compensation managementOrchestration simplifies failure recovery
Latency SensitivityTimeouts and retries must be tuned preciselyOrchestration allows explicit timeout controls
Testing and Dev resourcesAvailable dev time and ability to implement complex patternsChoreography has a quicker initial dev cycle

These deep dives into saga orchestration internals, recoveries in production scenarios, observability patterns, and decision checklists aim to empower engineers designing resilient distributed transactions with confidence and precision.

Summary

Sagas are an indispensable pattern for achieving distributed transaction consistency in microservice architectures without locking resources globally.

  • Choreography is best for simple event-driven workflows prioritizing loose coupling and scalability.
  • Orchestration suits complex flows requiring explicit control, retry, and observability.
  • Idempotency, compensation logic, and durable state tracking are foundational best practices.
  • Observability using structured logs, tracing, and metrics is critical for operating sagas reliably in production.

Use the flow diagrams, example implementations, and observability guidelines here as practical references when building your sagas.


References