Skip to content

Event Sourcing: Design Patterns, Pitfalls, and Rebuild Strategies

Event sourcing records every change as an immutable event. This article explains modeling choices, snapshots, upcasters, and safe external integrations.

Ekanatha Reddy Urivakili8 min1,236 words

Why event sourcing?

Event sourcing is a powerful architectural pattern that records every state change in an application as an immutable, chronological sequence of events — the single source of truth. Rather than storing just the current state of entities, event sourcing stores all the domain events that led to that state.

The benefits include:

  • Complete audit trail: Every change is timestamped and stored immutably, resulting in a full historical log.
  • Time travel and reproducibility: Rebuild state at any point in time by replaying events.
  • Complex domain modeling: Naturally models domain events with rich semantic meaning.
  • Temporal queries: Enables queries like “What was the state at time T?” or “How many active orders were there last month?”
  • Decoupled read models: Read models can be asynchronously derived, optimized for queries without impacting writes.

Event sourcing particularly shines in domains demanding strong auditability (e.g., financial ledgers, compliance systems) and in distributed or microservices architectures where event-driven workflows provide resilience and decoupling.


High-Level Architecture of Event Sourcing Systems

flowchart LR
  A[User Command] --> B[Command Handler]
  B --> C[Validate & Enrich Command]
  C --> D[Generate Domain Events]
  D --> E[Append Events to Event Store]
  E --> F[Publish Events to Event Bus]
  F --> G[Update Read Models / Projections]
  F --> H[Trigger External Side Effects]
  G --> I[Queries]

Figure 1: High-Level Flow of Command Handling, Event Storage, and Projection in Event Sourcing

Commands from users or upstream systems are validated and transformed into domain events stored immutably in the event store. Published events then update read models optimized for querying and may trigger side effects such as notifying external services.


Key Modeling Decisions

Event sourcing design is nuanced and requires careful modeling decisions. The following sections cover key trade-offs.

Event Shape

Events must be self-contained — they should encapsulate all the data needed to:

  • Rebuild or replay the event.
  • Compensate or reverse it if required.
  • Perform projections without looking up additional context from outside the event store.

Avoid storing references to external mutable state within events, as that breaks the replay-determinism principle.

ApproachProsCons
Full data eventSimple replay, idempotent projectionsHigher storage footprint
Event with referencesSmaller sizeExternal dependencies complicate replay

Use JSON Schema or Avro to version and validate event data consistently.

Aggregates and Event Granularity

Aggregate design impacts replay efficiency and system complexity.

  • Minimize aggregate size: small aggregates mean smaller event streams to replay for state rebuild.
  • Design domain events around aggregate boundaries.
  • Event streams must be ordered per aggregate for transactional consistency.

Large aggregates with long event histories can slow rebuilds.

Snapshots

Replaying an entire event stream from the first event can grow costly as aggregate lifetime and event count increases. Snapshots capture aggregate state at a point in time, allowing faster rebuilds by:

  • Loading the snapshot state.
  • Replaying only events after snapshot version.
Snapshot FrequencyBenefitsTrade-offs
Frequent (every N events)Fast rebuild, less replay lagIncreased storage, snapshot overhead
Rare (e.g. daily)Storage efficientLonger replay lag
NoneSimplestRebuild times degrade over time

Design snapshotting strategy based on expected event rates, aggregate lifetime, and rebuild frequency requirements.


Upcasters & Versioning: Evolving Your Event Schema Safely

Event schemas inevitably evolve — adding fields, renaming properties, or changing event types — while maintaining immutability is crucial for auditing and compliance.

The best strategy: do not rewrite stored events. Instead implement an upcasting layer that:

  • Transforms older event JSON into the current schema upon reading.
  • Keeps the event store immutable and consistent.

Upcasters are typically implemented in your event deserialization layer or projection pipeline.

[
  {
    "oldSchemaVersion": 1,
    "transform": "Add 'customerId' field defaulting to null"
  },
  {
    "oldSchemaVersion": 2,
    "transform": "Rename 'totalAmount' to 'amountIncludingTax'"
  }
]

Maintain exhaustive upcaster tests to ensure correct replay behaviour and validate migrations.


Handling External Integrations and Side Effects

One of the trickier aspects of event sourcing is interacting with external systems during event replay and live handling.

Side Effects During Replay

If event handlers trigger side effects — such as calling external APIs or sending emails — replays can cause undesired duplication or mutations outside the event store.

There are two common patterns to address this:

  • Side effect suppression during replay: Introduce a flag in the event handler or gateway layer to suppress external calls when reprocessing events.
  • Persist responses as events: Save external integration results as domain events (query-events) allowing deterministic replays including external data without actual network calls.

Gateway Wrapping Example (TypeScript)

class ApiGateway {
  private isReplayMode: boolean;

  constructor() {
    this.isReplayMode = false;
  }

  setReplayMode(flag: boolean) {
    this.isReplayMode = flag;
  }

  async callExternalApi(params: any): Promise<any> {
    if (this.isReplayMode) {
      // Return cached or stub data during replay
      return this.getCachedResponse(params);
    }
    // Actual API call
    return fetch('https://example.com/api', { body: JSON.stringify(params) })
      .then(res => res.json());
  }

  private getCachedResponse(params: any) {
    // Retrieve cached response saved as an event
    // Implementation depends on your event store or cache
  }
}

Operational Playbook and Best Practices

Robust event sourcing requires operational rigor.

Backups

  • The event store is the single source of truth — protect it with regular backups.
  • Automate backups and verify restore procedures frequently.

Replaying

  • Build tooling for replaying events into staging environments for debugging, testing migrations, and verifying data integrity.
  • Replay tools should allow selecting aggregates, time windows, and filtering event types.

Testing & Documentation

  • Test all upcasters extensively with backward compatibility tests.
  • Document snapshot policies and strategies clearly.
  • Monitor event store size and replay durations proactively.

Event Sourcing vs. Traditional CRUD: Trade-Off Table

CriterionEvent SourcingCRUD
State StorageAppend-only event logLatest state in DB
Audit TrailFull, tamper-proofMust be built separately
Query ComplexityRequires projections/read modelsDirect queries
Schema EvolutionHandle via upcasters/adaptersSchema migrations
State ReconstructionReplay events, possibly with snapshotsRead latest state
DebuggingReplay events, time travelView logs or snapshots
Integration ComplexityManaging side effects carefullyUsually simpler
Rebuild PerformanceCan be slow without snapshotsFast

Examples

Minimal Event Sourcing Usage with TypeScript

interface Event {
  type: string;
  timestamp: number;
  data: any;
}

class EventStore {
  private events: Event[] = [];

  append(event: Event) {
    this.events.push(event);
  }

  getEvents() {
    return [...this.events];
  }
}

class AccountAggregate {
  private balance = 0;

  apply(event: Event) {
    switch (event.type) {
      case 'Deposit':
        this.balance += event.data.amount;
        break;
      case 'Withdrawal':
        this.balance -= event.data.amount;
        break;
    }
  }

  getBalance() {
    return this.balance;
  }
}

// Usage
const store = new EventStore();
store.append({ type: 'Deposit', timestamp: Date.now(), data: { amount: 100 }});
store.append({ type: 'Withdrawal', timestamp: Date.now(), data: { amount: 40 }});

const account = new AccountAggregate();
for (const ev of store.getEvents()) {
  account.apply(ev);
}

console.log('Balance:', account.getBalance());

Notice how the aggregate’s state is reconstructed by replaying events.


Realistic Event Sourcing with Snapshots and Upcasting (Node.js + TypeScript)

interface Event {
  id: string;
  type: string;
  timestamp: number;
  version: number; // schema version
  data: any;
}

interface Snapshot {
  aggregateId: string;
  lastEventId: string;
  state: any;
  version: number; // schema version
}

class EventStore {
  private events: Event[] = [];
  private snapshots: Map<string, Snapshot> = new Map();

  append(event: Event) {
    this.events.push(event);
  }

  getEventsSince(aggregateId: string, lastEventId?: string) {
    const startIndex = lastEventId
      ? this.events.findIndex(e => e.id === lastEventId) + 1
      : 0;
    return this.events.filter(e => e.id.startsWith(aggregateId)).slice(startIndex);
  }

  saveSnapshot(snapshot: Snapshot) {
    this.snapshots.set(snapshot.aggregateId, snapshot);
  }

  getSnapshot(aggregateId: string): Snapshot | undefined {
    return this.snapshots.get(aggregateId);
  }
}

class AccountAggregate {
  balance = 0;
  lastEventId = '';
  version = 1;

  apply(event: Event) {
    // Upcaster example - transform old event format to current
    const upcastedEvent = this.upcast(event);

    switch (upcastedEvent.type) {
      case 'Deposit':
        this.balance += upcastedEvent.data.amount;
        break;
      case 'Withdrawal':
        this.balance -= upcastedEvent.data.amount;
        break;
    }
    this.lastEventId = upcastedEvent.id;
    this.version = upcastedEvent.version;
  }

  upcast(event: Event): Event {
    // Example: schema version 1 added 'amount' field in data
    if (event.version === 0) {
      // Convert old event to version 1 format
      return {
        ...event,
        version: 1,
        data: { amount: event.data.value } // renamed field
      };
    }
    return event;
  }
}

// Replay using snapshot
function rebuildAggregate(store: EventStore, aggregateId: string): AccountAggregate {
  const snapshot = store.getSnapshot(aggregateId);
  const aggregate = new AccountAggregate();
  if (snapshot) {
    aggregate.balance = snapshot.state.balance;
  }
  const events = store.getEventsSince(aggregateId, snapshot?.lastEventId);
  for (const ev of events) {
    aggregate.apply(ev);
  }
  return aggregate;
}

// Usage
const store = new EventStore();
const aggregateId = 'account-123';

store.append({ id: `${aggregateId}-1`, type: 'Deposit', timestamp: Date.now(), version: 1, data: { amount: 100 }});
store.append({ id: `${aggregateId}-2`, type: 'Withdrawal', timestamp: Date.now(), version: 1, data: { amount: 35 }});

store.saveSnapshot({ aggregateId, lastEventId: `${aggregateId}-2`, state: { balance: 65 }, version: 1 });

const account = rebuildAggregate(store, aggregateId);
console.log('Rebuilt balance:', account.balance);

Highlights: snapshot usage prevents replay of all events, schema versioning through upcasting.


Observability in Event Sourced Systems

Effective monitoring, logging, and tracing are critical in complex event-driven architectures.

What to Log

  • Command received: command type, aggregate id, timestamp.
  • Event appended: event type, aggregate id, event id, timestamp.
  • Snapshot created: aggregate id, event id, snapshot timestamp.
  • Errors: failures during event handling, projection update, or upcasting.
  • Replays: when replaying events, log replay start/end, aggregate id, event ranges.
  • External calls: suppress vs. actual external API interactions with outcome.

Example structured log (JSON):

{
  "timestamp": "2024-04-27T15:30:45Z",
  "level": "info",
  "event": "event_appended",
  "aggregateId": "account-123",
  "eventId": "account-123-27",
  "eventType": "Deposit",
  "commandId": "cmd-98765",
  "durationMs": 15
}

Key Metrics

MetricTypeDescription
Commands processedCounterNumber of commands successfully handled
Events appendedCounterNumber of events written to the store
Snapshots createdCounterNumber of snapshots saved
Event store read latencyHistogramLatency of event store reads
Projection lagGaugeDelay between event and projection update
Replay durationHistogramTime taken to replay event streams
External call failuresCounterCount of failed external API calls

Use USE (Utilization, Saturation, Errors) for storage infrastructure and RED (Rate, Errors, Duration) for services.

Distributed Tracing (OpenTelemetry Example in Node.js)

import { trace, context } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-base';

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
provider.register();

const tracer = trace.getTracer('event-sourcing-tracer');

async function handleCommand(command: any) {
  const span = tracer.startSpan('handleCommand', {
    attributes: { commandType: command.type, aggregateId: command.aggregateId },
  });

  try {
    // Processing command
  } catch (err) {
    span.recordException(err);
    throw err;
  } finally {
    span.end();
  }
}

Span attributes help trace commands, event storage operations, projections, and external calls.

SLI/SLO Table

SLISLO TargetDescription
Event append success rate> 99.99%Percentage of commands producing stored events successfully
Replay completion time< 5 seconds (typical)Time to replay aggregate event stream for rebuild
Projection lag< 1 secondMax allowed delay between event and a queryable model update
Snapshot creation error rate< 0.1%Snapshot operation failures

Deep Dive: Internal Mechanics of Event Store Consistency and Concurrency Control

Building a reliable event-sourced system hinges on ensuring event store consistency and effectively managing concurrent command handling against aggregates. Since the event store is the single source of truth, any anomalies here ripple through projections, external integrations, and replay.

Optimistic Concurrency Control via Expected Versioning

Event stores typically implement optimistic concurrency control (OCC) to prevent lost updates and conflicting writes. The key principle:

  • When appending new events to an aggregate's event stream, the client specifies the expected current stream version (typically the last event sequence number seen).
  • The event store atomically verifies that the stream's current version matches the expected version.
  • If matches, the append succeeds with new events appended at incremented version numbers.
  • If no match (concurrent conflicting write), the store rejects with a concurrency conflict error.

This approach avoids locking, scales well, and provides concurrency control naturally.

interface AppendEventsParams {
  aggregateId: string;
  expectedVersion: number; // version of the last known event in stream
  events: Event[]; // new events to append
}

async function appendEvents(params: AppendEventsParams): Promise<void> {
  const currentVersion = await eventStore.getStreamVersion(params.aggregateId);

  if (currentVersion !== params.expectedVersion) {
    throw new Error('ConcurrencyConflictError');
  }

  await eventStore.append(params.aggregateId, params.events);
}

Versioning & Event IDs

Each event typically carries a global or stream-scoped monotonically increasing version or sequence number:

  • Enables strict ordering of events.
  • Facilitates consistent replay and snapshot versioning.
  • Acts as an optimistic concurrency token for appends.

Globally unique event IDs (e.g., UUIDv4 or ULID) help trace and debug event lineage, especially in distributed setups where events may be published to external buses.

Handling Concurrency Conflicts

Clients must implement retry logic on concurrency conflicts:

  • Re-fetch the current aggregate snapshot and event stream tail.
  • Rebuild aggregate state applying the missed events.
  • Retry the command with updated expected version and state.

Idempotent Command Processing

Because retries may re-submit the same command, the system should ensure idempotency:

  • Use a unique command ID with each command.
  • Store processed command IDs and their resulting events.
  • Reject or return cached results for duplicate command IDs.

This prevents duplicate events from being appended due to retries.

// Simplified pseudocode for idempotency check
if (eventStore.hasCommandBeenProcessed(command.id)) {
  return eventStore.getEventsByCommandId(command.id);
}

A Complete Production Scenario: From Command to Projection Update with Replay Recovery

Consider an online retail order system implementing event sourcing with snapshotting and projections backed by a NoSQL read store.

Step 1: Command Reception and Validation

User submits a PlaceOrder command:

{
  "commandId": "cmd-789",
  "aggregateId": "order-456",
  "type": "PlaceOrder",
  "payload": {
    "items": [{"productId": "p01", "quantity": 2}],
    "customerId": "cust-123"
  }
}

The command handler:

  • Validates command syntax and domain invariants (e.g., items in stock).
  • Rebuilds the aggregate state:
    • Loads latest snapshot for order-456.
    • Fetches events since snapshot.
    • Applies events to the order aggregate.
  • Calls the aggregate method placeOrder() which returns one or more domain events.

Step 2: Event Append with Concurrency Handling

Use the aggregate's last known version as expectedVersion to append events atomically to the event store.

try {
  await eventStore.appendEvents({
    aggregateId: "order-456",
    expectedVersion: aggregate.lastVersion,
    events: [/* OrderPlaced event */]
  });
} catch (err) {
  if (err.message === 'ConcurrencyConflictError') {
    // Reload aggregate and retry
  } else {
    throw err;
  }
}

Step 3: Publish and Project Events

Events appended trigger the event bus:

  • Projection services asynchronously consume new events.
  • Projections are updated (e.g., read models store order summary, status, inventory stock decrements).
  • The updated read models are query-efficient and denormalized.

Step 4: Handling External Side Effects

E.g., send order confirmation email:

  • Use the side effect suppression flag during replay.
  • When replay is live or incremental, the side effect handler performs external calls.
  • Results (emails sent) can optionally be recorded as events to keep the event log deterministic.

Step 5: Snapshot Creation

After every N events appended per aggregate (configurable, e.g., every 100 events), create a snapshot:

  • Serialize aggregate state.
  • Save snapshot with last event version.
  • Snapshots ensure faster rebuild during subsequent command handling or debug replay.

Step 6: Replay for Debugging or Recovery

Replay all events for an order aggregate:

  • Load snapshot.
  • Apply all subsequent events.
  • Validate correctness of read models or fix bugs with code patches.

Edge Cases and Failure Modes in Event Sourcing Systems

Event sourcing systems have distinctive failure points requiring explicit strategies.

1. Event Store Corruption or Data Loss

Being the canonical source of truth, any corruption (partial writes, data loss) can cause irrecoverable application state corruption.

Mitigations:

  • Use ACID-compliant event stores or append-only logs with checksums.
  • Frequent backups and restore drills.
  • Event replication across multiple nodes or datacenters for resilience.

2. Snapshot Inconsistencies

Out-of-sync snapshots can cause replays leading to incorrect state:

  • Snapshot creation must be atomic with event append or performed on isolated, stable state.
  • On replay, validate snapshots (e.g., checksum, version) before accepting.

3. Upcaster Failures and Backward Compatibility Breaks

A rogue upcaster can silently corrupt event streams on replay:

  • Maintain exhaustive unit tests for all upcasters.
  • Apply upcasters as pure, deterministic functions.
  • Use event namespace versioning to route events to proper upcasters.

4. Eventual Consistency & Projection Lag

Projections may lag behind events, causing stale reads:

  • Monitor projection lag metrics.
  • Avoid critical workflows depending on immediate consistency.
  • For strict consistency, consider CQRS-safe synchronous reads off aggregates at command time.

5. External Side Effect Duplication

Failed or retried side effect calls during replays risk duplicated communications (emails, payments):

  • Always suppress side effects during replay or
  • Persist external call outcomes as events and replay them deterministically.

Performance Considerations and Tuning Strategies

Event sourcing introduces unique latency and storage patterns.

Rebuild Time vs. Snapshot Frequency

  • Frequent snapshots speed aggregate rebuild at a cost of storage and I/O.
  • Choose snapshot interval based on aggregate event rate and rebuild urgency.

Event Store Write Throughput

  • Append-only writes tend to be sequential and performant.
  • Indexing and querying the event log by aggregateId and timestamp critical for replay latency.
  • Horizontal partitioning shards by aggregate ID can support scale.

Read Model Projection Tuning

  • Projection pipelines should be horizontally scalable consumers.
  • Use idempotent projection handlers to safely replay and recover.
  • Monitor consumer lag closely to detect bottlenecks.

Storage Format & Compression

  • Compact event data using binary formats (Avro, Protobuf) for size and speed.
  • Use compression judiciously on older, rarely accessed events.

Decision Checklist: When to Use Event Sourcing and How to Design

QuestionGuidance
Does domain require full audit trail?Event sourcing shines for auditability
Is temporal query or time travel essential?Supports complex historical queries
Will your aggregates have long life spans?Snapshot strategy critical
Can system handle eventual consistency trade-offs?Benefit of projection decoupling
Is side effect integration manageable?Invest in suppression or event-result pattern
Are projections and rebuilds operationally tested?Essential for reliability
Does your team have capacity to maintain schema evolutions and upcasters?Necessary investment
Do current CRUD origins suffice for your needs?Keep simple unless audit/temporal needs arise

Answering yes to most of these favors event sourcing. Otherwise, balance cost and complexity carefully.


Observability Enhancements: Detecting and Diagnosing Concurrency Conflicts and Data Drift

Extending observability, focus on key signals around concurrency and correctness:

  • Log ConcurrencyConflictError with command and aggregate details.
  • Trace command retry loops with durations and counts.
  • Implement alerting on rising concurrency conflicts (> baseline).
  • Monitor projection divergence by occasionally comparing aggregates rebuilt from events vs. projections.
  • Integrate shadow replays in staging to detect potential data drift silently.

A sample structured log for a concurrency conflict:

{
  "timestamp": "2024-04-27T16:20:33Z",
  "level": "warn",
  "event": "concurrency_conflict",
  "commandId": "cmd-321",
  "aggregateId": "order-456",
  "expectedVersion": 45,
  "actualVersion": 47,
  "retryCount": 2
}

This data enriches post-mortem root cause analysis and feature engineering to mitigate repeated conflicts.


The above sections add comprehensive insights on inner workings, practical failure scenarios, and operational best practices that mature engineers must master to build robust event sourcing systems.

Conclusion

Event sourcing is a powerful pattern with trade-offs. Careful design of event shape, aggregate granularity, snapshotting, and upcasting strategies is required for scalable, maintainable systems. Pay special attention to safe handling of side effects during event replays. Finally, invest in operability — tooling to replay events, structured logging, metrics, and tracing to maintain confidence and agility as your system evolves.


For more on event-driven architectures and CQRS, see the official Martin Fowler Event Sourcing guide.