Skip to content

Load Balancing Traffic Steering: A Control Plane and Data Plane Design

Design traffic steering as a versioned control plane that publishes safe routing policy to fast, independent data planes, with explicit health, locality, capacity, and failure behavior.

Ekanatha Reddy Urivakili11 min2,017 words

Design goal

Traffic steering decides which destination should receive a request; load balancing distributes requests among eligible destinations. A production design needs both decisions, but they should not be confused. A global policy might prefer a nearby region, then a failover region; a regional load balancer then selects a healthy instance in that region.

This design uses an edge or regional proxy as the data plane and a separate control plane to author and distribute routing policy. It applies whether the proxy is an application gateway, an ingress tier, or a service-owned reverse proxy. The central principle is that request routing must continue when the control plane is unavailable.

  • Steering inputs: destination health, client or workload locality, configured weights, capacity signals, and policy version.
  • Data-plane responsibility: make a bounded, fast decision from a locally held policy snapshot; forward the request; report telemetry.
  • Control-plane responsibility: validate and publish policy, coordinate rollout, and provide operators with status and audit history.
  • Not a routing input: identity authorization. A routing decision must never grant access to a destination or replace application authorization.

Requirements and boundaries

Functional requirements

  • Route requests among configured destinations using a policy that can express locality preference, weights, and failover order.
  • Exclude destinations that are unhealthy, administratively disabled, or over a defined capacity limit.
  • Keep routing decisions available during control-plane, network, or configuration-store outages by retaining the last known good policy.
  • Support gradual traffic shifts and rollback without requiring a data-plane process restart.
  • Expose the selected destination and policy version in logs and metrics, while avoiding sensitive affinity values.

Operational requirements

  • Bound the decision path: no database lookup, synchronous control-plane call, or unbounded retry per request.
  • Make policy updates validated, versioned, auditable, and atomic at each data-plane instance.
  • Have an explicit stale-policy limit and a defined behavior when no destination is eligible.
  • Keep health and capacity signals distinct from policy. A weight is an operator preference, not proof that a destination can serve traffic.

Keep the first version deliberately narrow. Do not build a universal policy language or let arbitrary request headers become trusted routing commands. A small policy model is easier to reason about, validate, and operate during an incident.

High-level design

Policy changes travel from an authenticated operator API through validation and durable storage to a publisher. Each data-plane instance receives a complete, versioned snapshot and swaps it atomically. Health observations are collected independently. Requests use only local state, so a control-plane outage affects management, not the routing hot path.

HLD flow: policy distribution and request routing

flowchart LR
  Operator["Operator"] --> Api["Management API"]
  Api --> Validate["Validate and persist policy"]
  Validate --> Publisher["Snapshot publisher"]
  Publisher --> Stream["Config distribution"]
  Stream --> Proxy["Data-plane proxy"]
  Probe["Health and capacity probes"] --> Proxy
  Client["Client request"] --> Proxy
  Proxy --> Select["Select eligible destination"]
  Select --> Backend["Backend pool"]
  Proxy --> Metrics["Metrics and decision logs"]

The proxy can be a fleet of independent instances: each instance has the same policy snapshot and its own recent health view. That avoids a shared decision service becoming a latency and availability dependency.

Storage model

Persist policy intent and publication history separately from transient health. A policy revision is immutable after publication; a new change creates a new revision. Health samples are ephemeral observations and should not be written into the policy record as if they were operator intent.

ERD: durable policy entities

erDiagram
  ROUTE_POLICY ||--o{ POLICY_REVISION : has
  POLICY_REVISION ||--o{ DESTINATION_RULE : defines
  DESTINATION ||--o{ DESTINATION_RULE : referenced_by
  POLICY_REVISION ||--o{ PUBLISH_EVENT : produces
  ROUTE_POLICY {
    string id
    string name
    string active_revision_id
  }
  POLICY_REVISION {
    string id
    string policy_id
    int version
    string state
    datetime created_at
  }
  DESTINATION {
    string id
    string endpoint
    string region
    bool enabled
  }
  DESTINATION_RULE {
    string revision_id
    string destination_id
    int weight
    int failover_order
  }
  PUBLISH_EVENT {
    string id
    string revision_id
    string outcome
    datetime created_at
  }
  • Store endpoint references, not credentials. Resolve credentials through the platform’s normal secret mechanism.
  • Enforce uniqueness for policy revision numbers and destination rules within a revision.
  • Use a transactional write for the revision and its rules. Mark a revision active only after validation and publication requirements succeed.

Management API and policy contract

The management API is not called on each user request. It is an administrative interface and should use authenticated, authorized, auditable operations. For updates, require an expected version or ETag so two operators cannot silently overwrite each other.

Method and pathPurposeImportant behavior
GET /admin/routes/{id}Read active policy and versionReturn policy state, not live per-proxy health.
POST /admin/routes/{id}/revisionsCreate a draft revisionValidate destinations, weights, regions, and failover rules.
POST /admin/routes/{id}/revisions/{version}/publishPublish a revisionRequire an idempotency key; record actor and outcome.
POST /admin/routes/{id}/rollbackActivate a prior known-good revisionCreate a new publication event; do not rewrite history.
GET /admin/routes/{id}/statusInspect rollout stateReport observed policy versions by proxy group and stale instances.

A policy payload should be intentionally constrained. For example, define a policy identifier, monotonically increasing version, selection mode, locality rule, and a list of destination identifiers with positive integer weights. Keep endpoint resolution and probe configuration in separately validated destination records. Reject unknown fields if the API contract requires strict configuration, and reject empty or duplicate destinations rather than guessing operator intent.

Low-level design

Keep the decision engine a small, deterministic component. Its inputs are an immutable policy snapshot, a request’s routing context, and the latest health/capacity view. Its output is a destination plus a reason code and policy version. The proxy owns forwarding, timeout enforcement, connection management, and any bounded retry policy.

LLD class diagram

classDiagram
  class PolicyPublisher {
    +Publish(revision)
    +Rollback(revisionId)
  }
  class SnapshotStore {
    +GetCurrent()
    +Swap(snapshot)
  }
  class HealthMonitor {
    +GetStatus(destinationId)
    +UpdateSample(sample)
  }
  class RoutingEngine {
    +Choose(request, snapshot, health)
  }
  class ProxyHandler {
    +Forward(request)
  }
  class Destination {
    +Id
    +Region
    +Weight
  }
  PolicyPublisher --> SnapshotStore
  ProxyHandler --> RoutingEngine
  RoutingEngine --> SnapshotStore
  RoutingEngine --> HealthMonitor
  RoutingEngine --> Destination

Use immutable snapshots so a request cannot observe half of a policy update. Validate an incoming snapshot before swapping it; if validation fails, retain the previous snapshot and emit an alert. The health view can be mutable internally, but reads should be thread-safe and bounded in cost.

Minimal example: weighted selection

var destinations = new[]
{
    new Destination("east", 80),
    new Destination("west", 20)
};

var selected = WeightedPicker.Pick(destinations, Random.Shared);
Console.WriteLine(selected.Id);

record Destination(string Id, int Weight);

static class WeightedPicker
{
    public static Destination Pick(
        IReadOnlyList<Destination> destinations,
        Random random)
    {
        ArgumentNullException.ThrowIfNull(destinations);
        ArgumentNullException.ThrowIfNull(random);

        var eligible = destinations.Where(d => d.Weight > 0).ToArray();
        if (eligible.Length == 0)
            throw new InvalidOperationException("No weighted destinations.");

        var total = eligible.Sum(d => (long)d.Weight);
        var slot = random.NextInt64(total);
        foreach (var destination in eligible)
        {
            if (slot < destination.Weight)
                return destination;
            slot -= destination.Weight;
        }

        throw new InvalidOperationException("Selection invariant failed.");
    }
}

This demonstrates weighted choice only: production steering must filter eligibility first and define locality, stickiness, observability, and the no-candidate behavior.

Realistic example: local preference, health, and stable affinity

using System.Buffers.Binary;
using System.Security.Cryptography;
using System.Text;

var request = new RouteRequest("account-482", "eu-west");
var destinations = new[]
{
    new Destination("eu-a", "eu-west", 70, true, 120),
    new Destination("eu-b", "eu-west", 30, true, 80),
    new Destination("us-a", "us-east", 100, true, 200)
};

var decision = RoutingEngine.Choose(request, "policy-v17", destinations);
Console.WriteLine($"{decision.Destination.Id}: {decision.Reason}");

record Destination(
    string Id, string Region, int Weight, bool Healthy, int AvailableSlots);
record RouteRequest(string AffinityKey, string ClientRegion);
record RouteDecision(Destination Destination, string Reason);

static class RoutingEngine
{
    public static RouteDecision Choose(
        RouteRequest request,
        string policyVersion,
        IReadOnlyList<Destination> destinations)
    {
        var eligible = destinations
            .Where(d => d.Healthy && d.Weight > 0 && d.AvailableSlots > 0)
            .OrderBy(d => d.Id, StringComparer.Ordinal)
            .ToArray();

        var local = eligible
            .Where(d => d.Region == request.ClientRegion)
            .ToArray();
        var candidates = local.Length > 0 ? local : eligible;
        if (candidates.Length == 0)
            throw new InvalidOperationException("No destination is eligible.");

        var total = candidates.Sum(d => (long)d.Weight);
        var input = Encoding.UTF8.GetBytes(
            $"{policyVersion}\n{request.AffinityKey}");
        var hash = SHA256.HashData(input);
        var slot = (long)(BinaryPrimitives.ReadUInt64BigEndian(hash) % (ulong)total);

        foreach (var destination in candidates)
        {
            if (slot < destination.Weight)
                return new RouteDecision(
                    destination,
                    local.Length > 0 ? "local-weighted" : "failover-weighted");
            slot -= destination.Weight;
        }

        throw new InvalidOperationException("Selection invariant failed.");
    }
}

Notice that locality is a preference rather than a hard requirement: if no local destination qualifies, the engine uses the remaining healthy pool, and the policy version deliberately participates in affinity mapping.

This example is a decision function, not a complete proxy. In production, validate request keys and policy inputs, cap destination counts and weights, use a stable destination order, and avoid logging the raw affinity key. If affinity is not required, a per-request weighted choice may distribute load more evenly as membership changes.

Request and update flows

A request should not wait for health probes or policy publication. The data plane evaluates the latest local view, selects once, and forwards with a deadline. Retries, if enabled, must be bounded and must not turn an overloaded backend into a retry storm.

Sequence: request to backend

sequenceDiagram
  participant C as Client
  participant P as Proxy
  participant R as Routing engine
  participant H as Health view
  participant B as Backend
  C->>P: Request with routing context
  P->>R: Choose from current snapshot
  R->>H: Read eligibility state
  H-->>R: Health and capacity
  R-->>P: Destination and reason
  P->>B: Forward with deadline
  B-->>P: Response
  P-->>C: Response

Sequence: policy publication

sequenceDiagram
  participant O as Operator
  participant A as Admin API
  participant S as Policy store
  participant D as Distributor
  participant P as Proxy
  O->>A: Publish revision with expected version
  A->>A: Validate policy
  A->>S: Commit immutable revision
  S-->>A: Commit confirmed
  A->>D: Publish versioned snapshot
  D->>P: Deliver complete snapshot
  P->>P: Validate then atomically swap
  P-->>D: Acknowledge version
  D-->>A: Report rollout status

Publication should be considered successful only according to a declared rollout policy. For example, require acknowledgement from a defined proxy group before reporting complete, while exposing partial rollout as a distinct state. Do not claim fleet-wide activation based solely on a successful database commit.

Key algorithm and routing semantics

One understandable algorithm is locality-first weighted selection with health and capacity filtering. The rules are applied in order, and the final selection uses the policy’s stable destination order. A more advanced system can add latency-aware weights, but that signal needs smoothing and strict bounds; reacting to every short-lived latency spike causes oscillation.

Algorithm flow: choose one destination

flowchart TD
  Start["Receive request and snapshot"] --> Validate["Check snapshot and request context"]
  Validate --> Filter["Remove disabled, unhealthy, or full destinations"]
  Filter --> Any{ "Any eligible destination?" }
  Any -- Yes --> Local["Keep local region if one is eligible"]
  Any -- No --> NoRoute["Return controlled no-route result"]
  Local --> Weighted["Select by configured weights"]
  Weighted --> Forward["Forward with deadline"]
  Forward --> Observe["Record version, destination, and outcome"]
  • Locality: decide whether local preference is soft or hard. Soft preference improves availability but may cross regions; hard locality can produce avoidable failures when a region is impaired.
  • Weight: apply weights only among currently eligible destinations. A configured weight of zero should mean intentionally excluded, not “try it if everything else fails.” Define failover separately.
  • Affinity: use a stable, privacy-conscious key only when the application benefits from it. A policy or membership change may remap users; document that behavior and never treat affinity as session security.
  • Capacity: a coarse available-slot signal can prevent obvious overload, but it is stale by nature. Combine it with backend admission control and queue limits.
  • No route: return a controlled service-unavailable response or use an explicitly configured failover pool. Do not silently send traffic to a disabled destination.

Health, telemetry, and safety

Health checks answer whether a destination is eligible, not whether every request will succeed. Use a shallow readiness check for routing and observe real request outcomes as a separate signal. A single failed probe should not immediately eject a destination unless the failure is decisive; use thresholds, recovery hysteresis, and a minimum healthy duration to avoid flapping.

  • Probe from the relevant network location. A backend reachable from one region may not be reachable from another.
  • Separate liveness from readiness. Restarting a process because a dependency is temporarily unavailable can amplify an incident.
  • Track decision count and request outcome by destination, region, policy version, and reason. Keep metric dimensions bounded; do not use request IDs or arbitrary tenant IDs as metric labels.
  • Alert on stale snapshots, missing acknowledgements, no eligible destinations, health flapping, and a growing difference between configured and observed traffic share.
  • Use sampled structured logs for individual decisions when needed. Do not emit a high-cardinality log or metric for every routing detail without a retention and cost plan.

Failure modes and trade-offs

FailureExpected behaviorDesign trade-off
Control plane unavailableContinue with last known good snapshot; block new changes.Availability is favored over immediate policy freshness.
Snapshot is invalid or incompleteReject it and retain the prior snapshot; report the rejected version.Some proxies may temporarily run an older version.
Health data is staleApply a configured staleness threshold, then fail conservatively or use a declared fallback.Fail-open can route to a dead backend; fail-closed can reduce capacity unnecessarily.
All destinations are unhealthyReturn a bounded no-route response and alert; do not loop through retries.Fast failure sacrifices a chance that a failing backend might recover for one request.
Proxy instances have different snapshotsExpose version skew, alert, and pause or roll back publication if required.Strict synchronized rollout improves consistency but reduces rollout availability.
Capacity signal is inaccurateKeep backend admission control authoritative and limit retries.More frequent signals add load and still cannot guarantee capacity at decision time.
Affinity key is attacker-controlledValidate size and format; do not let it affect authorization or reach arbitrary endpoints.Stable affinity improves continuity but can be abused to concentrate traffic.

DNS steering versus proxy steering: DNS-based global steering is useful when clients connect directly to regional endpoints, but resolver caching and TTLs mean changes are not instantaneous. Proxy steering offers request-level decisions, consistent telemetry, and fast policy application, at the cost of proxy capacity, an extra hop, and a more demanding availability design. Choose based on where connections terminate; do not promise immediate failover from a DNS change.

Centralized versus distributed decisions: a central routing service can simplify policy ownership, but adds a network dependency to every decision unless it is cached. A local data plane scales decisions with traffic and tolerates control-plane outages, but requires careful configuration distribution and version observability. This design chooses distributed decisions and centralized policy management.

Weights versus adaptive routing: static weights are explainable and easy to roll back. Adaptive routing can react to changing load but may oscillate, amplify noisy measurements, and obscure why traffic moved. Start with explicit weights and health; introduce adaptive changes only with bounded adjustment rates, a stable control interval, and an operator-visible reason.

Phased implementation plan

  1. Phase 1 — single-region data plane. Implement destination configuration, a simple weighted picker, active health checks, request deadlines, no-route behavior, and decision metrics. Test that unhealthy and disabled destinations are never selected.
  2. Phase 2 — versioned policy snapshots. Add immutable revisions, validation, atomic data-plane swaps, acknowledgement status, audit events, and rollback. Inject malformed and delayed updates in tests.
  3. Phase 3 — locality and failover. Add region labels and explicit local-preference/failover semantics. Test complete regional failure, partial destination failure, and recovery hysteresis.
  4. Phase 4 — controlled traffic shifts. Add weighted rollout procedures, compare observed distribution with intended distribution, and provide a fast operator rollback. Keep the previous known-good revision available.
  5. Phase 5 — resilience and operations. Exercise control-plane outages, stale probes, proxy restarts, network partitions, and skewed snapshots. Document on-call actions and define alert thresholds from service objectives.

Before increasing sophistication, prove the basics under load and failure: the routing path is local, bounded, observable, and deterministic for the same inputs; publication is safe; and every failure has a documented outcome. That is the difference between a policy that looks correct in a control panel and traffic steering that remains predictable during an incident.