Design goal
Traffic steering decides which destination should receive a request; load balancing distributes requests among eligible destinations. A production design needs both decisions, but they should not be confused. A global policy might prefer a nearby region, then a failover region; a regional load balancer then selects a healthy instance in that region.
This design uses an edge or regional proxy as the data plane and a separate control plane to author and distribute routing policy. It applies whether the proxy is an application gateway, an ingress tier, or a service-owned reverse proxy. The central principle is that request routing must continue when the control plane is unavailable.
- Steering inputs: destination health, client or workload locality, configured weights, capacity signals, and policy version.
- Data-plane responsibility: make a bounded, fast decision from a locally held policy snapshot; forward the request; report telemetry.
- Control-plane responsibility: validate and publish policy, coordinate rollout, and provide operators with status and audit history.
- Not a routing input: identity authorization. A routing decision must never grant access to a destination or replace application authorization.
Requirements and boundaries
Functional requirements
- Route requests among configured destinations using a policy that can express locality preference, weights, and failover order.
- Exclude destinations that are unhealthy, administratively disabled, or over a defined capacity limit.
- Keep routing decisions available during control-plane, network, or configuration-store outages by retaining the last known good policy.
- Support gradual traffic shifts and rollback without requiring a data-plane process restart.
- Expose the selected destination and policy version in logs and metrics, while avoiding sensitive affinity values.
Operational requirements
- Bound the decision path: no database lookup, synchronous control-plane call, or unbounded retry per request.
- Make policy updates validated, versioned, auditable, and atomic at each data-plane instance.
- Have an explicit stale-policy limit and a defined behavior when no destination is eligible.
- Keep health and capacity signals distinct from policy. A weight is an operator preference, not proof that a destination can serve traffic.
Keep the first version deliberately narrow. Do not build a universal policy language or let arbitrary request headers become trusted routing commands. A small policy model is easier to reason about, validate, and operate during an incident.
High-level design
Policy changes travel from an authenticated operator API through validation and durable storage to a publisher. Each data-plane instance receives a complete, versioned snapshot and swaps it atomically. Health observations are collected independently. Requests use only local state, so a control-plane outage affects management, not the routing hot path.
HLD flow: policy distribution and request routing
flowchart LR Operator["Operator"] --> Api["Management API"] Api --> Validate["Validate and persist policy"] Validate --> Publisher["Snapshot publisher"] Publisher --> Stream["Config distribution"] Stream --> Proxy["Data-plane proxy"] Probe["Health and capacity probes"] --> Proxy Client["Client request"] --> Proxy Proxy --> Select["Select eligible destination"] Select --> Backend["Backend pool"] Proxy --> Metrics["Metrics and decision logs"]
The proxy can be a fleet of independent instances: each instance has the same policy snapshot and its own recent health view. That avoids a shared decision service becoming a latency and availability dependency.
Storage model
Persist policy intent and publication history separately from transient health. A policy revision is immutable after publication; a new change creates a new revision. Health samples are ephemeral observations and should not be written into the policy record as if they were operator intent.
ERD: durable policy entities
erDiagram
ROUTE_POLICY ||--o{ POLICY_REVISION : has
POLICY_REVISION ||--o{ DESTINATION_RULE : defines
DESTINATION ||--o{ DESTINATION_RULE : referenced_by
POLICY_REVISION ||--o{ PUBLISH_EVENT : produces
ROUTE_POLICY {
string id
string name
string active_revision_id
}
POLICY_REVISION {
string id
string policy_id
int version
string state
datetime created_at
}
DESTINATION {
string id
string endpoint
string region
bool enabled
}
DESTINATION_RULE {
string revision_id
string destination_id
int weight
int failover_order
}
PUBLISH_EVENT {
string id
string revision_id
string outcome
datetime created_at
}- Store endpoint references, not credentials. Resolve credentials through the platform’s normal secret mechanism.
- Enforce uniqueness for policy revision numbers and destination rules within a revision.
- Use a transactional write for the revision and its rules. Mark a revision active only after validation and publication requirements succeed.
Management API and policy contract
The management API is not called on each user request. It is an administrative interface and should use authenticated, authorized, auditable operations. For updates, require an expected version or ETag so two operators cannot silently overwrite each other.
| Method and path | Purpose | Important behavior |
|---|---|---|
| GET /admin/routes/{id} | Read active policy and version | Return policy state, not live per-proxy health. |
| POST /admin/routes/{id}/revisions | Create a draft revision | Validate destinations, weights, regions, and failover rules. |
| POST /admin/routes/{id}/revisions/{version}/publish | Publish a revision | Require an idempotency key; record actor and outcome. |
| POST /admin/routes/{id}/rollback | Activate a prior known-good revision | Create a new publication event; do not rewrite history. |
| GET /admin/routes/{id}/status | Inspect rollout state | Report observed policy versions by proxy group and stale instances. |
A policy payload should be intentionally constrained. For example, define a policy identifier, monotonically increasing version, selection mode, locality rule, and a list of destination identifiers with positive integer weights. Keep endpoint resolution and probe configuration in separately validated destination records. Reject unknown fields if the API contract requires strict configuration, and reject empty or duplicate destinations rather than guessing operator intent.
Low-level design
Keep the decision engine a small, deterministic component. Its inputs are an immutable policy snapshot, a request’s routing context, and the latest health/capacity view. Its output is a destination plus a reason code and policy version. The proxy owns forwarding, timeout enforcement, connection management, and any bounded retry policy.
LLD class diagram
classDiagram
class PolicyPublisher {
+Publish(revision)
+Rollback(revisionId)
}
class SnapshotStore {
+GetCurrent()
+Swap(snapshot)
}
class HealthMonitor {
+GetStatus(destinationId)
+UpdateSample(sample)
}
class RoutingEngine {
+Choose(request, snapshot, health)
}
class ProxyHandler {
+Forward(request)
}
class Destination {
+Id
+Region
+Weight
}
PolicyPublisher --> SnapshotStore
ProxyHandler --> RoutingEngine
RoutingEngine --> SnapshotStore
RoutingEngine --> HealthMonitor
RoutingEngine --> DestinationUse immutable snapshots so a request cannot observe half of a policy update. Validate an incoming snapshot before swapping it; if validation fails, retain the previous snapshot and emit an alert. The health view can be mutable internally, but reads should be thread-safe and bounded in cost.
Minimal example: weighted selection
var destinations = new[]
{
new Destination("east", 80),
new Destination("west", 20)
};
var selected = WeightedPicker.Pick(destinations, Random.Shared);
Console.WriteLine(selected.Id);
record Destination(string Id, int Weight);
static class WeightedPicker
{
public static Destination Pick(
IReadOnlyList<Destination> destinations,
Random random)
{
ArgumentNullException.ThrowIfNull(destinations);
ArgumentNullException.ThrowIfNull(random);
var eligible = destinations.Where(d => d.Weight > 0).ToArray();
if (eligible.Length == 0)
throw new InvalidOperationException("No weighted destinations.");
var total = eligible.Sum(d => (long)d.Weight);
var slot = random.NextInt64(total);
foreach (var destination in eligible)
{
if (slot < destination.Weight)
return destination;
slot -= destination.Weight;
}
throw new InvalidOperationException("Selection invariant failed.");
}
}
This demonstrates weighted choice only: production steering must filter eligibility first and define locality, stickiness, observability, and the no-candidate behavior.
Realistic example: local preference, health, and stable affinity
using System.Buffers.Binary;
using System.Security.Cryptography;
using System.Text;
var request = new RouteRequest("account-482", "eu-west");
var destinations = new[]
{
new Destination("eu-a", "eu-west", 70, true, 120),
new Destination("eu-b", "eu-west", 30, true, 80),
new Destination("us-a", "us-east", 100, true, 200)
};
var decision = RoutingEngine.Choose(request, "policy-v17", destinations);
Console.WriteLine($"{decision.Destination.Id}: {decision.Reason}");
record Destination(
string Id, string Region, int Weight, bool Healthy, int AvailableSlots);
record RouteRequest(string AffinityKey, string ClientRegion);
record RouteDecision(Destination Destination, string Reason);
static class RoutingEngine
{
public static RouteDecision Choose(
RouteRequest request,
string policyVersion,
IReadOnlyList<Destination> destinations)
{
var eligible = destinations
.Where(d => d.Healthy && d.Weight > 0 && d.AvailableSlots > 0)
.OrderBy(d => d.Id, StringComparer.Ordinal)
.ToArray();
var local = eligible
.Where(d => d.Region == request.ClientRegion)
.ToArray();
var candidates = local.Length > 0 ? local : eligible;
if (candidates.Length == 0)
throw new InvalidOperationException("No destination is eligible.");
var total = candidates.Sum(d => (long)d.Weight);
var input = Encoding.UTF8.GetBytes(
$"{policyVersion}\n{request.AffinityKey}");
var hash = SHA256.HashData(input);
var slot = (long)(BinaryPrimitives.ReadUInt64BigEndian(hash) % (ulong)total);
foreach (var destination in candidates)
{
if (slot < destination.Weight)
return new RouteDecision(
destination,
local.Length > 0 ? "local-weighted" : "failover-weighted");
slot -= destination.Weight;
}
throw new InvalidOperationException("Selection invariant failed.");
}
}
Notice that locality is a preference rather than a hard requirement: if no local destination qualifies, the engine uses the remaining healthy pool, and the policy version deliberately participates in affinity mapping.
This example is a decision function, not a complete proxy. In production, validate request keys and policy inputs, cap destination counts and weights, use a stable destination order, and avoid logging the raw affinity key. If affinity is not required, a per-request weighted choice may distribute load more evenly as membership changes.
Request and update flows
A request should not wait for health probes or policy publication. The data plane evaluates the latest local view, selects once, and forwards with a deadline. Retries, if enabled, must be bounded and must not turn an overloaded backend into a retry storm.
Sequence: request to backend
sequenceDiagram participant C as Client participant P as Proxy participant R as Routing engine participant H as Health view participant B as Backend C->>P: Request with routing context P->>R: Choose from current snapshot R->>H: Read eligibility state H-->>R: Health and capacity R-->>P: Destination and reason P->>B: Forward with deadline B-->>P: Response P-->>C: Response
Sequence: policy publication
sequenceDiagram participant O as Operator participant A as Admin API participant S as Policy store participant D as Distributor participant P as Proxy O->>A: Publish revision with expected version A->>A: Validate policy A->>S: Commit immutable revision S-->>A: Commit confirmed A->>D: Publish versioned snapshot D->>P: Deliver complete snapshot P->>P: Validate then atomically swap P-->>D: Acknowledge version D-->>A: Report rollout status
Publication should be considered successful only according to a declared rollout policy. For example, require acknowledgement from a defined proxy group before reporting complete, while exposing partial rollout as a distinct state. Do not claim fleet-wide activation based solely on a successful database commit.
Key algorithm and routing semantics
One understandable algorithm is locality-first weighted selection with health and capacity filtering. The rules are applied in order, and the final selection uses the policy’s stable destination order. A more advanced system can add latency-aware weights, but that signal needs smoothing and strict bounds; reacting to every short-lived latency spike causes oscillation.
Algorithm flow: choose one destination
flowchart TD
Start["Receive request and snapshot"] --> Validate["Check snapshot and request context"]
Validate --> Filter["Remove disabled, unhealthy, or full destinations"]
Filter --> Any{ "Any eligible destination?" }
Any -- Yes --> Local["Keep local region if one is eligible"]
Any -- No --> NoRoute["Return controlled no-route result"]
Local --> Weighted["Select by configured weights"]
Weighted --> Forward["Forward with deadline"]
Forward --> Observe["Record version, destination, and outcome"]- Locality: decide whether local preference is soft or hard. Soft preference improves availability but may cross regions; hard locality can produce avoidable failures when a region is impaired.
- Weight: apply weights only among currently eligible destinations. A configured weight of zero should mean intentionally excluded, not “try it if everything else fails.” Define failover separately.
- Affinity: use a stable, privacy-conscious key only when the application benefits from it. A policy or membership change may remap users; document that behavior and never treat affinity as session security.
- Capacity: a coarse available-slot signal can prevent obvious overload, but it is stale by nature. Combine it with backend admission control and queue limits.
- No route: return a controlled service-unavailable response or use an explicitly configured failover pool. Do not silently send traffic to a disabled destination.
Health, telemetry, and safety
Health checks answer whether a destination is eligible, not whether every request will succeed. Use a shallow readiness check for routing and observe real request outcomes as a separate signal. A single failed probe should not immediately eject a destination unless the failure is decisive; use thresholds, recovery hysteresis, and a minimum healthy duration to avoid flapping.
- Probe from the relevant network location. A backend reachable from one region may not be reachable from another.
- Separate liveness from readiness. Restarting a process because a dependency is temporarily unavailable can amplify an incident.
- Track decision count and request outcome by destination, region, policy version, and reason. Keep metric dimensions bounded; do not use request IDs or arbitrary tenant IDs as metric labels.
- Alert on stale snapshots, missing acknowledgements, no eligible destinations, health flapping, and a growing difference between configured and observed traffic share.
- Use sampled structured logs for individual decisions when needed. Do not emit a high-cardinality log or metric for every routing detail without a retention and cost plan.
Failure modes and trade-offs
| Failure | Expected behavior | Design trade-off |
|---|---|---|
| Control plane unavailable | Continue with last known good snapshot; block new changes. | Availability is favored over immediate policy freshness. |
| Snapshot is invalid or incomplete | Reject it and retain the prior snapshot; report the rejected version. | Some proxies may temporarily run an older version. |
| Health data is stale | Apply a configured staleness threshold, then fail conservatively or use a declared fallback. | Fail-open can route to a dead backend; fail-closed can reduce capacity unnecessarily. |
| All destinations are unhealthy | Return a bounded no-route response and alert; do not loop through retries. | Fast failure sacrifices a chance that a failing backend might recover for one request. |
| Proxy instances have different snapshots | Expose version skew, alert, and pause or roll back publication if required. | Strict synchronized rollout improves consistency but reduces rollout availability. |
| Capacity signal is inaccurate | Keep backend admission control authoritative and limit retries. | More frequent signals add load and still cannot guarantee capacity at decision time. |
| Affinity key is attacker-controlled | Validate size and format; do not let it affect authorization or reach arbitrary endpoints. | Stable affinity improves continuity but can be abused to concentrate traffic. |
DNS steering versus proxy steering: DNS-based global steering is useful when clients connect directly to regional endpoints, but resolver caching and TTLs mean changes are not instantaneous. Proxy steering offers request-level decisions, consistent telemetry, and fast policy application, at the cost of proxy capacity, an extra hop, and a more demanding availability design. Choose based on where connections terminate; do not promise immediate failover from a DNS change.
Centralized versus distributed decisions: a central routing service can simplify policy ownership, but adds a network dependency to every decision unless it is cached. A local data plane scales decisions with traffic and tolerates control-plane outages, but requires careful configuration distribution and version observability. This design chooses distributed decisions and centralized policy management.
Weights versus adaptive routing: static weights are explainable and easy to roll back. Adaptive routing can react to changing load but may oscillate, amplify noisy measurements, and obscure why traffic moved. Start with explicit weights and health; introduce adaptive changes only with bounded adjustment rates, a stable control interval, and an operator-visible reason.
Phased implementation plan
- Phase 1 — single-region data plane. Implement destination configuration, a simple weighted picker, active health checks, request deadlines, no-route behavior, and decision metrics. Test that unhealthy and disabled destinations are never selected.
- Phase 2 — versioned policy snapshots. Add immutable revisions, validation, atomic data-plane swaps, acknowledgement status, audit events, and rollback. Inject malformed and delayed updates in tests.
- Phase 3 — locality and failover. Add region labels and explicit local-preference/failover semantics. Test complete regional failure, partial destination failure, and recovery hysteresis.
- Phase 4 — controlled traffic shifts. Add weighted rollout procedures, compare observed distribution with intended distribution, and provide a fast operator rollback. Keep the previous known-good revision available.
- Phase 5 — resilience and operations. Exercise control-plane outages, stale probes, proxy restarts, network partitions, and skewed snapshots. Document on-call actions and define alert thresholds from service objectives.
Before increasing sophistication, prove the basics under load and failure: the routing path is local, bounded, observable, and deterministic for the same inputs; publication is safe; and every failure has a documented outcome. That is the difference between a policy that looks correct in a control panel and traffic steering that remains predictable during an incident.