Skip to content

Choosing Where Capacity Goes: Horizontal and Vertical Scaling

Vertical scaling gives one machine more capacity; horizontal scaling spreads work across machines. The right choice depends on bottlenecks, state, failure tolerance, and operational cost.

Ekanatha Reddy Urivakili8 min1,560 words

The short version

Scaling is the work of meeting more demand while keeping latency, reliability, and cost within acceptable limits. The two basic directions are:

  • Vertical scaling (scale up): give a machine more resources, such as CPU, memory, or disk throughput.
  • Horizontal scaling (scale out): add machines or service instances and distribute work among them.
  • Neither is an automatic performance fix: more capacity helps only when it addresses the constrained resource.
  • Most production systems use both: scale individual nodes to a sensible size, then add nodes when demand exceeds that size or redundancy is required.

Choose based on measured bottlenecks and failure requirements—not on a belief that one direction is always cheaper, simpler, or more resilient.


What changes in each direction?

With vertical scaling, the application continues to run on the same logical machine. A larger instance can provide more cores, memory, or I/O capacity without requiring the application to coordinate multiple copies of itself. This can be an effective first step when a service is small, stateful, or difficult to distribute.

Vertical scaling has a hard ceiling. Hardware and cloud instance sizes are limited, and cost often increases nonlinearly with capacity. Additionally, putting all work on one big machine concentrates risk: if it fails, capacity instantly disappears.

With horizontal scaling, work is spread across multiple instances. A load balancer can send requests to healthy instances, and adding instances can increase aggregate capacity. However, this works well only when the workload supports distribution. Issues like shared state, uneven workload distribution, lock contention, and upstream dependency bottlenecks may prevent linear capacity gains.

AspectVertical ScalingHorizontal Scaling
Capacity changeIncreases resources per nodeIncreases number of nodes
ComplexityLower; no distributed coordination neededHigher; requires load balancing, coordination
Fault toleranceLow; a single node failure impacts availabilityHigher; instances can fail independently
Cost efficiencyPotentially costly at scale due to hardware pricingMore predictable costs, but overhead exists
LimitsHard ceiling due to hardware/cloud limitsScaling complexity limits (state, locking)
Use casesStateful, hard-to-distribute workloadsStateless or partitioned scalable workloads

Capacity is bounded by the constrained component

A service handles requests through a chain of components: application CPU, memory, network, databases, caches, message queues, and external dependencies. System capacity is only as large as the component with the least available headroom. Adding more application nodes without addressing a saturated database or external service leads to contention but not increased throughput.

Key indicators to measure

  • Demand: request rate (RPS), concurrent requests, queue depth, workload mix.
  • Latency: especially tail latencies (p95, p99) to catch worst cases, not just averages.
  • Saturation: CPU, memory pressure, disk & network I/O, database lock or connection waits.
  • Errors: timeouts, retries, throttling, rejection rates.
  • Cost & headroom: resource use at peak and ramp-up duration for capacity changes.

A rough capacity estimate starts with benchmarking one instance at the target load and latency. For example, if one instance sustains 100 RPS at <200ms p99 latency, you should not assume 4 instances handle 400 RPS without verifying shared dependencies' load.

Minimal runnable example: estimate application capacity

const double measuredRpsPerInstance = 100;
const double targetUtilization = 0.65;
const int instanceCount = 4;

var estimatedSafeRps = measuredRpsPerInstance
    * targetUtilization
    * instanceCount;

Console.WriteLine($"Estimated target: {estimatedSafeRps:F0} requests/second");

Notice how this applies headroom (targetUtilization) to measured per-instance throughput, guiding capacity planning rather than promising linear scaling.


Horizontal scaling depends on how work and state are handled

Stateless request handling is the simplest case for scaling out: any healthy instance can process the next request, because request-specific state is externalized. Sessions, uploads, and user state require a shared data store accessible to all instances.

Background jobs require similar considerations. Multiple workers can consume from a queue only if the workload is idempotent or proper locking/partitioning mechanisms exist to prevent duplicated or conflicting work. Opening a large connection pool in many workers can saturate DB connections without increasing throughput.

Key practice points:

  • Store durable or session state outside individual instances.
  • Design jobs and request handlers to be safe for retries or duplicates.
  • Limit per-instance concurrency and connection pools to keep dependencies stable.
  • Ensure load balancers, queue partitioning, and databases can support increased concurrency.

Decision flow for scaling

flowchart TD
    D["Demand rises"] --> M["Measure latency, queue, and saturation"]
    M --> B{"Is the bottleneck identified?"}
    B -->|No| I["Instrument and test the system"]
    I --> M
    B -->|Yes| H{"Can this resource scale vertically?"}
    H -->|Yes| V["Increase node capacity"]
    H -->|No| P{"Can work be partitioned or replicated?"}
    P -->|Yes| N["Add nodes and distribute work"]
    P -->|No| R["Reduce work or redesign the constraint"]
    V --> C["Validate latency, errors, and cost"]
    N --> C
    R --> C

This loop emphasises iterative measurement and validation over guesswork, ensuring scaling targets actual bottlenecks.


Autoscaling is a control loop, not a capacity plan

Autoscaling adjusts capacity dynamically by observing metrics and changing the number or size of instances, usually guided by a target per-instance load metric or queue length.

Key challenges:

  • Signal noise and transient spikes.
  • Balancing cooldown periods and scaling speed.
  • Startup/warm-up times leading to capacity lag.
  • Constraints such as min/max instances or budget.
  • Avoiding oscillations by smoothing and stabilization.

Realistic calculation example: bounded replica recommendation

var sample = new CapacitySample(
    CurrentReplicas: 6,
    ObservedRpsPerReplica: 145,
    TargetRpsPerReplica: 100,
    MinReplicas: 3,
    MaxReplicas: 20);

var desiredReplicas = CapacityPlanner.Recommend(sample);
Console.WriteLine($"Recommended replicas: {desiredReplicas}");

public sealed record CapacitySample(
    int CurrentReplicas,
    double ObservedRpsPerReplica,
    double TargetRpsPerReplica,
    int MinReplicas,
    int MaxReplicas);

public static class CapacityPlanner
{
    public static int Recommend(CapacitySample sample)
    {
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(sample.CurrentReplicas);
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(sample.ObservedRpsPerReplica);
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(sample.TargetRpsPerReplica);
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(sample.MinReplicas);

        if (sample.MaxReplicas < sample.MinReplicas ||
            sample.CurrentReplicas < sample.MinReplicas ||
            sample.CurrentReplicas > sample.MaxReplicas)
        {
            throw new ArgumentOutOfRangeException(nameof(sample));
        }

        var ratio = sample.ObservedRpsPerReplica / sample.TargetRpsPerReplica;
        var rawTarget = (int)Math.Ceiling(sample.CurrentReplicas * ratio);
        return Math.Clamp(rawTarget, sample.MinReplicas, sample.MaxReplicas);
    }
}

Note the careful bounding to prevent scale extremes, and that production systems should incorporate smoothing and cooldowns.


Database scaling deserves separate treatment

Databases commonly limit overall system throughput. Adding more app instances leads to more simultaneous queries, which can saturate connections, locks, or I/O even if CPUs are idle.

Before scaling the application layer horizontally, inspect:

  • Query plans and indices to optimize performance.
  • Lock contention and transaction durations.
  • Connection pool saturation.
  • Application retry logic and cascading effects.

Database vertical scaling buys more resources without application redesign but has limits and high cost. Horizontal database techniques like read replicas, sharding, or partitioning entail complexity in routing and consistency guarantees. Each technique requires validation against a clear bottleneck.


Common failure modes and trade-offs

Failure ModeCauseMitigation
Adding replicas to bottleneckSaturated DB, queue partition, API limitOptimize dependencies, add sharding, or rate-limit concurrency
Assuming replicas fully interchangeableLocal state tied to instance (sessions, files)Externalize state, use shared or replicated storage
Scaling on noisy signalsShort-term spikes or out-of-date metricsUse moving averages, cooldown periods, and exclusion windows
Ignoring cold start latencyNew instances take time to warm caches or trafficPre-warm instances or keep buffer capacity
Ignoring failure domainsMultiple replicas in the same zone or hostSpread instances across failure domains
Over-reliance on vertical scalingHitting hardware limits unexpectedlyDefine ceilings, plan horizontal fallback
Premature horizontal scalingAdd complexity before need arisesProfile state sharing and workload partitioning first

A practical decision process

  1. Define the objective. Determine workload and latency targets, error and availability thresholds, and budget constraints.
  2. Identify limiting resources. Use telemetry and load tests under realistic mixes.
  3. Try the least disruptive change. Vertical scale for node-level limits or optimize code before scaling out.
  4. Evaluate workload distribution. Assess statefulness, coordination needs, shared dependencies, and failure isolation.
  5. Test end-to-end. Validate latency, errors, throughput, recovery, and cost after scaling changes.
  6. Have a fallback and ceiling. Define overload behaviors, maximum spend, and downgrade paths for overload.

Vertical scaling is often simpler and faster but limited in scope; horizontal scaling provides higher aggregate capacity and fault tolerance but at cost of complexity.


Observability for scaling and capacity management

Accurate scaling decisions demand robust observability, covering logging, metrics, and tracing.

What to log

  • Scaling decisions and actions: reason, metric values, capacity before/after.
  • Resource saturation events (CPU, memory, disk, network).
  • Latency outliers and error details.
  • Load balancer request distribution and failures.
  • Dependency errors and retries.

Example structured log snippet:

{
  "timestamp": "2024-06-01T15:23:10Z",
  "level": "info",
  "event": "ScalingAction",
  "scaleType": "Horizontal",
  "previousReplicas": 5,
  "newReplicas": 7,
  "reason": "ObservedRpsPerReplica exceeded target",
  "observedRpsPerReplica": 145,
  "targetRpsPerReplica": 100
}

Key metrics to collect

Use the RED and USE frameworks to cover service and infrastructure health:

Metric CategoryMetrics ExamplesPurpose
RED (Request, Error, Duration)Request rate (RPS), error rate (%), request duration (p95, p99)Track workload, errors, latency
USE (Utilization, Saturation, Errors)CPU %, memory %, disk queue length, DB connection saturationMonitor resource consumption

Distributed tracing with OpenTelemetry

Instrument the application with OpenTelemetry to capture end-to-end latency and error propagation, especially across service and database calls.

Example for a C# HTTP service:

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddJaegerExporter()  // or other exporters
    .Build();

// Use the tracerProvider for startup diagnostics and trace correlation.

This helps identify hotspots and verify that scaling actions affect the entire request path.

SLI/SLO examples for a scalable service

SLITarget (SLO)Alert (SLI violation)Action
Request success rate> 99.9%< 99.5% sustained for 5 minPager on-call, investigate errors
P99 request latency< 500ms> 700ms sustained for 5 minCheck capacity, latency causes
CPU utilization< 70% on average> 85% sustained for 5 minTrigger autoscale
DB connection pool saturation< 80%> 90% sustained for 3 minAlert DB team, apply throttling
Replica countWithin min/max boundsOutside configured replica limitsFail-safe autoscaler action

Alerts worth paging on

  • Unexpected drop in replica count or failed scaling operations.
  • Sudden spikes in error rates or timeouts.
  • Persistent resource saturation causing latency degradation.
  • Backend database or external third-party service failures affecting throughput.
  • Significant rollback or scaling flaps indicating instability.

Summary

Scaling is fundamentally about removing constraints while maintaining availability, reliability, and cost efficiency. Vertical and horizontal scaling are complementary:

  • Use vertical scaling to eliminate node-level capacity bottlenecks quickly.
  • Use horizontal scaling to increase aggregate capacity, add redundancy, and improve fault tolerance.
  • Always base decisions on measured bottlenecks, workload characteristics, and failure-domain awareness.
  • Adopt autoscaling as a control mechanism rather than a capacity plan, ensuring smoothing and limits.
  • Pay particular attention to shared dependencies like databases to avoid unintended bottlenecks.
  • Maintain detailed observability to validate assumptions, understand scaling impact, and alert on failures.

By following an iterative, data-driven process, you can reliably grow your system capacity and meet your service objectives without unnecessary complexity or cost.


Additional references


This detailed guide aims to help senior engineers make confident capacity decisions, balancing engineering effort, cost, and resilience in scalable systems.