Latency vs Throughput

Latency is how fast a car travels. Throughput is how many cars pass the toll booth. If you confuse them, you'll optimize the wrong thing.

What you will learn

• Decouple Time (Latency) from Volume (Throughput)
• Understand Bandwidth as the physical limit
• Use Little's Law to calculate server capacity
• Learn why 'Average Latency' is a lie and P99 is the truth

Latency vs Throughput

In the previous chapter (Performance vs Scalability), we learned that "Performance" is about speed. But "Speed" is a vague word.

When a user says "The system is slow", they could mean two completely different things:

1. "I clicked the button and it took 5 seconds to load." (Latency)
2. "I tried to upload a file and it failed because too many people are using it." (Throughput)

To act like an architect, you must stop saying "Speed" and start distinguishing between the Time it takes and the Volume you carry.

1. The Three Metrics

You cannot design a system without understanding the relationship between these three.

1. Latency (Time)

"How long does it take for a packet to get from A to B?"

• Unit: Milliseconds (ms).
• Goal: Lower is better.

2. Throughput (Flow)

"How many items are successfully processed per second?"

• Unit: Requests Per Second (RPS) or Bits Per Second (bps).
• Goal: Higher is better.

3. Bandwidth (The Tube Size)

"What is the theoretical maximum throughput?"

• Unit: Bits Per Second (bps).
• Insight: Throughput can never exceed Bandwidth.

Key Difference: Bandwidth is the size of the pipe. Throughput is how much water is actually flowing through it.

2. The Highway Analogy

Imagine a highway connecting San Francisco to Los Angeles.

• Latency: The time it takes to drive (e.g., 6 hours).
• Bandwidth: The number of lanes (e.g., 4 lanes).
• Throughput: The number of cars arriving in LA per hour.

flowchart LR
    subgraph Latency["Latency = Travel Time"]
        CityA[SF] -->|Car travels 65mph| CityB[LA]
    end

    subgraph Bandwidth["Bandwidth = Number of Lanes"]
        L1[Lane 1]
        L2[Lane 2]
        L3[Lane 3]
    end

Scenario A: Empty Road (3 AM)

• Latency: Low (Fast trip).
• Throughput: Low (Few cars).
• Status: Great experience for the few users.

Scenario B: Rush Hour (5 PM)

• Latency: High (Traffic jam).
• Throughput: Maximum (The road is full).
• Status: The system is processing as much as it can, but individual experience suffers.

The Tuning Trap: Often, to increase Throughput (fitting more cars on the road), you inadvertently harm Latency (cars move slower).

3. Little's Law: The Capacity Formula

This is one of the few equations you actually need to memorize. It helps you decide how many servers you need.

$ Concurrency = Throughput \times Latency $

(In academic terms: $L = \lambda W$)

The Real World Problem

You are building an API.

• You expect 1,000 requests per second (Throughput).
• Processing one request takes 0.5 seconds (Latency).

How many worker threads (or database connections) do you need active at any moment?

$ C = 1,000 \text{ req/s} \times 0.5 \text{ s} = 500 \text{ connections} $

Why does this matter? If your database only supports 200 max connections, your system will crash. To fix it, you have two choices:

1. Reduce Latency: Optimize queries to take 0.2s. ($1000 \times 0.2 = 200$)
2. Increase Capacity: Buy a bigger database that supports 500 connections.

4. The "Average" Lie

If you are monitoring your system and you see "Average Latency: 100ms", you might think everything is fine. You are wrong. "Average" is widely considered the worst metric in system engineering.

The Power User Problem

Imagine 100 users:

• 99 users: Load the page in 10ms.
• 1 user: Loads the page in 10,000ms (10 seconds) because they have a huge account history.

Average: ~110ms. (Looks great!) Reality: Your most important user (the one with the most data) is furious.

Enter Percentiles (P99)

Instead of averages, we use Percentiles.

• P50 (Median): 50% of users are faster than this. (The "Normal" experience).
• P95: 95% of users are faster than this. (The "Slow" experience).
• P99: 99% of users are faster than this. (The "Worst Case").

Amazon's Rule: Amazon optimizes for P99.9. They found that the 0.1% of users with the slowest load times were actually their highest spenders (people with huge shopping carts).

5. Summary

The Architect's Takeaway

1. Don't optimize blindly. Ask specifically: "Do we need to lower Latency (make it faster) or increase Throughput (handle more users)?"
2. Use Little's Law to sizings. Don't guess how much RAM/Connections you need. Calculate it.
3. Kill the Average. Set your alarms on P95 or P99 latency. That's where the real problems hide.