Noisy neighbor and CPU overcommit: when one Pod breaks another

What CPU overcommit is

Assume a node:

16 CPU

We have four workloads:

A request = 2 CPU
B request = 2 CPU
C request = 2 CPU
D request = 2 CPU

The scheduler sees:

SUM(requests) = 8 CPU

that is, only half of the node's capacity.

But each workload may in reality burst up to:

8 CPU

if it has no limit.

The theoretical maximum demand:

4 x 8 CPU = 32 CPU

on a machine with:

16 CPU

That is overcommit.

We did not physically allocate:

32 CPU

We simply assume:

not everyone will need their maximum at the same time

Overcommit on its own is not a problem

Assume:

A uses 6 CPU
B uses 1 CPU
C uses 0.5 CPU
D uses 1 CPU

In total:

8.5 CPU

on a node with:

16 CPU

Everything works fine.

Workload A uses far more than its:

request = 2 CPU

and that is perfectly normal.

It is simply using the idle capacity the others left behind.

Overcommit is therefore a mechanism that lets us increase infrastructure utilisation.

Without it we could have:

requests = 16 CPU
actual usage = 5 CPU

and half the machine sitting idle.

The problem starts with a simultaneous peak

Now every workload suddenly needs:

8 CPU

Demand:

A = 8
B = 8
C = 8
D = 8

SUM = 32 CPU

Capacity:

16 CPU

We have:

demand = 2 x capacity

All that work cannot be executed at once.

Tasks start waiting in the runqueue.

We get:

CPU contention

and that is exactly when the CPU weights of the individual workloads start to matter.

A request starts acting as a negotiating position

Assume two Pods:

Pod A request = 1 CPU
Pod B request = 4 CPU

Both have no limit and both want to use:

8 CPU

As long as the node has plenty of free CPU, both may get more than their request.

But once the processor starts running short, their relative weights derived from the requests start to matter.

Simplified:

A : B
1 : 4

So Pod B has a much stronger position under contention.

That means an understated request is not only a kube-scheduler problem.

It may also mean:

a smaller share of CPU
when the node is overloaded

Noisy neighbor

Assume a node:

8 CPU

Two workloads run on it.

API

request = 4 CPU
normal usage = 3 CPU

Batch

request = 500m
normal usage = 200m

In the normal situation:

API   = 3 CPU
Batch = 0.2 CPU

There is no problem.

Now batch starts intensive processing and wants:

8 CPU

Total demand:

API wants   4 CPU
Batch wants 8 CPU

SUM = 12 CPU

on a node with:

8 CPU

Batch becomes a noisy neighbor.

Not because it is doing anything "illegal".

If it has no limit, it may try to use all the available capacity.

The problem is that its burst starts affecting the API's latency.

Why a noisy neighbor can be hard to spot

We look at the API:

CPU usage = 3 CPU

Normally:

CPU usage = 3 CPU

During the problem:

CPU usage = 3 CPU

At first glance nothing changed.

But the API may now want:

5 CPU

and only be getting:

3 CPU

The missing execution time materialises as:

runnable waiting

That is:

PSI rises
runqueue latency rises
application latency rises

The application's CPU usage itself may not go up at all.

That is why a noisy neighbor is often misdiagnosed as:

the application slowed down by itself

Understated requests lead to aggressive bin packing

Assume a node:

16 CPU

We have 16 Pods.

Each really needs, during normal traffic:

1 CPU

but declares:

request = 250m

The scheduler sees:

16 x 0.25
=
4 CPU

So the node looks very loosely filled.

It may therefore accept more Pods.

The real demand, however, is already:

16 CPU

that is, the whole capacity.

Every further burst leads straight to contention.

In such a system the problem is not that Linux schedules badly.

The problem was created earlier:

the request does not describe the real demand

Overstated requests have the opposite cost

We can also overshoot in the other direction.

The workload really uses:

500m

but has:

request = 4 CPU

Ten such workloads:

real usage = 5 CPU
requests   = 40 CPU

The scheduler needs far more nodes than the real usage would suggest.

We get:

poor bin packing
low utilisation
higher cost

That is why a correct request is not:

as small as possible

nor:

as large as possible, to be safe

It is an attempt to describe the workload's real CPU demand in a way that makes sense for that specific kind of application.

Batch and a latency-sensitive workload on one node

This is the classic noisy neighbor case.

Assume:

API:
request = 4 CPU
no limit

Batch:
request = 1 CPU
no limit

Node:

8 CPU

Normally:

API   = 3 CPU
Batch = 1 CPU

That leaves:

4 CPU idle

Batch can use that capacity.

That is good.

The problem starts when:

API wants = 6 CPU
Batch wants = 8 CPU

Both workloads try to burst.

Demand:

14 CPU

Capacity:

8 CPU

The scheduler has to divide the processor.

Batch may still do a huge amount of work while at the same time increasing the API's scheduler wait.

That is the typical conflict:

throughput workload
vs
latency-sensitive workload

Does a CPU limit solve noisy neighbor?

It may.

Assume:

Batch:
request = 1 CPU
limit   = 2 CPU

Now batch cannot use:

8 CPU

even if it wants to.

Its maximum bandwidth:

2 CPU

That protects the other workloads from a very aggressive batch burst.

But we pay for it with another property:

if the node has 6 CPU idle,
batch may still use at most 2 CPU

So a limit improves isolation at the cost of being able to use idle capacity.

That is a trade-off.

Without limits we use CPU more efficiently

Assume:

node = 16 CPU

The workloads normally need:

8 CPU

but one batch job can use an extra:

8 CPU

Without a limit:

normal:
8 CPU workload
8 CPU idle

batch active:
16 CPU used

Excellent hardware utilisation.

With a batch limit of:

2 CPU

during batch processing we may have:

10 CPU used
6 CPU idle

even though the application has work to do.

Limits give predictability.

No limits give flexibility.

CPU overcommit is therefore a capacity-planning decision

There is no universal value like:

good overcommit = 2:1

Everything depends on the characteristics of the workloads.

If you have 100 services that:

use 10% of their peak CPU most of the time

heavy overcommit may work very well.

If they all scale from the same event:

Black Friday
a cron at 00:00
a restart after a deploy
cache invalidation
a sudden traffic spike

their bursts are correlated.

Then the assumption:

not everyone needs CPU at the same time

stops working.

Correlated bursts

This is an especially important case.

Assume ten workloads.

Each with:

average = 500m
peak    = 2 CPU

If the peaks appear randomly, on average we can share capacity well.

But if they all react to the same traffic:

request enters frontend
       |
       v
service A
       |
       +--> B
       +--> C
       +--> D

a rise in traffic can increase the demand of the whole service graph at once.

Then:

peak A
peak B
peak C
peak D

are not independent.

Overcommit based only on average CPU may look excellent right up until real production traffic.

How to recognise a noisy neighbor

The typical picture:

application latency rises

but:

the application is not being throttled

At the same time:

CPU PSI rises

and on the node:

other workloads increase their CPU usage sharply

That is a signal:

the problem may be shared,
not local to a single Pod

What matters most is comparing the behaviour of:

the workload

with:

the rest of the node

If one Pod's problem overlaps in time with another one's large burst, we have a very strong lead.

Mental model

Overcommit:

SUM(possible CPU demand)
>
physical CPU capacity

is not a problem in itself.

The problem starts when, at the same time:

actual runnable demand
>
physical CPU capacity

That is when contention appears.

A noisy neighbor is a workload whose change in demand worsens the execution conditions of other workloads sharing the same resource.

Most importantly:

a CPU request

is not just a number needed to pass the admission scheduler.

It influences:

placement
bin packing
relative CPU weight
behaviour under contention

Understated requests let you use a cluster very efficiently, right up to the moment real demand arrives all at once.

And then overcommit stops being an optimisation.

It becomes a queue of runnable tasks.