CPU Limits and throttling: what a CPU limit really does

limits.cpu: 1 does not mean one core

Assume:

resources:
  limits:
    cpu: "1"

The natural interpretation:

the container gets one CPU

That is not exactly what happens.

The kernel does not have to assign the container to:

CPU 4

and keep it there permanently.

A limit defines the maximum amount of CPU time a cgroup may consume within a given time window.

So it is more like:

you may consume the equivalent of one CPU

than:

you may only execute on one specific CPU

That is a meaningful difference.

How limits.cpu turns into quota

The Kubernetes limit:

resources:
  limits:
    cpu: 500m

does not reach the kernel as the value 500m.

The kernel works with CPU time.

In cgroup v2 the CPU limiting mechanism lives in:

cpu.max

The format looks like this:

MAX PERIOD

Both values, when MAX is a number, are expressed in microseconds.

The relationship can be simplified to:

quota = CPU limit x period

If the period is:

100 ms

then for:

limit = 0.5 CPU

we get:

quota = 0.5 x 100 ms
      = 50 ms CPU time

That is:

limit  = 500m
period = 100 ms
quota  = 50 ms

In cpu.max this will be represented as:

50000 100000

that is:

quota  = 50 000 us
period = 100 000 us

So the general relationship is very simple:

CPU limit = quota / period

and the other way round:

quota = CPU limit x period

With a 100 ms period:

limit      quota

250m   ->   25 ms
500m   ->   50 ms
1 CPU  ->  100 ms
2 CPU  ->  200 ms
4 CPU  ->  400 ms

At first glance:

quota  = 200 ms
period = 100 ms

may look odd.

How can you consume 200 ms of time within 100 ms?

Through parallelism.

Two threads executing simultaneously for 100 ms:

CPU 0: 100 ms
CPU 1: 100 ms

consume a total of:

200 ms CPU time

even though only:

100 ms wall-clock time

has passed.

That is why a 2 CPU limit can be represented as:

quota  = 200 ms
period = 100 ms

So limit is an abstraction of the Kubernetes API.

quota and period are the mechanism through which the kernel enforces that value.

And what if there is no CPU limit?

Quota does not have to be finite.

In cgroup v2 cpu.max may look like this:

max 100000

max is a special value meaning:

no maximum quota

So the CPU bandwidth controller does not impose a hard CPU-time limit on that cgroup.

Compare:

50000 100000

which means:

quota  = 50 ms
period = 100 ms
limit  = 0.5 CPU

with:

max 100000

which means:

no CPU bandwidth limit

In the second case the workload will not be throttled because of exhausted quota in cpu.max.

It may of course fail to get CPU for other reasons - for example because other runnable tasks also want to execute - but that does not come from a CPU limit.

CPU time can be consumed in parallel

This is the most important detail of the whole mechanism.

Assume a limit of:

1 CPU

and a period of:

100 ms

So we have:

quota = 100 ms CPU time

The container has one CPU-bound thread.

It may look like this:

0 ms ----------------------------- 100 ms

CPU:
[========== 100 ms ===============]

The quota lasts for the whole period.

But now the application has four runnable threads.

If the scheduler executes them in parallel on four CPUs:

CPU 0: [ T1 ]
CPU 1: [ T2 ]
CPU 2: [ T3 ]
CPU 3: [ T4 ]

every millisecond of wall-clock time may cost about:

4 ms CPU time

So the quota of:

100 ms

may be consumed after about:

25 ms wall-clock time

Then:

0 ms       25 ms                         100 ms
|-----------|-------------------------------|
   running             throttled

Four threads:

4 x 25 ms = 100 ms CPU time

consumed the entire budget corresponding to a 1 CPU limit.

That is exactly why:

limit = 1 CPU

does not mean:

one thread runs along quietly the whole time

A multithreaded workload can burn through the whole budget very quickly.

What happens once the quota is used up

If a cgroup consumes its available bandwidth before the period ends, the kernel throttles it.

That means the runnable tasks belonging to that cgroup cannot continue executing on a CPU even though they have work.

They wait until bandwidth becomes available again.

Example:

period = 100 ms
quota  = 50 ms

The workload uses all:

50 ms CPU time

before the period ends.

Simplified:

|---------- period: 100 ms ----------|

| running 50 ms | throttled 50 ms   |

At the start of the next period the workload receives CPU bandwidth again.

So the cycle may look like this:

RUN -> THROTTLE -> RUN -> THROTTLE

A CPU limit is a hard bandwidth cap, enforced by the kernel once the available budget is used up.

Free CPU on the node changes nothing

This often surprises people.

Assume a node:

32 CPU

Current utilisation:

3 CPU

So a huge part of the machine is free.

But the container has:

limits:
  cpu: "1"

If it uses up its CPU bandwidth, the kernel may throttle it even though right next to it there are:

29 free CPUs

Why?

Because a limit does not describe a current shortage of the resource.

It describes the maximum bandwidth allowed for that cgroup.

That is the fundamental difference between:

CPU contention

and:

CPU throttling

Under throttling the CPU may physically be available.

The process cannot use it because it has exhausted the budget derived from its limit.

A 500m limit

Assume:

resources:
  limits:
    cpu: 500m

With a 100 ms period:

quota  = 50 ms
period = 100 ms

So the workload may on average consume:

50 ms CPU time / 100 ms wall time
=
0.5 CPU

That does not necessarily mean:

RUN 50 ms
WAIT 50 ms

The kernel accounts CPU time for the whole group.

So it may be:

1 thread x 50 ms

or:

2 threads x 25 ms

or:

4 threads x 12.5 ms

In every case the total is:

50 ms CPU time

How the quota gets consumed depends on when the tasks are runnable and how many of them execute in parallel.

The limit is accounted for the cgroup, not a single thread

Assume a container with:

limit = 1 CPU

and eight workers:

T1
T2
T3
T4
T5
T6
T7
T8

They do not each get:

1 CPU

They share their cgroup's budget.

If many of them execute in parallel, together they consume the quota.

This matters for multithreaded applications: the limit applies to the whole workload covered by a given cgroup, not to every thread separately.

How to see throttling

A cgroup exposes statistics in:

cpu.stat

In cgroup v2 we can see, among others:

nr_periods
nr_throttled
throttled_usec

nr_periods says how many enforcement periods have elapsed.

nr_throttled says in how many periods the group was throttled.

throttled_usec shows the cumulative throttling time.

For example:

usage_usec     245231843
user_usec      220012391
system_usec     25219452

nr_periods          10000
nr_throttled         3200
throttled_usec   48231000

The mere fact that:

nr_throttled > 0

does not yet mean there is a problem.

It does tell you that the workload really was reaching the configured CPU bandwidth limit.

Request and limit do two different things

After the previous article we can put these mechanisms side by side without mixing up their meanings.

Request:

requests.cpu
     |
     v
declared demand
+
relative CPU weight

Limit:

limits.cpu
     |
     v
maximum CPU bandwidth
     |
     v
quota / period
     |
     v
throttling once the budget is used up

The most important distinction:

request:
how to treat the workload during placement
and competition for CPU

limit:
how much CPU time the workload
is allowed to consume at most

Mental model

For:

resources:
  limits:
    cpu: "1"

do not think:

the container has one core

Think:

the container has a CPU-time budget,
accounted over consecutive periods

If the budget is used up:

the task still has work
        |
        v
the task is runnable
        |
        v
quota exhausted
        |
        v
THROTTLED

And if:

cpu.max = max 100000

then there is no finite quota the workload could exhaust.

And that is the essence of a CPU limit:

a CPU limit does not determine where execution happens.

a CPU limit determines maximum CPU bandwidth.