Why 1 CPU does not mean one core

CPU time and CPU placement are separate mechanisms

Assume a node:

CPU 0
CPU 1
CPU 2
CPU 3
CPU 4
CPU 5
CPU 6
CPU 7

The container has:

limit = 1 CPU

That does not mean:

container -> CPU 3

It may look like this:

t0: thread A -> CPU 2
t1: thread A -> CPU 5
t2: thread A -> CPU 1
t3: thread A -> CPU 7

The Linux scheduler may migrate a task between logical CPUs.

The limit still stays:

1 CPU

because it concerns total CPU time, not a processor identifier.

A multithreaded application shows the difference even better

Assume:

limit = 1 CPU

but the application has four runnable threads:

T1
T2
T3
T4

The kernel may execute them in parallel for a while:

CPU 0 -> T1
CPU 1 -> T2
CPU 2 -> T3
CPU 3 -> T4

If all four execute for:

25 ms

then the workload used:

4 x 25 ms
=
100 ms CPU time

So with a quota matching 1 CPU and a 100 ms period it just used up its entire budget.

That shows something fundamental:

limit = 1 CPU

does not mean:

at most one thread may be running

It means:

total CPU bandwidth matches one CPU

Logical CPU

When Kubernetes says:

1 CPU

it works with a unit matching a CPU as visible to the operating system.

On a typical x86 machine with SMT the system may see, for example:

16 physical cores
32 logical CPUs

Linux will number them roughly:

CPU 0
CPU 1
...
CPU 31

The scheduler works exactly on those logical CPUs.

That is why:

1 CPU

in Kubernetes does not automatically mean:

one physical core

This distinction becomes especially important with SMT, but at this stage it is enough to say:

Kubernetes CPU
~ logical CPU capacity visible to the system

So who decides where a task executes?

The Linux scheduler.

If a task may execute on many CPUs, the scheduler picks the specific processor.

It may also move the task later:

CPU 2 -> CPU 6

that is, perform a CPU migration.

There can be many reasons, for instance an attempt to balance runnable work across CPUs.

From the limit's point of view nothing special happened.

The task still consumes the same shared CPU-time budget of its cgroup.

The only thing that changed is:

where it was executed

CPU affinity

If we want to restrict a task to specific CPUs, we need a separate mechanism:

CPU affinity

We can say, for instance:

the task may execute only on CPU 2 and CPU 3

Conceptually:

allowed CPUs = {2,3}

The scheduler still decides:

CPU 2 or CPU 3?

but it cannot move the task to:

CPU 0
CPU 1
CPU 4
...

So affinity answers the question:

where may the task be executed?

Not:

how much CPU time may it consume?

That is yet another independent dimension.

cpuset

For groups of processes Linux has a controller:

cpuset

In cgroup v2 we may meet:

cpuset.cpus

For example:

2-3

means that tasks belonging to that cgroup are to be executed on CPUs from that set.

The kernel documents cpuset.cpus as the list of CPUs that tasks in a given cgroup may use; the effective set is additionally constrained by the hierarchy of parent cgroups.

Example:

cpuset.cpus = 4-5

gives:

T1 -> CPU 4
T2 -> CPU 5

but not:

T1 -> CPU 8

This is a real constraint on placement.

A limit and a cpuset can work at the same time

Assume a cgroup with:

cpuset.cpus = 4-5

and:

CPU limit = 1

So we have two independent constraints.

Placement

the workload may execute only on:

CPU 4
CPU 5

Bandwidth

the workload may on average consume
the equivalent of 1 CPU

So two threads may execute in parallel:

CPU 4 -> T1
CPU 5 -> T2

but together they still consume one CPU-time budget.

For example:

T1 = 50 ms
T2 = 50 ms

gives:

100 ms CPU time

A request does not define placement either

Likewise:

resources:
  requests:
    cpu: "1"

does not mean:

reserve CPU #7 for me

As we know from the previous articles, a request governs:

kube-scheduler resource accounting
+
relative CPU weight under contention

Not the location of a task on a specific processor.

That is why an ordinary Pod with:

requests:
  cpu: "1"

may execute its tasks on various CPUs of the node.

When does Kubernetes really assign specific CPUs?

That is what the kubelet's CPU Manager is for.

The default policy does not give ordinary Pods dedicated CPUs.

The policy:

static

however, makes it possible to assign exclusive CPUs to selected containers.

Simplified, containers belonging to Guaranteed Pods with integer CPU requests qualify for such an assignment. BestEffort, Burstable and Guaranteed workloads with fractional CPU requests stay in the shared CPU pool.

Example:

resources:
  requests:
    cpu: "2"
  limits:
    cpu: "2"

with the right CPU Manager configuration may end up with an assignment of:

CPU 6
CPU 7

for that container.

Then we really do enter the world of:

CPU placement

and not just:

CPU bandwidth

"Exclusive" also needs precision

Even the term:

exclusive CPU

does not unconditionally mean:

nobody outside my process will ever execute instructions there

The Kubernetes documentation points out that CPU Manager exclusivity concerns other Pods. System processes such as the kubelet or the container runtime may still do work on such CPUs.

So:

exclusive for workload placement

does not necessarily mean:

full CPU isolation from the whole operating system

Fuller processor isolation is a separate problem.

Fractional CPU and integer CPU

This distinction matters.

A request of:

500m

is fractional.

There is no such thing as:

half a logical CPU

that could be physically carved out through a cpuset.

That is why fractional CPU naturally works as:

time sharing

Example:

500m

may mean executing a task:

on CPU 1
then CPU 5
then CPU 2

with the matching share of CPU time.

Whereas:

2 CPU

may - with the right CPU Manager policy - be mapped onto two specific logical CPUs.

Three separate questions

It is best to split all the mechanisms we have met into three questions.

How much do I declare?

requests.cpu

Example:

500m

How much may I consume at most?

limits.cpu

Example:

1 CPU

Where may I execute?

affinity / cpuset

Example:

CPU 4-5

Those are three different configurations.

So we can have a workload with:

request = 500m
limit   = 1 CPU
cpuset  = CPU 4-5

which, simplified, means:

Kubernetes accounting:
0.5 CPU

CPU bandwidth:
at most 1 CPU

CPU placement:
only CPU 4 and 5

These values are not contradictory.

They describe different properties of execution.

Mental model

Do not think:

1 CPU
=
one specific core

Think:

CPU request / limit
=
an amount of CPU capacity / CPU time

Whereas:

CPU affinity / cpuset
=
the set of CPUs a task may execute on

That is why:

limits:
  cpu: "1"

does not pin a process to one CPU.

And:

cpuset.cpus = 4

does not automatically imply a limit of:

1 CPU bandwidth

One controls when execution happens.

The other controls where it happens.

Only when we deliberately combine these mechanisms can we talk about really assigning a workload to specific CPUs.