NUMA, SMT and cache: when hardware topology starts to matter

A logical CPU does not mean a physical core

On a machine with SMT the system may see:

16 physical cores
32 logical CPUs

Each physical core may expose two logical CPUs:

Physical Core 0
├── CPU 0
└── CPU 16

For the Linux scheduler those are two separate places to execute.

So we can have:

Task A -> CPU 0
Task B -> CPU 16

at the same time.

But both tasks still run on the same physical core.

That means they share part of the hardware resources:

execution units
load/store resources
branch prediction
cache
frontend

That is why:

2 logical CPUs
!=
2 full physical cores

SMT contention

Assume two identical workloads.

Case A:

Core 0 -> workload A
Core 1 -> workload B

Case B:

Core 0
├── logical CPU -> workload A
└── logical CPU -> workload B

In both cases the scheduler sees two logical CPUs.

But performance can be completely different.

If both workloads use the same execution resources intensively, they will start getting in each other's way on SMT siblings.

What matters is that both can still be:

running

They are not waiting in a runqueue.

They are not being throttled.

CPU usage looks fine.

And yet throughput drops.

That is a different kind of contention from the classic:

more runnable tasks than CPUs

Here the problem is not a shortage of scheduler time.

The problem is sharing the physical resources of a core.

CPU time is not a unit of work done

Assume a task executing for:

100 ms CPU time

With an empty SMT sibling it may execute in that time:

1 000 000 operations

With a heavily loaded sibling:

700 000 operations

In both cases CPU accounting will show:

100 ms

But real performance is different.

That is a very important conclusion:

CPU time
!=
a fixed amount of work

And that is why a millicore is not a unit of performance.

Cache locality

A CPU does not read every value straight from RAM.

Between the processor and memory sits a cache hierarchy:

L1
L2
L3 / LLC
RAM

The closer the data sits to the CPU, the cheaper the access.

If a task runs on one CPU and repeatedly uses the same data, some of it may sit in cache.

Now the scheduler migrates the task:

CPU 2 -> CPU 11

The new CPU may not have that same data in its local caches.

So the task starts rebuilding locality.

As a result:

scheduler migration
        |
        v
cache misses
        |
        v
more access to lower cache levels / RAM
        |
        v
higher latency

The migration itself does not have to be expensive.

The cost comes from losing data locality.

Two workloads can compete for cache

Assume two processes using the same shared LLC.

Each has a working set of:

20 MB

And the available shared cache has:

24 MB

Separately each workload may perform very well.

Together:

20 MB + 20 MB > 24 MB

they start evicting each other's data.

The number of:

cache misses

grows, and more requests go to RAM.

The CPU may still look entirely busy.

Both processes are running.

There is no throttling.

And performance drops.

That is another example of a situation where:

CPU capacity looks fine

but:

hardware locality is bad

NUMA

On larger servers RAM is not uniform either.

A typical topology may look like this:

NUMA node 0
├── CPU 0-15
└── Memory A

NUMA node 1
├── CPU 16-31
└── Memory B

CPUs belonging to node 0 have faster access to node 0's memory.

If a CPU from node 0 reaches into node 1's memory, we have:

remote memory access

The data has to travel across the interconnect between NUMA nodes.

That usually means:

higher latency
lower bandwidth
higher cost of communication

CPU may be pinned well and performance may still be bad

Assume a workload executing only on:

CPU 2-5

all of which belong to:

NUMA node 0

That looks sensible.

But the application's memory was allocated mostly on:

NUMA node 1

So we have:

CPU -> node 0
RAM -> node 1

Every memory-intensive access is remote.

The CPU placement is correct.

The memory placement is not.

That is why, for memory-intensive workloads, pinning CPUs alone does not guarantee good performance.

First touch

Linux often uses a rule called:

first touch

A physical page of memory is usually placed close to the CPU that actually starts using it first.

Assume:

thread init -> NUMA node 0

initialises a large structure:

50 GB

Later the worker threads execute on:

NUMA node 1

We may end up with:

workers -> node 1
memory  -> node 0

The application does not have to know it has just created an enormous amount of remote memory traffic.

How these three things connect

Imagine a workload with:

4 CPU

On paper it looks simple.

But two placements are possible.

The good one

NUMA node 0

Core 0 -> workload
Core 1 -> workload
Core 2 -> workload
Core 3 -> workload

Memory -> node 0

We have:

no SMT sibling contention
good cache locality
local memory

The worse one

NUMA node 0

Core 0:
  sibling 1 -> workload
  sibling 2 -> another workload

Core 1:
  sibling 1 -> workload
  sibling 2 -> another workload

Memory -> NUMA node 1

Formally the workload may still have:

4 logical CPUs

but the real execution characteristics are much worse.

Why Kubernetes cannot fully hide the hardware

Kubernetes works with a resource abstraction:

CPU
memory

That is enough for most workloads.

But the Kubernetes scheduler does not interpret every:

1000m

as a guarantee of identical compute power.

At the hardware level:

CPU A != CPU B

if they differ in:

SMT sibling load
cache locality
NUMA locality

That does not mean the Kubernetes model is bad.

It is simply an abstraction.

And every abstraction has a boundary.

When topology starts to really matter

Not every workload should worry about NUMA and SMT.

For a typical lightweight web service the differences may be small.

It matters more for workloads that are:

CPU-bound
memory-bound
low-latency
large JVMs
in-memory databases
packet processing
HPC
ML inference

The more performance depends on individual microseconds, cache misses and memory bandwidth, the more important the physical placement of resources becomes.

Mental model

So far we could think:

workload
   |
   v
CPU time

Now the model has to be extended:

workload
   |
   v
logical CPU
   |
   v
physical core
   |
   +--> SMT sibling
   |
   +--> cache
   |
   v
NUMA node
   |
   v
memory

Every one of those levels can influence real performance.

Most importantly:

1 CPU

means an amount of capacity in the scheduler's model.

It does not mean:

a fixed number of instructions per second

Because real performance also depends on:

who you share a physical core with,
what cache locality looks like,
and how far away the memory is.

That is the moment where the clean CPU-time abstraction ends and the physics of the machine begins.