Konrad Kowalski (rootsher)Principal Platform & Reliability Architect010000100001001001111000011011110001100101011000

CPU in Kubernetes from the kernel up

date
category
Capacity & Performance
also in
Containers
reading
1 min / 262 words

CPU in Kubernetes only looks simple at the YAML level.

yaml
resources:
  requests:
    cpu: 500m
  limits:
    cpu: "1"

As long as everything works, those two values seem to be enough.

The trouble starts when throttling shows up, p99 climbs, the workload uses less CPU than you expected, the node looks underloaded, and the application is still waiting for a processor.

To diagnose cases like that correctly, you have to go lower than Kubernetes.

Because in the end:

text
Pod
  |
  v
cgroup
  |
  v
Linux scheduler
  |
  v
logical CPU
  |
  v
physical core

Kubernetes does not execute your application's instructions, does not hand CPU time to threads, and does not implement processor scheduling.

Those mechanisms live lower down, in the Linux kernel and in the hardware itself.

This series builds the CPU model bottom-up: from runnable tasks and the scheduler, through cgroups, requests and limits, all the way to throttling, contention, PSI, topology and practical diagnostics.

The goal is not to teach Kubernetes from scratch.

The goal is to understand what actually happens between:

text
requests.cpu: 500m

and the moment a specific thread really executes instructions on a processor.

Because only then can you properly tell apart:

text
high CPU usage

CPU throttling

CPU contention

CPU pressure

bad resource sizing

bad placement

and stop treating:

text
CPU = 80%

as a diagnosis.

It is just one number describing one layer of a much larger mechanism.

If you have ever seen a Pod with seemingly fine CPU usage, no throttling, and a rising p99 at the same time, this series is about exactly that: how to get from such a symptom to a specific mechanism in the kernel.

No guessing. No reducing everything to "not enough CPU". No treating requests and limits as magic YAML fields.

We start at the lowest layer: how Linux really gives a process CPU.