PSI: how Linux measures real CPU pressure
- date
- category
- Observability
- also in
- Capacity & Performance
- reading
- 5 min / 954 words
In the previous article we arrived at this situation:
the task has work
|
v
the task is runnable
|
v
the CPU is executing other tasks
|
v
the task waits
That is CPU contention.
The problem with classic CPU usage is that it mostly shows the time a workload got.
It does not directly show the time a workload lost because it wanted to execute but had no access to a CPU.
Linux has a separate mechanism for that:
PSI
that is:
Pressure Stall Information
PSI measures the time during which tasks could not do work because a resource was unavailable. Linux exposes PSI for CPU, memory and I/O; here we care only about CPU.
CPU usage and CPU pressure answer different questions
Assume a workload:
CPU usage = 2 CPU
That number says:
the workload received about 2 CPU of execution time
It does not say whether it wanted:
2 CPU
or:
8 CPU
If the workload wanted 8 CPU but the node was overloaded and the scheduler gave it only 2 CPU, the difference materialises as runnable tasks waiting.
PSI tries to capture exactly that time.
So we can think:
CPU usage:
how much work the scheduler let me do
CPU pressure:
how often I had work
but a lack of CPU delayed it
These are two different axes for observing a system.
/proc/pressure/cpu
At the system level Linux exposes CPU PSI through:
/proc/pressure/cpu
A sample result:
some avg10=12.50 avg60=8.20 avg300=3.10 total=91827364
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
There are two basic concepts here:
some
full
and four values for each of them:
avg10
avg60
avg300
total
some
For CPU, some means the time during which at least one task was delayed by a lack of available CPU.
Assume four CPUs:
CPU 0 -> A
CPU 1 -> B
CPU 2 -> C
CPU 3 -> D
E -> runnable, waiting
The system is still doing work.
Four tasks are running.
But E also wanted a CPU and could not get one.
So we have CPU pressure:
some
This is very important.
PSI does not ask:
is the CPU doing anything?
It asks:
is a lack of CPU delaying some runnable workload?
How to read avg10
Assume:
some avg10=25.00
That does not mean:
CPU utilisation = 25%
Nor:
25% of tasks are waiting
It roughly means:
In the last 10-second window the workload experienced at least a partial CPU stall for about 25% of the time.
So, simplified:
10 seconds x 25%
=
2.5 seconds
for about 2.5 seconds of that window at least one task was waiting for CPU.
That is completely different information from CPU utilisation.
avg10, avg60, avg300
PSI reports three moving averages:
avg10
avg60
avg300
that is, trends over roughly:
10 seconds
60 seconds
300 seconds
Example:
some avg10=42.00 avg60=12.00 avg300=3.00
That suggests a sudden, fresh deterioration.
The last few minutes looked relatively calm:
avg300 = 3%
but the last dozen or so seconds:
avg10 = 42%
look completely different.
We can also have the opposite situation:
avg10=1.00
avg60=12.00
avg300=20.00
that is, pressure was high earlier but is falling right now.
Linux maintains these values as short-term, medium-term and longer pressure trends.
total
Next to the averages we have:
total
Example:
some ... total=91827364
total is the cumulative stall time expressed in microseconds.
It is not a moving average.
It is a counter that grows over time.
That lets us compute the increase between two moments.
Example:
t1:
total = 1000000
t2:
total = 1300000
The difference:
300000 us
=
300 ms
So between the measurements about 300 ms of extra CPU stall time accumulated.
total is especially useful if you want to build your own rates or observe short events that avg10 would smooth away.
And what is full?
PSI also has:
full
The general definition of full covers the situation where all non-idle tasks are stalled at the same time and the workload as a whole is doing no productive work.
For memory and I/O that is very useful information.
For CPU there is an important detail, though.
At the whole-system level:
/proc/pressure/cpu
CPU full has no sensible interpretation matching memory or I/O. The kernel documents that system-wide CPU full is reported as zero for compatibility reasons.
That is why a typical system-level result looks like:
some avg10=18.20 avg60=10.40 avg300=5.60 total=...
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
And that does not mean:
there is no serious CPU pressure
For system-wide CPU what matters most is above all:
some
PSI at the cgroup level
PSI does not have to be measured globally only.
In cgroup v2 every cgroup can have:
cpu.pressure
that is, pressure computed for the tasks belonging to that specific group.
Example:
/sys/fs/cgroup/.../cpu.pressure
may return:
some avg10=31.20 avg60=15.70 avg300=8.30 total=...
full avg10=7.50 avg60=3.20 avg300=1.10 total=...
That is far more interesting than the node's global pressure alone.
A node may have many workloads, but the problem may concern only a specific part of the cgroup hierarchy.
So we can ask:
does the whole host have CPU pressure?
and, separately:
does this particular workload experience CPU pressure?
some=0 versus some=50
Consider two cases.
Case A
CPU usage = 100%
CPU PSI some = 0%
The CPU works all the time.
But the runnable workload practically never waits for access to the processor.
That may be a perfectly healthy situation.
For instance one CPU-bound worker using all of its assigned CPU:
RUN RUN RUN RUN RUN
There is no queue of competing tasks.
Case B
CPU usage = 100%
CPU PSI some = 50%
The CPU also works all the time.
But for a significant part of that time at least one additional task wanted to execute and could not.
That is:
running:
A B C D
waiting:
E F G
Usage in both cases may look similar.
Pressure is completely different.
Why PSI is so useful
Classic utilisation answers this question very well:
how hard are we using the CPU?
But for capacity problems the more interesting question is:
is a lack of CPU already stopping the workload from doing work?
PSI answers that second one.
That lets us tell apart:
the CPU is busy
from:
the CPU is the resource limiting the workload
Those two things are not equivalent.
An example
Assume a node with:
8 CPU
and a workload generating exactly:
8 runnable CPU-bound threads
In the ideal case:
CPU 0 -> T1
CPU 1 -> T2
...
CPU 7 -> T8
CPU utilisation:
~100%
but few tasks have to wait.
Pressure may be small.
Now let us add another eight runnable threads:
T9 .. T16
Capacity stays at:
8 CPU
Demand grows to around:
16 CPU
CPU utilisation is still:
~100%
It cannot be 200% of the whole node, after all.
But now some tasks are constantly waiting.
That is exactly what we will see in a rising:
CPU PSI some
Usage has practically stopped being useful here for judging the scale of contention.
PSI still changes.
CPU PSI in Kubernetes
On modern Kubernetes, PSI can be collected by the kubelet at the level of:
node
pod
container
Kubernetes exposes that data through the Summary API and the kubelet metrics endpoint. In Kubernetes 1.36 kubelet PSI support is already a stable feature.
The mechanism itself still comes from Linux, though.
Kubernetes only collects and exposes the data produced by PSI in the kernel.
Mental model
CPU utilisation:
how much time the CPU spent doing work
CPU PSI:
how often a lack of CPU delayed runnable work
The key signal:
some
means:
for this fraction of the time
at least one task wanted CPU
but had to wait
Whereas:
avg10
avg60
avg300
show the pressure trend over different time windows, and:
total
is the cumulative stall time.
So PSI is not another utilisation metric.
It measures something far more interesting:
the time lost to CPU being unavailable