How to debug CPU in Kubernetes without guessing

1. First establish how much CPU the workload actually got

The basic metric is CPU usage.

In Prometheus that will very often be:

container_cpu_usage_seconds_total

It is a counter.

It does not show "current CPU" directly. It shows the cumulative CPU time consumed by the container.

If the counter grew by:

1.0

within one second, the workload used about:

1 CPU-second

that is, on average:

1 CPU

in that interval.

So we typically use:

rate(container_cpu_usage_seconds_total[5m])

If the result is:

0.5

the workload consumes on average:

0.5 CPU

that is:

500m

If it is:

2.4

then about:

2.4 CPU

That is the first fact we need:

how much CPU was actually executed?

2. Compare usage with the request and the limit, but do not draw conclusions yet

Assume:

request = 500m
limit   = 2 CPU
usage   = 1.4 CPU

That means:

usage > request

but still:

usage < limit

There is nothing suspicious about it.

The workload is simply bursting above its request.

A request is not a ceiling.

A far more interesting situation:

request = 500m
limit   = 1 CPU
usage   ~ 1 CPU

Here the workload may be hitting its limit.

But usage alone still does not tell us whether throttling is actually happening.

We have to go a level lower.

3. Check throttling

If the workload has a CPU limit, we check the cgroup statistics.

In cgroup v2:

cpu.stat

may contain:

usage_usec
user_usec
system_usec
nr_periods
nr_throttled
throttled_usec

The most important ones for throttling:

nr_throttled
throttled_usec

If:

nr_throttled

is growing, the workload regularly uses up its available CPU bandwidth.

If:

throttled_usec

is also growing quickly, the kernel really is blocking further execution because the quota is exhausted.

Conceptually:

task runnable
     |
     v
quota exhausted
     |
     v
throttled

That is a very concrete diagnosis.

Not:

CPU is probably too high

but:

the workload is being stopped by the CPU bandwidth controller

4. No throttling does not mean no CPU problem

Assume:

nr_throttled = 0

Can we say:

CPU is healthy

No.

The workload may have no limit:

cpu.max = max ...

and still suffer from contention.

Example:

node capacity = 8 CPU
runnable demand = 16 CPU

Nobody has to be throttled.

Some tasks are simply waiting in the runqueue.

That leads to the next question:

is the workload waiting for CPU?

5. Check CPU PSI

This is where:

cpu.pressure

comes in.

At the system level:

/proc/pressure/cpu

or at the cgroup level:

cpu.pressure

What we care about above all is:

some

Example:

some avg10=18.00 avg60=8.50 avg300=3.20 total=...

avg10=18 roughly means that for about 18% of the last short window at least one task experienced a CPU stall.

That is:

the workload had runnable work,
but did not get CPU right away

That is already a contention signal.

Usage and PSI answer different questions:

usage:
how much CPU did I get?

PSI:
how often did a lack of CPU block me?

6. Interpret usage and PSI together

The combinations are the interesting part.

High usage, low PSI

CPU usage = high
CPU PSI   = low

The workload uses CPU intensively but does not wait significantly for the scheduler.

That may be a healthy CPU-bound workload.

High usage, high PSI

CPU usage = high
CPU PSI   = high

The CPU is working hard and there is more runnable demand than available capacity.

That is classic contention.

Moderate usage, high PSI

This is the more interesting case.

It may mean the workload has a restricted set of CPUs, or that contention occurs locally even though the whole node does not look heavily loaded.

The node's average utilisation may then be misleading.

7. If PSI is rising, check runqueue latency

PSI says:

somebody is waiting

but it does not say exactly:

how long a specific task waits for the scheduler

For that we need to observe the scheduler.

The key events are:

sched_wakeup
sched_switch

Conceptually:

sched_wakeup
    |
    | the task becomes runnable
    |
    v
    ... waits ...
    |
    v
sched_switch
    |
    | the task becomes running

The time difference between those moments gives us:

runqueue latency

Example:

wakeup:  12:00:00.100
running: 12:00:00.107

that is:

7 ms scheduler wait

That is already a direct answer to the question:

is the application's latency rising because
the task is waiting for CPU?

8. kubectl top is only the beginning

The command:

kubectl top pod

is convenient.

But it mostly shows aggregated usage.

We may see:

NAME       CPU
api-123    780m

And that is all.

From that we do not know:

whether the Pod is being throttled
whether it is waiting in a runqueue
whether it has high PSI
whether usage is steady or bursty

So kubectl top is a good tool for quickly checking:

who is using CPU right now

but a poor tool for answering:

why does the application have a CPU problem

9. A minimal diagnostic workflow

Assume the symptom:

p99 latency went up

Do not start by changing requests or limits.

Go in order.

Step 1

Check:

CPU usage

Question:

how much CPU does the workload actually consume?

Step 2

Check:

request
limit

Question:

is usage approaching the limit?

Step 3

Check:

cpu.stat

Question:

is the workload being throttled?

If yes:

problem = bandwidth limit

Step 4

If there is no throttling, check:

cpu.pressure

Question:

are runnable tasks waiting for CPU?

If PSI is rising:

problem = contention

Step 5

If you need precision:

sched_wakeup
sched_switch

Question:

how long do tasks wait in the runqueue?

10. Example: two similar symptoms, two different causes

Workload A

usage = 1 CPU
limit = 1 CPU

nr_throttled growing fast
throttled_usec growing fast

CPU PSI = low

Diagnosis:

CPU throttling

The kernel is capping the workload to its configured bandwidth.

Workload B

usage = 1 CPU
limit = none

nr_throttled = 0

CPU PSI some = high
runqueue latency = high

Diagnosis:

CPU contention

The workload is not being capped by its own quota.

It does not get CPU because the scheduler has too much runnable work.

On a chart:

CPU usage = 1

both workloads may look identical.

The cause of the problem is completely different.

11. Do not start diagnostics from percentages

A popular dashboard:

CPU / request = 180%

or:

CPU / limit = 95%

is useful as a signal.

But it is not a diagnosis.

180% request may be perfectly healthy.

95% limit may work without any problem.

50% node CPU may coexist with local contention.

So good diagnostics does not ask:

what percentage does the dashboard show?

but:

did the process get CPU?

if not:
why?

Mental model

The simplest workflow:

symptom
   |
   v
CPU usage
   |
   v
request / limit
   |
   v
cpu.stat
   |
   +--> throttling?
   |        |
   |        v
   |   quota problem
   |
   v
CPU PSI
   |
   +--> pressure?
            |
            v
       contention
            |
            v
     runqueue latency

Every layer answers a different question:

usage:
how much CPU did I get?

cpu.stat:
did the kernel stop me through a limit?

PSI:
did a lack of CPU delay runnable work?

runqueue latency:
how long exactly did I wait?

Only after walking that path is it worth changing the configuration.

Because "a CPU problem" may mean several completely different mechanisms, and each of them needs a different fix.