---
title: profile_cpu
sidebar_position: 0
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# profile_cpu

The profile cpu gadget takes samples of stack traces.

## Requirements

- Minimum Kernel Version : *5.4

*This is the minimal kernel version we have tried for this Gadget, however it's possible that it works with earlier versions.

## Getting started

Running the gadget:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        $ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% --map-fetch-interval 0 [flags]
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ sudo ig run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% --map-fetch-interval 0 [flags]
        ```
    </TabItem>
</Tabs>

## Guide

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        Here we deploy a small demo pod "random":

        ```bash
        $ kubectl run --restart=Never --image=busybox random -- sh -c 'cat /dev/urandom > /dev/null'
        pod/random created
        ```

        Using the profile cpu gadget, we can see the list of stack traces.
        The following command filters only for pods named "random", execute the command
        and interrupt it after ~30 seconds. The `--kernel-stacks-only` option is passed to show only the
        kernel stack traces.

        ```bash
        $ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% --podname random --kernel-stacks-only --map-fetch-interval 0
        ```

        After a while press with Ctrl-C to stop trace collection

        ```
        $ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% --timeout 5 --podname random --kernel-stacks-only
        K8S.NODE                 K8S.NAMESPACE            K8S.PODNAME             K8S.CONTAINERNAME              PID COMM             SAMPLES              KERN_STACK
        minikube-docker          default                  random                  random                       38130 containerd-shim  5                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_block_generic;…
        minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…
        ```

        From the traces above, you can see that the pod is spending CPU time in the
        Linux function `urandom_read`.

        Instead of waiting, you can use the `--timeout` argument:

        ```bash
        $ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% --timeout 5 --podname random --kernel-stacks-only --map-fetch-interval 0
        K8S.NODE                 K8S.NAMESPACE            K8S.PODNAME             K8S.CONTAINERNAME              PID COMM             SAMPLES              KERN_STACK
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]_copy_to_iter; [1]get…
        minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  5                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
        minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]get_random_bytes_user…
        ```

        Finally, we need to clean up our pod:

        ```bash
        $ kubectl delete pod random
        ```
    </TabItem>
    <TabItem value="ig" label="ig">
        * Generate some kernel load:

        ```bash
        $ docker run -d --rm --name random busybox cat /dev/urandom > /dev/null
        ```

        * Start `ig`:

        ```bash
        $ sudo ig run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG%  --kernel-stacks-only --containername random --runtimes docker --map-fetch-interval 0
        ```

        * Observe the results:

        ```bash
        RUNTIME.CONTAINERNAME     KERNEL_IP            COMM                    PID        TID SAMPLES              KERN_STACK
        random                    18446603336363922440 cat                   83451      83451 2                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336357767960 cat                   83451      83451 1                    [0]rcu_all_qs; [1]__cond_resched; [2]get_random_bytes_user; [3]ura…
        random                    18446603336363922292 cat                   83451      83451 2                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363922360 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363922208 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363922380 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363862956 cat                   83451      83451 1                    [0]push_pipe; [1]_copy_to_iter; [2]get_random_bytes_user; [3]urand…
        random                    18446603336363922252 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363922260 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336357585584 cat                   83451      83451 1                    [0]mutex_spin_on_owner; [1]__mutex_lock.constprop.0; [2]__mutex_lo…
        random                    18446603336363922196 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        random                    18446603336363922444 cat                   83451      83451 1                    [0]chacha_permute; [1]get_random_bytes_user; [2]urandom_read_iter;…
        ```

        * Remove the docker container:

        ```bash
        $ docker stop random
        ```

    </TabItem>
</Tabs>

## GPU-idle attribution (`--gpu-idle-only`)

On a machine running the [gpu-ebpf-bridge](../../pkg/gpu-ebpf-bridge), the
`--gpu-idle-only` flag turns `profile_cpu` into a GPU-starvation profiler: it
only records a stack sample when the sampled process **owns GPU memory** and
its **GPU is underutilized** at that instant. The resulting flamegraph shows
exactly the CPU code that runs while the GPU the process paid for sits idle —
the on-CPU work sitting between successive GPU kernels.

Because it is sampling-based, it has no dependence on the bridge poll interval
and therefore also catches "many small gaps" starvation (for example a
per-frame CPU preprocessing step that keeps an inference GPU at only ~50% duty
cycle), which the recency-based [trace_gpu_starvation](../trace_gpu_starvation)
gadget cannot see.

```bash
$ sudo ig run ghcr.io/inspektor-gadget/gadget/profile_cpu:%IG_TAG% \
    --gpu-idle-only
```

Relevant flags:

- `--gpu-idle-only`: enable the filter (default `false`).
- `--gpu-util-max-pct`: only record a sample when the primary GPU's SM
  utilization is below this percentage (default `70`).
- `--min-gpu-mem-bytes`: minimum resident GPU memory for a process to count as
  a GPU holder (default `1`, i.e. any allocation).
- `--stale-threshold-ms`: discard samples when the bridge data is older than
  this (default `300`). If the bridge is not running the maps are empty and
  every sample is discarded, yielding an empty profile rather than
  mis-attributed stacks.

> **Note on symbolization:** by default `--gpu-idle-only` symbolizes stacks
> from native symbol tables only (the `user_stack_raw.symbols` field), so C/C++
> and other compiled frames (for example `cv::resize` / numpy) resolve. To also
> get interpreted-language frames — most importantly Python — add
> `--collect-otel-stack --symbolizers otel-ebpf-profiler`, which unwinds the
> interpreter stack via the OTel eBPF profiler (supported for `profile_cpu`'s
> `perf_event` program type).
