Demonstrations of slabratetop, the Linux eBPF/bcc version.
slabratetop shows the rate of allocations and total bytes from the kernel
memory allocation caches (SLAB or SLUB), in a top-like display that refreshes.
For example:
# ./slabratetop
<screen clears>
07:01:35 loadavg: 0.38 0.21 0.12 1/342 13297
CACHE ALLOCS BYTES
kmalloc-4096 3554 14557184
kmalloc-256 2382 609536
cred_jar 2568 493056
anon_vma_chain 2007 128448
anon_vma 972 77760
sighand_cache 24 50688
mm_struct 49 50176
RAW 52 49920
proc_inode_cache 59 38232
signal_cache 24 26112
dentry 135 25920
sock_inode_cache 29 18560
files_cache 24 16896
inode_cache 13 7696
TCP 2 3840
pid 24 3072
sigqueue 17 2720
ext4_inode_cache 2 2160
buffer_head 16 1664
xfs_trans 5 1160
By default the screen refreshes every one second, and only the top 20 caches
are shown. These can be tuned with options: see USAGE (-h).
The output above showed that the kmalloc-4096 cache allocated the most, about
14 Mbytes during this interval. This is a generic cache; other caches have
more meaningful names ("dentry", "TCP", "pid", etc).
slabtop(1) is a similar tool that shows the current static volume and usage
of the caches. slabratetop shows the active call rates and total size of the
allocations.
Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the
kernel stacks that led to this allocation. In the future (maybe by now) the
bcc trace tool could do this. As I'm writing this, it can't, so I'll use my
older ftrace-based kprobe tool as a workarond. This is from my perf-tools
collection: https://github.com/brendangregg/perf-tools.
# ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100
Tracing kprobe kmem_cache_alloc. Ctrl-C to end.
kprobe-3892 [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478333: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
kprobe-3892 [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478341: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
kprobe-3892 [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478346: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
kprobe-3892 [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478351: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
kprobe-3892 [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478355: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
kprobe-3892 [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096"
kprobe-3892 [002] d... 7888274.478359: <stack trace>
=> kmem_cache_alloc
=> user_path_at_empty
=> vfs_fstatat
=> SYSC_newstat
=> SyS_newstat
=> entry_SYSCALL_64_fastpath
[...]
This is just an example so that you can see it's possible to dig further.
Please don't copy-n-paste that kprobe command, as it's unlikely to work (the
"+0(+96(%di))" text is specific to a kernel version and architecture).
So these allocations are coming from user_path_at_empty(), which calls other
functions (not seen in the stack: I suspect it's a tail-call compiler
optimization).
USAGE:
# ./slabratetop -h
usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count]
Kernel SLAB/SLUB memory cache allocation rate top
positional arguments:
interval output interval, in seconds
count number of outputs
optional arguments:
-h, --help show this help message and exit
-C, --noclear don't clear the screen
-r MAXROWS, --maxrows MAXROWS
maximum rows to print, default 20
examples:
./slabratetop # kmem_cache_alloc() top, 1 second refresh
./slabratetop -C # don't clear the screen
./slabratetop 5 # 5 second summaries
./slabratetop 5 10 # 5 second summaries, 10 times only