Stack filtering and kernel/user stacks

Stack filtering#

To have a finer control of what parts of stack traces are saved and processed into flame graphs for example, Adaptyst offers an option of defining a dedicated filter using either regular expressions or a custom Python script implementing the filtering API.

Regular expressions#

A regex-based filter can be configured using a text file or a text sent to the standard input. The syntax is as follows:

# Each line can be either a comment starting with #, a regex prefixed
# by "SYM " indicating that symbol names should be queried in stack traces,
# a regex prefixed by "EXEC " indicating that executable paths (e.g.
# shared libraries) should be queried in stack traces, a regex prefixed
# by "ANY " indicating that both symbol names and executable paths should
# be queried, or "OR" separating the condition groups.
#
# A condition group is a set of SYM, EXEC, and ANY statements. The group
# is considered satisfied when *all* statements inside it evaluate to true.
#
# The entire filter evaluates to true when *any* condition group evaluates
# to true.
#
# In the example below, the filter for a stack trace element evalues to true
# when *either*:
# - the symbol name in the element matches <regex1> AND the executable path
# in the element matches <regex2> AND anything in the element matches <regex3>
# - the executable path in the element matches <regex4>
# - the executable path in the element matches <regex5>

SYM <regex1>
EXEC <regex2>
ANY <regex3>
OR
EXEC <regex4>
OR
EXEC <regex5>

Regular expressions should be written using the Python variant. The filter defined in this way processes every element in a stack trace one-by-one, so it is not possible to make e.g. conditions based on the contents of several elements at once. This can be done using the Python API instead.

After the filter is defined, it can be supplied to Adaptyst by running it with the -i <allow OR deny>:<file OR -> option. The filter can serve as either an allowlist (i.e. if the filter evaluates to true for an element, the element is saved, otherwise the element is cut) or a denylist (i.e. if the filter evaluates to true for an element, the element is cut, otherwise the element is saved).

Python API#

For more advanced use cases, Adaptyst allows running a Python script which is given a full stack trace at once per sample/event for processing.

The script should implement the following global methods:

• setup() (no arguments and no return value): called once at the beginning of profiling by each “perf”-based profiler (Adaptyst uses two such profilers by default: one for process/thread tracing and one for on-CPU/off-CPU profiling. Additionally, one profiler per custom performance counter is also used.)
• process(callchain) (callchain is a tuple and the return value is a list): called once per sample/event with an entire stack trace stored in callchain and returning the list of exactly the same size as callchain, where the only values are True or False, indicating at the i-th index whether the i-th element of the stack trace should be saved (True) or cut (False)
  • Every element of callchain is of form ((symbol name, executable path), offset address). The first element is the top of the stack.
  • symbol name is a string and can be either a demangled symbol name if found, an executable path enclosed in [] (e.g. [/lib/libc.so]) if found, or in the worst case, an instruction pointer address in hex enclosed in [] (e.g. [0xFF]).
  • executable path is a string and can be either an executable path if found or an empty string otherwise.
  • offset address is a string and can be either an offset address in hex within the executable pointed to by an executable path if found or an instruction pointer address in hex otherwise.

Once your script is defined, it can be supplied to Adaptyst by running it with the -i python:<script file> option. No standard input is accepted here.

Marking deleted elements#

By default, when a stack trace element is meant to be deleted, it is removed completely without leaving any traces (e.g. if B is cut from A -> B -> C, only A -> C remains).

However, it is possible to tell Adaptyst to mark deleted elements as (cut) instead, where all consecutive (cut)’s are squashed into one (cut), e.g. if B is cut from A -> B -> B -> C -> B -> D, the resulting trace is A -> (cut) -> C -> (cut) -> D. In order to do so, run Adaptyst with -k.

Kernel and user stacks#

By default, Adaptyst collects only stack traces from the user space of a program. However, there is an option of accessing the kernel space as well and obtaining traces from there. Please use the -m option for this: you can ask Adaptyst to extract either user stacks only (-m user), kernel stacks only (-m kernel), or both stack traces (which are merged into one then) (-m both).

Unless you profile a specialised program involving e.g. device drivers, you shouldn’t need to change where stack traces will be collected from for a given application.