Bug 4441

Following some ELC discussions and e-mails with Steven Rostedt, he came up with the idea of implementing 'ftrace event triggers', described in more detail below.

He posted a patchset to lkml which is now in linux-next implementing something similar for ftrace function tracing:

http://thread.gmane.org/gmane.linux.kernel/1457880

The code for the event triggers should try to re-use as much as possible from that patchset and should be scheduled for merge into the 3.11 kernel, which practicaly speaking means essentially having something postable following the 3.10 merge window.

'ftrace event triggers' basically add a new pseudo-file named 'trigger' to the subdirectory representing each ftrace event:

root@T420:/sys/kernel/debug/tracing/events/sched/sched_wakeup# ls -al
total 0
drwxr-xr-x  2 root root 0 Apr 10 17:34 .
drwxr-xr-x 20 root root 0 Apr 10 17:34 ..
-rw-r--r--  1 root root 0 Apr 10 17:34 enable
-rw-r--r--  1 root root 0 Apr 10 17:34 filter
-r--r--r--  1 root root 0 Apr 10 17:34 format
-r--r--r--  1 root root 0 Apr 10 17:34 id
-rw-r--r--  1 root root 0 Apr 10 17:34 trigger

Writing to 'trigger' allows that event, when encountered, to trigger
other actions, such as turning on/off other events, snapshotting the
current buffer, or logging a stacktrace.

Being able to trigger other events on/off allows for more targeted
control of a trace.  For example, suppose we wanted to see the sequence
of events that occur in the the block layer and the pagecache when doing
a single read of a file.  We could set it up like this:

# echo 'enable_event:filemap:mm_filemap_add_to_page_cache \
        enable_event:filemap:mm_filemap_delete_from_page_cache' > \
        /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger

# echo 'enable_event:block:*' >> \
        /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger

When the next sys_read happens, the events above are automatically
enabled and are logged to the ftrace trace buffer as they occur.

Those triggered events could in turn be set up to trigger others,
arbitrarily e.g. selected block events could turn on kmem:kmalloc and
get a picture of kernel memory allocation associated with the read, and
so on.

Since we're interested in just a single read, we'd also need triggers to
disable those events when the read completes.  That would be
accomplished by simply adding another set as above but replacing
'enable_event' with 'disable_event' and 'sys_enter_read' with
'sys_exit_read'.

Of course, we can't just rely on the next sys_exit_read to be the one
that matches the sys_enter_read that started the trace, so we need some
way to match those up.  To do that we take advantage of the trace event
filter mechanism already available for trace events and add it as a
optional condition for a given event trigger, which means that the
trigger will only occur if that condition is true.

For example, this would turn off all the block events only if the pid is
the one that was saved in mysaved_pid, which is a variable (not shown in
the above enabling, the syntax for that is undefined at this point, but
for now simply think of it a a way to capture an arbitrary event field
value in a variable when enabling an event) that has captured the pid
for the process that turned on the event:

# echo 'disable_event:block:* if common_pid == mysaved_pid' >> \
        /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger

The 'if <filter>' syntax can be used to qualify any trigger and the
conditions and fields available for trigger filters are exactly the same
set as described in the 'Event formats' and 'Event filtering sections of
Documentation/trace/events.txt.

As mentioned, there will also be other triggers available, such as
'snapshot' triggers, which snapshot the current trace buffer, again
using the same filtering mechanism.  This would be useful for use cases
that would involve enabling large numbers of events (all?) and then
taking a snapshot of the buffer when a rare event occurs, as described
by an event and the set of conditions associated with the filter
(speculative tracing).

I think this adds a completely new dimension to the utility of the
kernel trace events and makes them much more useful, especially to
embedded users - everything is in the kernel and simply available via
nothing more than reads and writes a set of debugfs files.  But, it's
also infrastructure that will become available to other tracers e.g
'perf trace'.  I think that it will also provide a good hook for Intel
hardware-specific tracing e.g. it should be possible to add an 'IPT
trigger', which would allow trace events to start, stop, and capture an
IPT trace, for example.

At this point, I've implemented only a small part, basically the event
triggers without filtering, and all that does so far is crash the
machine.  There have been so many more pressing things to do related to
the dylan release, and I've been so scattered that it's hard to make
progress, but it's getting there and I hope to get more time after that.