code:
=========================
Hardware Latency Detector
=========================

Introduction
-------------

The tracer hwlat_detector is a special purpose tracer that is used to
detect large system latencies induced by the behavior of certain underlying
hardware or firmware, independent of Linux itself. The code was developed
originally to detect SMIs (System Management Interrupts) on x86 systems,
however there is nothing x86 specific about this patchset. It was
originally written for use by the "RT" patch since the Real Time
kernel is highly latency sensitive.

SMIs are not serviced by the Linux kernel, which means that it does not
even know that they are occuring. SMIs are instead set up by BIOS code
and are serviced by BIOS code, usually for "critical" events such as
management of thermal sensors and fans. Sometimes though, SMIs are used for
other tasks and those tasks can spend an inordinate amount of time in the
handler (sometimes measured in milliseconds). Obviously this is a problem if
you are trying to keep event service latencies down in the microsecond range.

The hardware latency detector works by hogging one of the cpus for configurable
amounts of time (with interrupts disabled), polling the CPU Time Stamp Counter
for some period, then looking for gaps in the TSC data. Any gap indicates a
time when the polling was interrupted and since the interrupts are disabled,
the only thing that could do that would be an SMI or other hardware hiccup
(or an NMI, but those can be tracked).

Note that the hwlat detector should *NEVER* be used in a production environment.
It is intended to be run manually to determine if the hardware platform has a
problem with long system firmware service routines.

Usage
------

Write the ASCII text "hwlat" into the current_tracer file of the tracing system
(mounted at /sys/kernel/tracing or /sys/kernel/tracing). It is possible to
redefine the threshold in microseconds (us) above which latency spikes will
be taken into account.

Example::

	# echo hwlat > /sys/kernel/tracing/current_tracer
	# echo 100 > /sys/kernel/tracing/tracing_thresh

The /sys/kernel/tracing/hwlat_detector interface contains the following files:

  - width - time period to sample with CPUs held (usecs)
            must be less than the total window size (enforced)
  - window - total period of sampling, width being inside (usecs)

By default the width is set to 500,000 and window to 1,000,000, meaning that
for every 1,000,000 usecs (1s) the hwlat detector will spin for 500,000 usecs
(0.5s). If tracing_thresh contains zero when hwlat tracer is enabled, it will
change to a default of 10 usecs. If any latencies that exceed the threshold is
observed then the data will be written to the tracing ring buffer.

The minimum sleep time between periods is 1 millisecond. Even if width
is less than 1 millisecond apart from window, to allow the system to not
be totally starved.

If tracing_thresh was zero when hwlat detector was started, it will be set
back to zero if another tracer is loaded. Note, the last value in
tracing_thresh that hwlat detector had will be saved and this value will
be restored in tracing_thresh if it is still zero when hwlat detector is
started again.

The following tracing directory files are used by the hwlat_detector:

in /sys/kernel/tracing:

 - tracing_threshold	- minimum latency value to be considered (usecs)
 - tracing_max_latency	- maximum hardware latency actually observed (usecs)
 - tracing_cpumask	- the CPUs to move the hwlat thread across
 - hwlat_detector/width	- specified amount of time to spin within window (usecs)
 - hwlat_detector/window	- amount of time between (width) runs (usecs)
 - hwlat_detector/mode	- the thread mode

By default, one hwlat detector's kernel thread will migrate across each CPU
specified in cpumask at the beginning of a new window, in a round-robin
fashion. This behavior can be changed by changing the thread mode,
the available options are:

 - none:        do not force migration
 - round-robin: migrate across each CPU specified in cpumask [default]
 - per-cpu:     create one thread for each cpu in tracing_cpumask


code:
==========================
Linux Tracing Technologies
==========================

.. toctree::
   :maxdepth: 2

   ftrace-design
   tracepoint-analysis
   ftrace
   ftrace-uses
   fprobe
   kprobes
   kprobetrace
   uprobetracer
   tracepoints
   events
   events-kmem
   events-power
   events-nmi
   events-msr
   mmiotrace
   histogram
   histogram-design
   boottime-trace
   hwlat_detector
   osnoise-tracer
   timerlat-tracer
   intel_th
   ring-buffer-design
   stm
   sys-t
   coresight/index
   user_events
   rv/index
   hisi-ptt

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

=======================
Intel(R) Trace Hub (TH)
=======================

Overview
--------

Intel(R) Trace Hub (TH) is a set of hardware blocks that produce,
switch and output trace data from multiple hardware and software
sources over several types of trace output ports encoded in System
Trace Protocol (MIPI STPv2) and is intended to perform full system
debugging. For more information on the hardware, see Intel(R) Trace
Hub developer's manual [1].

It consists of trace sources, trace destinations (outputs) and a
switch (Global Trace Hub, GTH). These devices are placed on a bus of
their own ("intel_th"), where they can be discovered and configured
via sysfs attributes.

Currently, the following Intel TH subdevices (blocks) are supported:
  - Software Trace Hub (STH), trace source, which is a System Trace
    Module (STM) device,
  - Memory Storage Unit (MSU), trace output, which allows storing
    trace hub output in system memory,
  - Parallel Trace Interface output (PTI), trace output to an external
    debug host via a PTI port,
  - Global Trace Hub (GTH), which is a switch and a central component
    of Intel(R) Trace Hub architecture.

Common attributes for output devices are described in
Documentation/ABI/testing/sysfs-bus-intel_th-output-devices, the most
notable of them is "active", which enables or disables trace output
into that particular output device.

GTH allows directing different STP masters into different output ports
via its "masters" attribute group. More detailed GTH interface
description is at Documentation/ABI/testing/sysfs-bus-intel_th-devices-gth.

STH registers an stm class device, through which it provides interface
to userspace and kernelspace software trace sources. See
Documentation/trace/stm.rst for more information on that.

MSU can be configured to collect trace data into a system memory
buffer, which can later on be read from its device nodes via read() or
mmap() interface and directed to a "software sink" driver that will
consume the data and/or relay it further.

On the whole, Intel(R) Trace Hub does not require any special
userspace software to function; everything can be configured, started
and collected via sysfs attributes, and device nodes.

[1] https://software.intel.com/sites/default/files/managed/d3/3c/intel-th-developer-manual.pdf

Bus and Subdevices
------------------

For each Intel TH device in the system a bus of its own is
created and assigned an id number that reflects the order in which TH
devices were enumerated. All TH subdevices (devices on intel_th bus)
begin with this id: 0-gth, 0-msc0, 0-msc1, 0-pti, 0-sth, which is
followed by device's name and an optional index.

Output devices also get a device node in /dev/intel_thN, where N is
the Intel TH device id. For example, MSU's memory buffers, when
allocated, are accessible via /dev/intel_th0/msc{0,1}.

Quick example
-------------

# figure out which GTH port is the first memory controller::

	$ cat /sys/bus/intel_th/devices/0-msc0/port
	0

# looks like it's port 0, configure master 33 to send data to port 0::

	$ echo 0 > /sys/bus/intel_th/devices/0-gth/masters/33

# allocate a 2-windowed multiblock buffer on the first memory
# controller, each with 64 pages::

	$ echo multi > /sys/bus/intel_th/devices/0-msc0/mode
	$ echo 64,64 > /sys/bus/intel_th/devices/0-msc0/nr_pages

# enable wrapping for this controller, too::

	$ echo 1 > /sys/bus/intel_th/devices/0-msc0/wrap

# and enable tracing into this port::

	$ echo 1 > /sys/bus/intel_th/devices/0-msc0/active

# .. send data to master 33, see stm.txt for more details ..
# .. wait for traces to pile up ..
# .. and stop the trace::

	$ echo 0 > /sys/bus/intel_th/devices/0-msc0/active

# and now you can collect the trace from the device node::

	$ cat /dev/intel_th0/msc0 > my_stp_trace

Host Debugger Mode
------------------

It is possible to configure the Trace Hub and control its trace
capture from a remote debug host, which should be connected via one of
the hardware debugging interfaces, which will then be used to both
control Intel Trace Hub and transfer its trace data to the debug host.

The driver needs to be told that such an arrangement is taking place
so that it does not touch any capture/port configuration and avoids
conflicting with the debug host's configuration accesses. The only
activity that the driver will perform in this mode is collecting
software traces to the Software Trace Hub (an stm class device). The
user is still responsible for setting up adequate master/channel
mappings that the decoder on the receiving end would recognize.

In order to enable the host mode, set the 'host_mode' parameter of the
'intel_th' kernel module to 'y'. None of the virtual output devices
will show up on the intel_th bus. Also, trace configuration and
capture controlling attribute groups of the 'gth' device will not be
exposed. The 'sth' device will operate as usual.

Software Sinks
--------------

The Memory Storage Unit (MSU) driver provides an in-kernel API for
drivers to register themselves as software sinks for the trace data.
Such drivers can further export the data via other devices, such as
USB device controllers or network cards.

The API has two main parts::
 - notifying the software sink that a particular window is full, and
   "locking" that window, that is, making it unavailable for the trace
   collection; when this happens, the MSU driver will automatically
   switch to the next window in the buffer if it is unlocked, or stop
   the trace capture if it's not;
 - tracking the "locked" state of windows and providing a way for the
   software sink driver to notify the MSU driver when a window is
   unlocked and can be used again to collect trace data.

An example sink driver, msu-sink illustrates the implementation of a
software sink. Functionally, it simply unlocks windows as soon as they
are full, keeping the MSU running in a circular buffer mode. Unlike the
"multi" mode, it will fill out all the windows in the buffer as opposed
to just the first one. It can be enabled by writing "sink" to the "mode"
file (assuming msu-sink.ko is loaded).


code:
=======================
Kernel Probes (Kprobes)
=======================

:Author: Jim Keniston <jkenisto@us.ibm.com>
:Author: Prasanna S Panchamukhi <prasanna.panchamukhi@gmail.com>
:Author: Masami Hiramatsu <mhiramat@redhat.com>

.. CONTENTS

  1. Concepts: Kprobes, and Return Probes
  2. Architectures Supported
  3. Configuring Kprobes
  4. API Reference
  5. Kprobes Features and Limitations
  6. Probe Overhead
  7. TODO
  8. Kprobes Example
  9. Kretprobes Example
  10. Deprecated Features
  Appendix A: The kprobes debugfs interface
  Appendix B: The kprobes sysctl interface
  Appendix C: References

Concepts: Kprobes and Return Probes
=========================================

Kprobes enables you to dynamically break into any kernel routine and
collect debugging and performance information non-disruptively. You
can trap at almost any kernel code address [1]_, specifying a handler
routine to be invoked when the breakpoint is hit.

.. [1] some parts of the kernel code can not be trapped, see
       :ref:`kprobes_blacklist`)

There are currently two types of probes: kprobes, and kretprobes
(also called return probes).  A kprobe can be inserted on virtually
any instruction in the kernel.  A return probe fires when a specified
function returns.

In the typical case, Kprobes-based instrumentation is packaged as
a kernel module.  The module's init function installs ("registers")
one or more probes, and the exit function unregisters them.  A
registration function such as register_kprobe() specifies where
the probe is to be inserted and what handler is to be called when
the probe is hit.

There are also ``register_/unregister_*probes()`` functions for batch
registration/unregistration of a group of ``*probes``. These functions
can speed up unregistration process when you have to unregister
a lot of probes at once.

The next four subsections explain how the different types of
probes work and how jump optimization works.  They explain certain
things that you'll need to know in order to make the best use of
Kprobes -- e.g., the difference between a pre_handler and
a post_handler, and how to use the maxactive and nmissed fields of
a kretprobe.  But if you're in a hurry to start using Kprobes, you
can skip ahead to :ref:`kprobes_archs_supported`.

How Does a Kprobe Work?
-----------------------

When a kprobe is registered, Kprobes makes a copy of the probed
instruction and replaces the first byte(s) of the probed instruction
with a breakpoint instruction (e.g., int3 on i386 and x86_64).

When a CPU hits the breakpoint instruction, a trap occurs, the CPU's
registers are saved, and control passes to Kprobes via the
notifier_call_chain mechanism.  Kprobes executes the "pre_handler"
associated with the kprobe, passing the handler the addresses of the
kprobe struct and the saved registers.

Next, Kprobes single-steps its copy of the probed instruction.
(It would be simpler to single-step the actual instruction in place,
but then Kprobes would have to temporarily remove the breakpoint
instruction.  This would open a small time window when another CPU
could sail right past the probepoint.)

After the instruction is single-stepped, Kprobes executes the
"post_handler," if any, that is associated with the kprobe.
Execution then continues with the instruction following the probepoint.

Changing Execution Path
-----------------------

Since kprobes can probe into a running kernel code, it can change the
register set, including instruction pointer. This operation requires
maximum care, such as keeping the stack frame, recovering the execution
path etc. Since it operates on a running kernel and needs deep knowledge
of computer architecture and concurrent computing, you can easily shoot
your foot.

If you change the instruction pointer (and set up other related
registers) in pre_handler, you must return !0 so that kprobes stops
single stepping and just returns to the given address.
This also means post_handler should not be called anymore.

Note that this operation may be harder on some architectures which use
TOC (Table of Contents) for function call, since you have to setup a new
TOC for your function in your module, and recover the old one after
returning from it.

Return Probes
-------------

How Does a Return Probe Work?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When you call register_kretprobe(), Kprobes establishes a kprobe at
the entry to the function.  When the probed function is called and this
probe is hit, Kprobes saves a copy of the return address, and replaces
the return address with the address of a "trampoline."  The trampoline
is an arbitrary piece of code -- typically just a nop instruction.
At boot time, Kprobes registers a kprobe at the trampoline.

When the probed function executes its return instruction, control
passes to the trampoline and that probe is hit.  Kprobes' trampoline
handler calls the user-specified return handler associated with the
kretprobe, then sets the saved instruction pointer to the saved return
address, and that's where execution resumes upon return from the trap.

While the probed function is executing, its return address is
stored in an object of type kretprobe_instance.  Before calling
register_kretprobe(), the user sets the maxactive field of the
kretprobe struct to specify how many instances of the specified
function can be probed simultaneously.  register_kretprobe()
pre-allocates the indicated number of kretprobe_instance objects.

For example, if the function is non-recursive and is called with a
spinlock held, maxactive = 1 should be enough.  If the function is
non-recursive and can never relinquish the CPU (e.g., via a semaphore
or preemption), NR_CPUS should be enough.  If maxactive <= 0, it is
set to a default value.  If CONFIG_PREEMPT is enabled, the default
is max(10, 2*NR_CPUS).  Otherwise, the default is NR_CPUS.

It's not a disaster if you set maxactive too low; you'll just miss
some probes.  In the kretprobe struct, the nmissed field is set to
zero when the return probe is registered, and is incremented every
time the probed function is entered but there is no kretprobe_instance
object available for establishing the return probe.

Kretprobe entry-handler
^^^^^^^^^^^^^^^^^^^^^^^

Kretprobes also provides an optional user-specified handler which runs
on function entry. This handler is specified by setting the entry_handler
field of the kretprobe struct. Whenever the kprobe placed by kretprobe at the
function entry is hit, the user-defined entry_handler, if any, is invoked.
If the entry_handler returns 0 (success) then a corresponding return handler
is guaranteed to be called upon function return. If the entry_handler
returns a non-zero error then Kprobes leaves the return address as is, and
the kretprobe has no further effect for that particular function instance.

Multiple entry and return handler invocations are matched using the unique
kretprobe_instance object associated with them. Additionally, a user
may also specify per return-instance private data to be part of each
kretprobe_instance object. This is especially useful when sharing private
data between corresponding user entry and return handlers. The size of each
private data object can be specified at kretprobe registration time by
setting the data_size field of the kretprobe struct. This data can be
accessed through the data field of each kretprobe_instance object.

In case probed function is entered but there is no kretprobe_instance
object available, then in addition to incrementing the nmissed count,
the user entry_handler invocation is also skipped.

.. _kprobes_jump_optimization:

How Does Jump Optimization Work?
--------------------------------

If your kernel is built with CONFIG_OPTPROBES=y (currently this flag
is automatically set 'y' on x86/x86-64, non-preemptive kernel) and
the "debug.kprobes_optimization" kernel parameter is set to 1 (see
sysctl(8)), Kprobes tries to reduce probe-hit overhead by using a jump
instruction instead of a breakpoint instruction at each probepoint.

Init a Kprobe
^^^^^^^^^^^^^

When a probe is registered, before attempting this optimization,
Kprobes inserts an ordinary, breakpoint-based kprobe at the specified
address. So, even if it's not possible to optimize this particular
probepoint, there'll be a probe there.

Safety Check
^^^^^^^^^^^^

Before optimizing a probe, Kprobes performs the following safety checks:

- Kprobes verifies that the region that will be replaced by the jump
  instruction (the "optimized region") lies entirely within one function.
  (A jump instruction is multiple bytes, and so may overlay multiple
  instructions.)

- Kprobes analyzes the entire function and verifies that there is no
  jump into the optimized region.  Specifically:

  - the function contains no indirect jump;
  - the function contains no instruction that causes an exception (since
    the fixup code triggered by the exception could jump back into the
    optimized region -- Kprobes checks the exception tables to verify this);
  - there is no near jump to the optimized region (other than to the first
    byte).

- For each instruction in the optimized region, Kprobes verifies that
  the instruction can be executed out of line.

Preparing Detour Buffer
^^^^^^^^^^^^^^^^^^^^^^^

Next, Kprobes prepares a "detour" buffer, which contains the following
instruction sequence:

- code to push the CPU's registers (emulating a breakpoint trap)
- a call to the trampoline code which calls user's probe handlers.
- code to restore registers
- the instructions from the optimized region
- a jump back to the original execution path.

Pre-optimization
^^^^^^^^^^^^^^^^

After preparing the detour buffer, Kprobes verifies that none of the
following situations exist:

- The probe has a post_handler.
- Other instructions in the optimized region are probed.
- The probe is disabled.

In any of the above cases, Kprobes won't start optimizing the probe.
Since these are temporary situations, Kprobes tries to start
optimizing it again if the situation is changed.

If the kprobe can be optimized, Kprobes enqueues the kprobe to an
optimizing list, and kicks the kprobe-optimizer workqueue to optimize
it.  If the to-be-optimized probepoint is hit before being optimized,
Kprobes returns control to the original instruction path by setting
the CPU's instruction pointer to the copied code in the detour buffer
-- thus at least avoiding the single-step.

Optimization
^^^^^^^^^^^^

The Kprobe-optimizer doesn't insert the jump instruction immediately;
rather, it calls synchronize_rcu() for safety first, because it's
possible for a CPU to be interrupted in the middle of executing the
optimized region [3]_.  As you know, synchronize_rcu() can ensure
that all interruptions that were active when synchronize_rcu()
was called are done, but only if CONFIG_PREEMPT=n.  So, this version
of kprobe optimization supports only kernels with CONFIG_PREEMPT=n [4]_.

After that, the Kprobe-optimizer calls stop_machine() to replace
the optimized region with a jump instruction to the detour buffer,
using text_poke_smp().

Unoptimization
^^^^^^^^^^^^^^

When an optimized kprobe is unregistered, disabled, or blocked by
another kprobe, it will be unoptimized.  If this happens before
the optimization is complete, the kprobe is just dequeued from the
optimized list.  If the optimization has been done, the jump is
replaced with the original code (except for an int3 breakpoint in
the first byte) by using text_poke_smp().

.. [3] Please imagine that the 2nd instruction is interrupted and then
   the optimizer replaces the 2nd instruction with the jump *address*
   while the interrupt handler is running. When the interrupt
   returns to original address, there is no valid instruction,
   and it causes an unexpected result.

.. [4] This optimization-safety checking may be replaced with the
   stop-machine method that ksplice uses for supporting a CONFIG_PREEMPT=y
   kernel.

NOTE for geeks:
The jump optimization changes the kprobe's pre_handler behavior.
Without optimization, the pre_handler can change the kernel's execution
path by changing regs->ip and returning 1.  However, when the probe
is optimized, that modification is ignored.  Thus, if you want to
tweak the kernel's execution path, you need to suppress optimization,
using one of the following techniques:

- Specify an empty function for the kprobe's post_handler.

or

- Execute 'sysctl -w debug.kprobes_optimization=n'

.. _kprobes_blacklist:

Blacklist
---------

Kprobes can probe most of the kernel except itself. This means
that there are some functions where kprobes cannot probe. Probing
(trapping) such functions can cause a recursive trap (e.g. double
fault) or the nested probe handler may never be called.
Kprobes manages such functions as a blacklist.
If you want to add a function into the blacklist, you just need
to (1) include linux/kprobes.h and (2) use NOKPROBE_SYMBOL() macro
to specify a blacklisted function.
Kprobes checks the given probe address against the blacklist and
rejects registering it, if the given address is in the blacklist.

.. _kprobes_archs_supported:

Architectures Supported
=======================

Kprobes and return probes are implemented on the following
architectures:

- i386 (Supports jump optimization)
- x86_64 (AMD-64, EM64T) (Supports jump optimization)
- ppc64
- ia64 (Does not support probes on instruction slot1.)
- sparc64 (Return probes not yet implemented.)
- arm
- ppc
- mips
- s390
- parisc

Configuring Kprobes
===================

When configuring the kernel using make menuconfig/xconfig/oldconfig,
ensure that CONFIG_KPROBES is set to "y", look for "Kprobes" under
"General architecture-dependent options".

So that you can load and unload Kprobes-based instrumentation modules,
make sure "Loadable module support" (CONFIG_MODULES) and "Module
unloading" (CONFIG_MODULE_UNLOAD) are set to "y".

Also make sure that CONFIG_KALLSYMS and perhaps even CONFIG_KALLSYMS_ALL
are set to "y", since kallsyms_lookup_name() is used by the in-kernel
kprobe address resolution code.

If you need to insert a probe in the middle of a function, you may find
it useful to "Compile the kernel with debug info" (CONFIG_DEBUG_INFO),
so you can use "objdump -d -l vmlinux" to see the source-to-object
code mapping.

API Reference
=============

The Kprobes API includes a "register" function and an "unregister"
function for each type of probe. The API also includes "register_*probes"
and "unregister_*probes" functions for (un)registering arrays of probes.
Here are terse, mini-man-page specifications for these functions and
the associated probe handlers that you'll write. See the files in the
samples/kprobes/ sub-directory for examples.

register_kprobe
---------------

::

	#include <linux/kprobes.h>
	int register_kprobe(struct kprobe *kp);

Sets a breakpoint at the address kp->addr.  When the breakpoint is hit, Kprobes
calls kp->pre_handler.  After the probed instruction is single-stepped, Kprobe
calls kp->post_handler.  Any or all handlers can be NULL. If kp->flags is set
KPROBE_FLAG_DISABLED, that kp will be registered but disabled, so, its handlers
aren't hit until calling enable_kprobe(kp).

.. note::

   1. With the introduction of the "symbol_name" field to struct kprobe,
      the probepoint address resolution will now be taken care of by the kernel.
      The following will now work::

	kp.symbol_name = "symbol_name";

      (64-bit powerpc intricacies such as function descriptors are handled
      transparently)

   2. Use the "offset" field of struct kprobe if the offset into the symbol
      to install a probepoint is known. This field is used to calculate the
      probepoint.

   3. Specify either the kprobe "symbol_name" OR the "addr". If both are
      specified, kprobe registration will fail with -EINVAL.

   4. With CISC architectures (such as i386 and x86_64), the kprobes code
      does not validate if the kprobe.addr is at an instruction boundary.
      Use "offset" with caution.

register_kprobe() returns 0 on success, or a negative errno otherwise.

User's pre-handler (kp->pre_handler)::

	#include <linux/kprobes.h>
	#include <linux/ptrace.h>
	int pre_handler(struct kprobe *p, struct pt_regs *regs);

Called with p pointing to the kprobe associated with the breakpoint,
and regs pointing to the struct containing the registers saved when
the breakpoint was hit.  Return 0 here unless you're a Kprobes geek.

User's post-handler (kp->post_handler)::

	#include <linux/kprobes.h>
	#include <linux/ptrace.h>
	void post_handler(struct kprobe *p, struct pt_regs *regs,
			  unsigned long flags);

p and regs are as described for the pre_handler.  flags always seems
to be zero.

register_kretprobe
------------------

::

	#include <linux/kprobes.h>
	int register_kretprobe(struct kretprobe *rp);

Establishes a return probe for the function whose address is
rp->kp.addr.  When that function returns, Kprobes calls rp->handler.
You must set rp->maxactive appropriately before you call
register_kretprobe(); see "How Does a Return Probe Work?" for details.

register_kretprobe() returns 0 on success, or a negative errno
otherwise.

User's return-probe handler (rp->handler)::

	#include <linux/kprobes.h>
	#include <linux/ptrace.h>
	int kretprobe_handler(struct kretprobe_instance *ri,
			      struct pt_regs *regs);

regs is as described for kprobe.pre_handler.  ri points to the
kretprobe_instance object, of which the following fields may be
of interest:

- ret_addr: the return address
- rp: points to the corresponding kretprobe object
- task: points to the corresponding task struct
- data: points to per return-instance private data; see "Kretprobe
	entry-handler" for details.

The regs_return_value(regs) macro provides a simple abstraction to
extract the return value from the appropriate register as defined by
the architecture's ABI.

The handler's return value is currently ignored.

unregister_*probe
------------------

::

	#include <linux/kprobes.h>
	void unregister_kprobe(struct kprobe *kp);
	void unregister_kretprobe(struct kretprobe *rp);

Removes the specified probe.  The unregister function can be called
at any time after the probe has been registered.

.. note::

   If the functions find an incorrect probe (ex. an unregistered probe),
   they clear the addr field of the probe.

register_*probes
----------------

::

	#include <linux/kprobes.h>
	int register_kprobes(struct kprobe **kps, int num);
	int register_kretprobes(struct kretprobe **rps, int num);

Registers each of the num probes in the specified array.  If any
error occurs during registration, all probes in the array, up to
the bad probe, are safely unregistered before the register_*probes
function returns.

- kps/rps: an array of pointers to ``*probe`` data structures
- num: the number of the array entries.

.. note::

   You have to allocate(or define) an array of pointers and set all
   of the array entries before using these functions.

unregister_*probes
------------------

::

	#include <linux/kprobes.h>
	void unregister_kprobes(struct kprobe **kps, int num);
	void unregister_kretprobes(struct kretprobe **rps, int num);

Removes each of the num probes in the specified array at once.

.. note::

   If the functions find some incorrect probes (ex. unregistered
   probes) in the specified array, they clear the addr field of those
   incorrect probes. However, other probes in the array are
   unregistered correctly.

disable_*probe
--------------

::

	#include <linux/kprobes.h>
	int disable_kprobe(struct kprobe *kp);
	int disable_kretprobe(struct kretprobe *rp);

Temporarily disables the specified ``*probe``. You can enable it again by using
enable_*probe(). You must specify the probe which has been registered.

enable_*probe
-------------

::

	#include <linux/kprobes.h>
	int enable_kprobe(struct kprobe *kp);
	int enable_kretprobe(struct kretprobe *rp);

Enables ``*probe`` which has been disabled by disable_*probe(). You must specify
the probe which has been registered.

Kprobes Features and Limitations
================================

Kprobes allows multiple probes at the same address. Also,
a probepoint for which there is a post_handler cannot be optimized.
So if you install a kprobe with a post_handler, at an optimized
probepoint, the probepoint will be unoptimized automatically.

In general, you can install a probe anywhere in the kernel.
In particular, you can probe interrupt handlers.  Known exceptions
are discussed in this section.

The register_*probe functions will return -EINVAL if you attempt
to install a probe in the code that implements Kprobes (mostly
kernel/kprobes.c and ``arch/*/kernel/kprobes.c``, but also functions such
as do_page_fault and notifier_call_chain).

If you install a probe in an inline-able function, Kprobes makes
no attempt to chase down all inline instances of the function and
install probes there.  gcc may inline a function without being asked,
so keep this in mind if you're not seeing the probe hits you expect.

A probe handler can modify the environment of the probed function
-- e.g., by modifying kernel data structures, or by modifying the
contents of the pt_regs struct (which are restored to the registers
upon return from the breakpoint).  So Kprobes can be used, for example,
to install a bug fix or to inject faults for testing.  Kprobes, of
course, has no way to distinguish the deliberately injected faults
from the accidental ones.  Don't drink and probe.

Kprobes makes no attempt to prevent probe handlers from stepping on
each other -- e.g., probing printk() and then calling printk() from a
probe handler.  If a probe handler hits a probe, that second probe's
handlers won't be run in that instance, and the kprobe.nmissed member
of the second probe will be incremented.

As of Linux v2.6.15-rc1, multiple handlers (or multiple instances of
the same handler) may run concurrently on different CPUs.

Kprobes does not use mutexes or allocate memory except during
registration and unregistration.

Probe handlers are run with preemption disabled or interrupt disabled,
which depends on the architecture and optimization state.  (e.g.,
kretprobe handlers and optimized kprobe handlers run without interrupt
disabled on x86/x86-64).  In any case, your handler should not yield
the CPU (e.g., by attempting to acquire a semaphore, or waiting I/O).

Since a return probe is implemented by replacing the return
address with the trampoline's address, stack backtraces and calls
to __builtin_return_address() will typically yield the trampoline's
address instead of the real return address for kretprobed functions.
(As far as we can tell, __builtin_return_address() is used only
for instrumentation and error reporting.)

If the number of times a function is called does not match the number
of times it returns, registering a return probe on that function may
produce undesirable results. In such a case, a line:
kretprobe BUG!: Processing kretprobe d000000000041aa8 @ c00000000004f48c
gets printed. With this information, one will be able to correlate the
exact instance of the kretprobe that caused the problem. We have the
do_exit() case covered. do_execve() and do_fork() are not an issue.
We're unaware of other specific cases where this could be a problem.

If, upon entry to or exit from a function, the CPU is running on
a stack other than that of the current task, registering a return
probe on that function may produce undesirable results.  For this
reason, Kprobes doesn't support return probes (or kprobes)
on the x86_64 version of __switch_to(); the registration functions
return -EINVAL.

On x86/x86-64, since the Jump Optimization of Kprobes modifies
instructions widely, there are some limitations to optimization. To
explain it, we introduce some terminology. Imagine a 3-instruction
sequence consisting of a two 2-byte instructions and one 3-byte
instruction.

::

		IA
		|
	[-2][-1][0][1][2][3][4][5][6][7]
		[ins1][ins2][  ins3 ]
		[<-     DCR       ->]
		[<- JTPR ->]

	ins1: 1st Instruction
	ins2: 2nd Instruction
	ins3: 3rd Instruction
	IA:  Insertion Address
	JTPR: Jump Target Prohibition Region
	DCR: Detoured Code Region

The instructions in DCR are copied to the out-of-line buffer
of the kprobe, because the bytes in DCR are replaced by
a 5-byte jump instruction. So there are several limitations.

a) The instructions in DCR must be relocatable.
b) The instructions in DCR must not include a call instruction.
c) JTPR must not be targeted by any jump or call instruction.
d) DCR must not straddle the border between functions.

Anyway, these limitations are checked by the in-kernel instruction
decoder, so you don't need to worry about that.

Probe Overhead
==============

On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0
microseconds to process.  Specifically, a benchmark that hits the same
probepoint repeatedly, firing a simple handler each time, reports 1-2
million hits per second, depending on the architecture.  A return-probe
hit typically takes 50-75% longer than a kprobe hit.
When you have a return probe set on a function, adding a kprobe at
the entry to that function adds essentially no overhead.

Here are sample overhead figures (in usec) for different architectures::

  k = kprobe; r = return probe; kr = kprobe + return probe
  on same function

  i386: Intel Pentium M, 1495 MHz, 2957.31 bogomips
  k = 0.57 usec; r = 0.92; kr = 0.99

  x86_64: AMD Opteron 246, 1994 MHz, 3971.48 bogomips
  k = 0.49 usec; r = 0.80; kr = 0.82

  ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 1 virtual CPU per physical CPU)
  k = 0.77 usec; r = 1.26; kr = 1.45

Optimized Probe Overhead
------------------------

Typically, an optimized kprobe hit takes 0.07 to 0.1 microseconds to
process. Here are sample overhead figures (in usec) for x86 architectures::

  k = unoptimized kprobe, b = boosted (single-step skipped), o = optimized kprobe,
  r = unoptimized kretprobe, rb = boosted kretprobe, ro = optimized kretprobe.

  i386: Intel(R) Xeon(R) E5410, 2.33GHz, 4656.90 bogomips
  k = 0.80 usec; b = 0.33; o = 0.05; r = 1.10; rb = 0.61; ro = 0.33

  x86-64: Intel(R) Xeon(R) E5410, 2.33GHz, 4656.90 bogomips
  k = 0.99 usec; b = 0.43; o = 0.06; r = 1.24; rb = 0.68; ro = 0.30

TODO
====

a. SystemTap (http://sourceware.org/systemtap): Provides a simplified
   programming interface for probe-based instrumentation.  Try it out.
b. Kernel return probes for sparc64.
c. Support for other architectures.
d. User-space probes.
e. Watchpoint probes (which fire on data references).

Kprobes Example
===============

See samples/kprobes/kprobe_example.c

Kretprobes Example
==================

See samples/kprobes/kretprobe_example.c

Deprecated Features
===================

Jprobes is now a deprecated feature. People who are depending on it should
migrate to other tracing features or use older kernels. Please consider to
migrate your tool to one of the following options:

- Use trace-event to trace target function with arguments.

  trace-event is a low-overhead (and almost no visible overhead if it
  is off) statically defined event interface. You can define new events
  and trace it via ftrace or any other tracing tools.

  See the following urls:

    - https://lwn.net/Articles/379903/
    - https://lwn.net/Articles/381064/
    - https://lwn.net/Articles/383362/

- Use ftrace dynamic events (kprobe event) with perf-probe.

  If you build your kernel with debug info (CONFIG_DEBUG_INFO=y), you can
  find which register/stack is assigned to which local variable or arguments
  by using perf-probe and set up new event to trace it.

  See following documents:

  - Documentation/trace/kprobetrace.rst
  - Documentation/trace/events.rst
  - tools/perf/Documentation/perf-probe.txt

The kprobes debugfs interface
=============================

With recent kernels (> 2.6.20) the list of registered kprobes is visible
under the /sys/kernel/debug/kprobes/ directory (assuming debugfs is mounted at //sys/kernel/debug).

/sys/kernel/debug/kprobes/list: Lists all registered probes on the system::

	c015d71a  k  vfs_read+0x0
	c03dedc5  r  tcp_v4_rcv+0x0

The first column provides the kernel address where the probe is inserted.
The second column identifies the type of probe (k - kprobe and r - kretprobe)
while the third column specifies the symbol+offset of the probe.
If the probed function belongs to a module, the module name is also
specified. Following columns show probe status. If the probe is on
a virtual address that is no longer valid (module init sections, module
virtual addresses that correspond to modules that've been unloaded),
such probes are marked with [GONE]. If the probe is temporarily disabled,
such probes are marked with [DISABLED]. If the probe is optimized, it is
marked with [OPTIMIZED]. If the probe is ftrace-based, it is marked with
[FTRACE].

/sys/kernel/debug/kprobes/enabled: Turn kprobes ON/OFF forcibly.

Provides a knob to globally and forcibly turn registered kprobes ON or OFF.
By default, all kprobes are enabled. By echoing "0" to this file, all
registered probes will be disarmed, till such time a "1" is echoed to this
file. Note that this knob just disarms and arms all kprobes and doesn't
change each probe's disabling state. This means that disabled kprobes (marked
[DISABLED]) will be not enabled if you turn ON all kprobes by this knob.

The kprobes sysctl interface
============================

/proc/sys/debug/kprobes-optimization: Turn kprobes optimization ON/OFF.

When CONFIG_OPTPROBES=y, this sysctl interface appears and it provides
a knob to globally and forcibly turn jump optimization (see section
:ref:`kprobes_jump_optimization`) ON or OFF. By default, jump optimization
is allowed (ON). If you echo "0" to this file or set
"debug.kprobes_optimization" to 0 via sysctl, all optimized probes will be
unoptimized, and any new probes registered after that will not be optimized.

Note that this knob *changes* the optimized state. This means that optimized
probes (marked [OPTIMIZED]) will be unoptimized ([OPTIMIZED] tag will be
removed). If the knob is turned on, they will be optimized again.

References
==========

For additional information on Kprobes, refer to the following URLs:

- https://lwn.net/Articles/132196/
- https://www.kernel.org/doc/ols/2006/ols2006v2-pages-109-124.pdf


code:
==========================
Kprobe-based Event Tracing
==========================

:Author: Masami Hiramatsu

Overview
--------
These events are similar to tracepoint based events. Instead of Tracepoint,
this is based on kprobes (kprobe and kretprobe). So it can probe wherever
kprobes can probe (this means, all functions except those with
__kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL).
Unlike the Tracepoint based event, this can be added and removed
dynamically, on the fly.

To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y.

Similar to the events tracer, this doesn't need to be activated via
current_tracer. Instead of that, add probe points via
/sys/kernel/debug/tracing/kprobe_events, and enable it via
/sys/kernel/debug/tracing/events/kprobes/<EVENT>/enable.

You can also use /sys/kernel/debug/tracing/dynamic_events instead of
kprobe_events. That interface will provide unified access to other
dynamic events too.

Synopsis of kprobe_events
-------------------------
::

  p[:[GRP/][EVENT]] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS]	: Set a probe
  r[MAXACTIVE][:[GRP/][EVENT]] [MOD:]SYM[+0] [FETCHARGS]	: Set a return probe
  p[:[GRP/][EVENT]] [MOD:]SYM[+0]%return [FETCHARGS]	: Set a return probe
  -:[GRP/][EVENT]						: Clear a probe

 GRP		: Group name. If omitted, use "kprobes" for it.
 EVENT		: Event name. If omitted, the event name is generated
		  based on SYM+offs or MEMADDR.
 MOD		: Module name which has given SYM.
 SYM[+offs]	: Symbol+offset where the probe is inserted.
 SYM%return	: Return address of the symbol
 MEMADDR	: Address where the probe is inserted.
 MAXACTIVE	: Maximum number of instances of the specified function that
		  can be probed simultaneously, or 0 for the default value
		  as defined in Documentation/trace/kprobes.rst section 1.3.1.

 FETCHARGS	: Arguments. Each probe can have up to 128 args.
  %REG		: Fetch register REG
  @ADDR		: Fetch memory at ADDR (ADDR should be in kernel)
  @SYM[+|-offs]	: Fetch memory at SYM +|- offs (SYM should be a data symbol)
  $stackN	: Fetch Nth entry of stack (N >= 0)
  $stack	: Fetch stack address.
  $argN		: Fetch the Nth function argument. (N >= 1) (\*1)
  $retval	: Fetch return value.(\*2)
  $comm		: Fetch current task comm.
  +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
  \IMM		: Store an immediate value to the argument.
  NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
  FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
		  (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
		  (x8/x16/x32/x64), "string", "ustring" and bitfield
		  are supported.

  (\*1) only for the probe on function entry (offs == 0).
  (\*2) only for return probe.
  (\*3) this is useful for fetching a field of data structures.
  (\*4) "u" means user-space dereference. See :ref:`user_mem_access`.

Types
-----
Several types are supported for fetch-args. Kprobe tracer will access memory
by given type. Prefix 's' and 'u' means those types are signed and unsigned
respectively. 'x' prefix implies it is unsigned. Traced arguments are shown
in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32'
or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and
x86-64 uses x64).
These value types can be an array. To record array data, you can add '[N]'
(where N is a fixed number, less than 64) to the base type.
E.g. 'x16[4]' means an array of x16 (2bytes hex) with 4 elements.
Note that the array can be applied to memory type fetchargs, you can not
apply it to registers/stack-entries etc. (for example, '$stack1:x8[8]' is
wrong, but '+8($stack):x8[8]' is OK.)
String type is a special type, which fetches a "null-terminated" string from
kernel space. This means it will fail and store NULL if the string container
has been paged out. "ustring" type is an alternative of string for user-space.
See :ref:`user_mem_access` for more info..
The string array type is a bit different from other types. For other base
types, <base-type>[1] is equal to <base-type> (e.g. +0(%di):x32[1] is same
as +0(%di):x32.) But string[1] is not equal to string. The string type itself
represents "char array", but string array type represents "char * array".
So, for example, +0(%di):string[1] is equal to +0(+0(%di)):string.
Bitfield is another special type, which takes 3 parameters, bit-width, bit-
offset, and container-size (usually 32). The syntax is::

 b<bit-width>@<bit-offset>/<container-size>

Symbol type('symbol') is an alias of u32 or u64 type (depends on BITS_PER_LONG)
which shows given pointer in "symbol+offset" style.
For $comm, the default type is "string"; any other type is invalid.

.. _user_mem_access:

User Memory Access
------------------
Kprobe events supports user-space memory access. For that purpose, you can use
either user-space dereference syntax or 'ustring' type.

The user-space dereference syntax allows you to access a field of a data
structure in user-space. This is done by adding the "u" prefix to the
dereference syntax. For example, +u4(%si) means it will read memory from the
address in the register %si offset by 4, and the memory is expected to be in
user-space. You can use this for strings too, e.g. +u0(%si):string will read
a string from the address in the register %si that is expected to be in user-
space. 'ustring' is a shortcut way of performing the same task. That is,
+0(%si):ustring is equivalent to +u0(%si):string.

Note that kprobe-event provides the user-memory access syntax but it doesn't
use it transparently. This means if you use normal dereference or string type
for user memory, it might fail, and may always fail on some archs. The user
has to carefully check if the target data is in kernel or user space.

Per-Probe Event Filtering
-------------------------
Per-probe event filtering feature allows you to set different filter on each
probe and gives you what arguments will be shown in trace buffer. If an event
name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event
under tracing/events/kprobes/<EVENT>, at the directory you can see 'id',
'enable', 'format', 'filter' and 'trigger'.

enable:
  You can enable/disable the probe by writing 1 or 0 on it.

format:
  This shows the format of this probe event.

filter:
  You can write filtering rules of this event.

id:
  This shows the id of this probe event.

trigger:
  This allows to install trigger commands which are executed when the event is
  hit (for details, see Documentation/trace/events.rst, section 6).

Event Profiling
---------------
You can check the total number of probe hits and probe miss-hits via
/sys/kernel/debug/tracing/kprobe_profile.
The first column is event name, the second is the number of probe hits,
the third is the number of probe miss-hits.

Kernel Boot Parameter
---------------------
You can add and enable new kprobe events when booting up the kernel by
"kprobe_event=" parameter. The parameter accepts a semicolon-delimited
kprobe events, which format is similar to the kprobe_events.
The difference is that the probe definition parameters are comma-delimited
instead of space. For example, adding myprobe event on do_sys_open like below

  p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)

should be below for kernel boot parameter (just replace spaces with comma)

  p:myprobe,do_sys_open,dfd=%ax,filename=%dx,flags=%cx,mode=+4($stack)

Usage examples
--------------
To add a probe as a new event, write a new definition to kprobe_events
as below::

  echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events

This sets a kprobe on the top of do_sys_open() function with recording
1st to 4th arguments as "myprobe" event. Note, which register/stack entry is
assigned to each function argument depends on arch-specific ABI. If you unsure
the ABI, please try to use probe subcommand of perf-tools (you can find it
under tools/perf/).
As this example shows, users can choose more familiar names for each arguments.
::

  echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events

This sets a kretprobe on the return point of do_sys_open() function with
recording return value as "myretprobe" event.
You can see the format of these events via
/sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
::

  cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
  name: myprobe
  ID: 780
  format:
          field:unsigned short common_type;       offset:0;       size:2; signed:0;
          field:unsigned char common_flags;       offset:2;       size:1; signed:0;
          field:unsigned char common_preempt_count;       offset:3; size:1;signed:0;
          field:int common_pid;   offset:4;       size:4; signed:1;

          field:unsigned long __probe_ip; offset:12;      size:4; signed:0;
          field:int __probe_nargs;        offset:16;      size:4; signed:1;
          field:unsigned long dfd;        offset:20;      size:4; signed:0;
          field:unsigned long filename;   offset:24;      size:4; signed:0;
          field:unsigned long flags;      offset:28;      size:4; signed:0;
          field:unsigned long mode;       offset:32;      size:4; signed:0;

  print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip,
  REC->dfd, REC->filename, REC->flags, REC->mode

You can see that the event has 4 arguments as in the expressions you specified.
::

  echo > /sys/kernel/debug/tracing/kprobe_events

This clears all probe points.

Or,
::

  echo -:myprobe >> kprobe_events

This clears probe points selectively.

Right after definition, each event is disabled by default. For tracing these
events, you need to enable it.
::

  echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
  echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable

Use the following command to start tracing in an interval.
::

    # echo 1 > tracing_on
    Open something...
    # echo 0 > tracing_on

And you can see the traced information via /sys/kernel/debug/tracing/trace.
::

  cat /sys/kernel/debug/tracing/trace
  # tracer: nop
  #
  #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
  #              | |       |          |         |
             <...>-1447  [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
             <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe
             <...>-1447  [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
             <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
             <...>-1447  [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
             <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3

Each line shows when the kernel hits an event, and <- SYMBOL means kernel
returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
returns from do_sys_open to sys_open+0x1b).


code:
===================================
In-kernel memory-mapped I/O tracing
===================================

Home page and links to optional user space tools:

	https://nouveau.freedesktop.org/wiki/MmioTrace

MMIO tracing was originally developed by Intel around 2003 for their Fault
Injection Test Harness. In Dec 2006 - Jan 2007, using the code from Intel,
Jeff Muizelaar created a tool for tracing MMIO accesses with the Nouveau
project in mind. Since then many people have contributed.

Mmiotrace was built for reverse engineering any memory-mapped IO device with
the Nouveau project as the first real user. Only x86 and x86_64 architectures
are supported.

Out-of-tree mmiotrace was originally modified for mainline inclusion and
ftrace framework by Pekka Paalanen <pq@iki.fi>.

Preparation
-----------

Mmiotrace feature is compiled in by the CONFIG_MMIOTRACE option. Tracing is
disabled by default, so it is safe to have this set to yes. SMP systems are
supported, but tracing is unreliable and may miss events if more than one CPU
is on-line, therefore mmiotrace takes all but one CPU off-line during run-time
activation. You can re-enable CPUs by hand, but you have been warned, there
is no way to automatically detect if you are losing events due to CPUs racing.

Usage Quick Reference
---------------------
::

	$ mount -t debugfs debugfs /sys/kernel/debug
	$ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer
	$ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &
	Start X or whatever.
	$ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
	$ echo nop > /sys/kernel/debug/tracing/current_tracer
	Check for lost events.

Usage
-----

Make sure debugfs is mounted to /sys/kernel/debug.
If not (requires root privileges)::

	$ mount -t debugfs debugfs /sys/kernel/debug

Check that the driver you are about to trace is not loaded.

Activate mmiotrace (requires root privileges)::

	$ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer

Start storing the trace::

	$ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &

The 'cat' process should stay running (sleeping) in the background.

Load the driver you want to trace and use it. Mmiotrace will only catch MMIO
accesses to areas that are ioremapped while mmiotrace is active.

During tracing you can place comments (markers) into the trace by
$ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
This makes it easier to see which part of the (huge) trace corresponds to
which action. It is recommended to place descriptive markers about what you
do.

Shut down mmiotrace (requires root privileges)::

	$ echo nop > /sys/kernel/debug/tracing/current_tracer

The 'cat' process exits. If it does not, kill it by issuing 'fg' command and
pressing ctrl+c.

Check that mmiotrace did not lose events due to a buffer filling up. Either::

	$ grep -i lost mydump.txt

which tells you exactly how many events were lost, or use::

	$ dmesg

to view your kernel log and look for "mmiotrace has lost events" warning. If
events were lost, the trace is incomplete. You should enlarge the buffers and
try again. Buffers are enlarged by first seeing how large the current buffers
are::

	$ cat /sys/kernel/debug/tracing/buffer_size_kb

gives you a number. Approximately double this number and write it back, for
instance::

	$ echo 128000 > /sys/kernel/debug/tracing/buffer_size_kb

Then start again from the top.

If you are doing a trace for a driver project, e.g. Nouveau, you should also
do the following before sending your results::

	$ lspci -vvv > lspci.txt
	$ dmesg > dmesg.txt
	$ tar zcf pciid-nick-mmiotrace.tar.gz mydump.txt lspci.txt dmesg.txt

and then send the .tar.gz file. The trace compresses considerably. Replace
"pciid" and "nick" with the PCI ID or model name of your piece of hardware
under investigation and your nickname.

How Mmiotrace Works
-------------------

Access to hardware IO-memory is gained by mapping addresses from PCI bus by
calling one of the ioremap_*() functions. Mmiotrace is hooked into the
__ioremap() function and gets called whenever a mapping is created. Mapping is
an event that is recorded into the trace log. Note that ISA range mappings
are not caught, since the mapping always exists and is returned directly.

MMIO accesses are recorded via page faults. Just before __ioremap() returns,
the mapped pages are marked as not present. Any access to the pages causes a
fault. The page fault handler calls mmiotrace to handle the fault. Mmiotrace
marks the page present, sets TF flag to achieve single stepping and exits the
fault handler. The instruction that faulted is executed and debug trap is
entered. Here mmiotrace again marks the page as not present. The instruction
is decoded to get the type of operation (read/write), data width and the value
read or written. These are stored to the trace log.

Setting the page present in the page fault handler has a race condition on SMP
machines. During the single stepping other CPUs may run freely on that page
and events can be missed without a notice. Re-enabling other CPUs during
tracing is discouraged.

Trace Log Format
----------------

The raw log is text and easily filtered with e.g. grep and awk. One record is
one line in the log. A record starts with a keyword, followed by keyword-
dependent arguments. Arguments are separated by a space, or continue until the
end of line. The format for version 20070824 is as follows:

Explanation	Keyword	Space-separated arguments
---------------------------------------------------------------------------

read event	R	width, timestamp, map id, physical, value, PC, PID
write event	W	width, timestamp, map id, physical, value, PC, PID
ioremap event	MAP	timestamp, map id, physical, virtual, length, PC, PID
iounmap event	UNMAP	timestamp, map id, PC, PID
marker		MARK	timestamp, text
version		VERSION	the string "20070824"
info for reader	LSPCI	one line from lspci -v
PCI address map	PCIDEV	space-separated /proc/bus/pci/devices data
unk. opcode	UNKNOWN	timestamp, map id, physical, data, PC, PID

Timestamp is in seconds with decimals. Physical is a PCI bus address, virtual
is a kernel virtual address. Width is the data width in bytes and value is the
data value. Map id is an arbitrary id number identifying the mapping that was
used in an operation. PC is the program counter and PID is process id. PC is
zero if it is not recorded. PID is always zero as tracing MMIO accesses
originating in user space memory is not yet supported.

For instance, the following awk filter will pass all 32-bit writes that target
physical addresses in the range [0xfb73ce40, 0xfb800000]
::

	$ awk '/W 4 / { adr=strtonum($5); if (adr >= 0xfb73ce40 &&
	adr < 0xfb800000) print; }'

Tools for Developers
--------------------

The user space tools include utilities for:
  - replacing numeric addresses and values with hardware register names
  - replaying MMIO logs, i.e., re-executing the recorded writes


code:
==============
OSNOISE Tracer
==============

In the context of high-performance computing (HPC), the Operating System
Noise (*osnoise*) refers to the interference experienced by an application
due to activities inside the operating system. In the context of Linux,
NMIs, IRQs, SoftIRQs, and any other system thread can cause noise to the
system. Moreover, hardware-related jobs can also cause noise, for example,
via SMIs.

hwlat_detector is one of the tools used to identify the most complex
source of noise: *hardware noise*.

In a nutshell, the hwlat_detector creates a thread that runs
periodically for a given period. At the beginning of a period, the thread
disables interrupt and starts sampling. While running, the hwlatd
thread reads the time in a loop. As interrupts are disabled, threads,
IRQs, and SoftIRQs cannot interfere with the hwlatd thread. Hence, the
cause of any gap between two different reads of the time roots either on
NMI or in the hardware itself. At the end of the period, hwlatd enables
interrupts and reports the max observed gap between the reads. It also
prints a NMI occurrence counter. If the output does not report NMI
executions, the user can conclude that the hardware is the culprit for
the latency. The hwlat detects the NMI execution by observing
the entry and exit of a NMI.

The osnoise tracer leverages the hwlat_detector by running a
similar loop with preemption, SoftIRQs and IRQs enabled, thus allowing
all the sources of *osnoise* during its execution. Using the same approach
of hwlat, osnoise takes note of the entry and exit point of any
source of interferences, increasing a per-cpu interference counter. The
osnoise tracer also saves an interference counter for each source of
interference. The interference counter for NMI, IRQs, SoftIRQs, and
threads is increased anytime the tool observes these interferences' entry
events. When a noise happens without any interference from the operating
system level, the hardware noise counter increases, pointing to a
hardware-related noise. In this way, osnoise can account for any
source of interference. At the end of the period, the osnoise tracer
prints the sum of all noise, the max single noise, the percentage of CPU
available for the thread, and the counters for the noise sources.

Usage
-----

Write the ASCII text "osnoise" into the current_tracer file of the
tracing system (generally mounted at /sys/kernel/tracing).

For example::

        [root@f32 ~]# cd /sys/kernel/tracing/
        [root@f32 tracing]# echo osnoise > current_tracer

It is possible to follow the trace by reading the trace file::

        [root@f32 tracing]# cat trace
        # tracer: osnoise
        #
        #                                _-----=> irqs-off
        #                               / _----=> need-resched
        #                              | / _---=> hardirq/softirq
        #                              || / _--=> preempt-depth                            MAX
        #                              || /                                             SINGLE     Interference counters:
        #                              ||||               RUNTIME      NOISE   % OF CPU  NOISE    +-----------------------------+
        #           TASK-PID      CPU# ||||   TIMESTAMP    IN US       IN US  AVAILABLE  IN US     HW    NMI    IRQ   SIRQ THREAD
        #              | |         |   ||||      |           |             |    |            |      |      |      |      |      |
                   <...>-859     [000] ....    81.637220: 1000000        190  99.98100       9     18      0   1007     18      1
                   <...>-860     [001] ....    81.638154: 1000000        656  99.93440      74     23      0   1006     16      3
                   <...>-861     [002] ....    81.638193: 1000000       5675  99.43250     202      6      0   1013     25     21
                   <...>-862     [003] ....    81.638242: 1000000        125  99.98750      45      1      0   1011     23      0
                   <...>-863     [004] ....    81.638260: 1000000       1721  99.82790     168      7      0   1002     49     41
                   <...>-864     [005] ....    81.638286: 1000000        263  99.97370      57      6      0   1006     26      2
                   <...>-865     [006] ....    81.638302: 1000000        109  99.98910      21      3      0   1006     18      1
                   <...>-866     [007] ....    81.638326: 1000000       7816  99.21840     107      8      0   1016     39     19

In addition to the regular trace fields (from TASK-PID to TIMESTAMP), the
tracer prints a message at the end of each period for each CPU that is
running an osnoise/ thread. The osnoise specific fields report:

 - The RUNTIME IN US reports the amount of time in microseconds that
   the osnoise thread kept looping reading the time.
 - The NOISE IN US reports the sum of noise in microseconds observed
   by the osnoise tracer during the associated runtime.
 - The % OF CPU AVAILABLE reports the percentage of CPU available for
   the osnoise thread during the runtime window.
 - The MAX SINGLE NOISE IN US reports the maximum single noise observed
   during the runtime window.
 - The Interference counters display how many each of the respective
   interference happened during the runtime window.

Note that the example above shows a high number of HW noise samples.
The reason being is that this sample was taken on a virtual machine,
and the host interference is detected as a hardware interference.

Tracer Configuration
--------------------

The tracer has a set of options inside the osnoise directory, they are:

 - osnoise/cpus: CPUs at which a osnoise thread will execute.
 - osnoise/period_us: the period of the osnoise thread.
 - osnoise/runtime_us: how long an osnoise thread will look for noise.
 - osnoise/stop_tracing_us: stop the system tracing if a single noise
   higher than the configured value happens. Writing 0 disables this
   option.
 - osnoise/stop_tracing_total_us: stop the system tracing if total noise
   higher than the configured value happens. Writing 0 disables this
   option.
 - tracing_threshold: the minimum delta between two time() reads to be
   considered as noise, in us. When set to 0, the default value will
   be used, which is currently 5 us.
 - osnoise/options: a set of on/off options that can be enabled by
   writing the option name to the file or disabled by writing the option
   name preceded with the 'NO\_' prefix. For example, writing
   NO_OSNOISE_WORKLOAD disables the OSNOISE_WORKLOAD option. The
   special DEAFAULTS option resets all options to the default value.

Tracer Options
--------------

The osnoise/options file exposes a set of on/off configuration options for
the osnoise tracer. These options are:

 - DEFAULTS: reset the options to the default value.
 - OSNOISE_WORKLOAD: do not dispatch osnoise workload (see dedicated
   section below).
 - PANIC_ON_STOP: call panic() if the tracer stops. This option serves to
   capture a vmcore.
 - OSNOISE_PREEMPT_DISABLE: disable preemption while running the osnoise
   workload, allowing only IRQ and hardware-related noise.
 - OSNOISE_IRQ_DISABLE: disable IRQs while running the osnoise workload,
   allowing only NMIs and hardware-related noise, like hwlat tracer.

Additional Tracing
------------------

In addition to the tracer, a set of tracepoints were added to
facilitate the identification of the osnoise source.

 - osnoise:sample_threshold: printed anytime a noise is higher than
   the configurable tolerance_ns.
 - osnoise:nmi_noise: noise from NMI, including the duration.
 - osnoise:irq_noise: noise from an IRQ, including the duration.
 - osnoise:softirq_noise: noise from a SoftIRQ, including the
   duration.
 - osnoise:thread_noise: noise from a thread, including the duration.

Note that all the values are *net values*. For example, if while osnoise
is running, another thread preempts the osnoise thread, it will start a
thread_noise duration at the start. Then, an IRQ takes place, preempting
the thread_noise, starting a irq_noise. When the IRQ ends its execution,
it will compute its duration, and this duration will be subtracted from
the thread_noise, in such a way as to avoid the double accounting of the
IRQ execution. This logic is valid for all sources of noise.

Here is one example of the usage of these tracepoints::

       osnoise/8-961     [008] d.h.  5789.857532: irq_noise: local_timer:236 start 5789.857529929 duration 1845 ns
       osnoise/8-961     [008] dNh.  5789.858408: irq_noise: local_timer:236 start 5789.858404871 duration 2848 ns
     migration/8-54      [008] d...  5789.858413: thread_noise: migration/8:54 start 5789.858409300 duration 3068 ns
       osnoise/8-961     [008] ....  5789.858413: sample_threshold: start 5789.858404555 duration 8812 ns interferences 2

In this example, a noise sample of 8 microseconds was reported in the last
line, pointing to two interferences. Looking backward in the trace, the
two previous entries were about the migration thread running after a
timer IRQ execution. The first event is not part of the noise because
it took place one millisecond before.

It is worth noticing that the sum of the duration reported in the
tracepoints is smaller than eight us reported in the sample_threshold.
The reason roots in the overhead of the entry and exit code that happens
before and after any interference execution. This justifies the dual
approach: measuring thread and tracing.

Running osnoise tracer without workload
---------------------------------------

By enabling the osnoise tracer with the NO_OSNOISE_WORKLOAD option set,
the osnoise: tracepoints serve to measure the execution time of
any type of Linux task, free from the interference of other tasks.


code:
#!/usr/bin/env python
# add symbolic names to read_msr / write_msr in trace
# decode_msr msr-index.h < trace
import sys
import re

msrs = dict()

with open(sys.argv[1] if len(sys.argv) > 1 else "msr-index.h", "r") as f:
	for j in f:
		m = re.match(r'#define (MSR_\w+)\s+(0x[0-9a-fA-F]+)', j)
		if m:
			msrs[int(m.group(2), 16)] = m.group(1)

extra_ranges = (
	( "MSR_LASTBRANCH_%d_FROM_IP", 0x680, 0x69F ),
	( "MSR_LASTBRANCH_%d_TO_IP", 0x6C0, 0x6DF ),
	( "LBR_INFO_%d", 0xdc0, 0xddf ),
)

for j in sys.stdin:
	m = re.search(r'(read|write)_msr:\s+([0-9a-f]+)', j)
	if m:
		r = None
		num = int(m.group(2), 16)
		if num in msrs:
			r = msrs[num]
		else:
			for er in extra_ranges:
				if er[1] <= num <= er[2]:
					r = er[0] % (num - er[1],)
					break
		if r:
			j = j.replace(" " + m.group(2), " " + r + "(" + m.group(2) + ")")
	print j,


code:
#!/usr/bin/env perl
# This is a POC (proof of concept or piece of crap, take your pick) for reading the
# text representation of trace output related to page allocation. It makes an attempt
# to extract some high-level information on what is going on. The accuracy of the parser
# may vary considerably
#
# Example usage: trace-pagealloc-postprocess.pl < /sys/kernel/debug/tracing/trace_pipe
# other options
#   --prepend-parent	Report on the parent proc and PID
#   --read-procstat	If the trace lacks process info, get it from /proc
#   --ignore-pid	Aggregate processes of the same name together
#
# Copyright (c) IBM Corporation 2009
# Author: Mel Gorman <mel@csn.ul.ie>
use strict;
use Getopt::Long;

# Tracepoint events
use constant MM_PAGE_ALLOC		=> 1;
use constant MM_PAGE_FREE		=> 2;
use constant MM_PAGE_FREE_BATCHED	=> 3;
use constant MM_PAGE_PCPU_DRAIN		=> 4;
use constant MM_PAGE_ALLOC_ZONE_LOCKED	=> 5;
use constant MM_PAGE_ALLOC_EXTFRAG	=> 6;
use constant EVENT_UNKNOWN		=> 7;

# Constants used to track state
use constant STATE_PCPU_PAGES_DRAINED	=> 8;
use constant STATE_PCPU_PAGES_REFILLED	=> 9;

# High-level events extrapolated from tracepoints
use constant HIGH_PCPU_DRAINS		=> 10;
use constant HIGH_PCPU_REFILLS		=> 11;
use constant HIGH_EXT_FRAGMENT		=> 12;
use constant HIGH_EXT_FRAGMENT_SEVERE	=> 13;
use constant HIGH_EXT_FRAGMENT_MODERATE	=> 14;
use constant HIGH_EXT_FRAGMENT_CHANGED	=> 15;

my %perprocesspid;
my %perprocess;
my $opt_ignorepid;
my $opt_read_procstat;
my $opt_prepend_parent;

# Catch sigint and exit on request
my $sigint_report = 0;
my $sigint_exit = 0;
my $sigint_pending = 0;
my $sigint_received = 0;
sub sigint_handler {
	my $current_time = time;
	if ($current_time - 2 > $sigint_received) {
		print "SIGINT received, report pending. Hit ctrl-c again to exit\n";
		$sigint_report = 1;
	} else {
		if (!$sigint_exit) {
			print "Second SIGINT received quickly, exiting\n";
		}
		$sigint_exit++;
	}

	if ($sigint_exit > 3) {
		print "Many SIGINTs received, exiting now without report\n";
		exit;
	}

	$sigint_received = $current_time;
	$sigint_pending = 1;
}
$SIG{INT} = "sigint_handler";

# Parse command line options
GetOptions(
	'ignore-pid'	 =>	\$opt_ignorepid,
	'read-procstat'	 =>	\$opt_read_procstat,
	'prepend-parent' =>	\$opt_prepend_parent,
);

# Defaults for dynamically discovered regex's
my $regex_fragdetails_default = 'page=([0-9a-f]*) pfn=([0-9]*) alloc_order=([-0-9]*) fallback_order=([-0-9]*) pageblock_order=([-0-9]*) alloc_migratetype=([-0-9]*) fallback_migratetype=([-0-9]*) fragmenting=([-0-9]) change_ownership=([-0-9])';

# Dyanically discovered regex
my $regex_fragdetails;

# Static regex used. Specified like this for readability and for use with /o
#                      (process_pid)     (cpus      )   ( time  )   (tpoint    ) (details)
my $regex_traceevent = '\s*([a-zA-Z0-9-]*)\s*(\[[0-9]*\])\s*([0-9.]*):\s*([a-zA-Z_]*):\s*(.*)';
my $regex_statname = '[-0-9]*\s\((.*)\).*';
my $regex_statppid = '[-0-9]*\s\(.*\)\s[A-Za-z]\s([0-9]*).*';

sub generate_traceevent_regex {
	my $event = shift;
	my $default = shift;
	my $regex;

	# Read the event format or use the default
	if (!open (FORMAT, "/sys/kernel/debug/tracing/events/$event/format")) {
		$regex = $default;
	} else {
		my $line;
		while (!eof(FORMAT)) {
			$line = <FORMAT>;
			if ($line =~ /^print fmt:\s"(.*)",.*/) {
				$regex = $1;
				$regex =~ s/%p/\([0-9a-f]*\)/g;
				$regex =~ s/%d/\([-0-9]*\)/g;
				$regex =~ s/%lu/\([0-9]*\)/g;
			}
		}
	}

	# Verify fields are in the right order
	my $tuple;
	foreach $tuple (split /\s/, $regex) {
		my ($key, $value) = split(/=/, $tuple);
		my $expected = shift;
		if ($key ne $expected) {
			print("WARNING: Format not as expected '$key' != '$expected'");
			$regex =~ s/$key=\((.*)\)/$key=$1/;
		}
	}

	if (defined shift) {
		die("Fewer fields than expected in format");
	}

	return $regex;
}
$regex_fragdetails = generate_traceevent_regex("kmem/mm_page_alloc_extfrag",
			$regex_fragdetails_default,
			"page", "pfn",
			"alloc_order", "fallback_order", "pageblock_order",
			"alloc_migratetype", "fallback_migratetype",
			"fragmenting", "change_ownership");

sub read_statline($) {
	my $pid = $_[0];
	my $statline;

	if (open(STAT, "/proc/$pid/stat")) {
		$statline = <STAT>;
		close(STAT);
	}

	if ($statline eq '') {
		$statline = "-1 (UNKNOWN_PROCESS_NAME) R 0";
	}

	return $statline;
}

sub guess_process_pid($$) {
	my $pid = $_[0];
	my $statline = $_[1];

	if ($pid == 0) {
		return "swapper-0";
	}

	if ($statline !~ /$regex_statname/o) {
		die("Failed to math stat line for process name :: $statline");
	}
	return "$1-$pid";
}

sub parent_info($$) {
	my $pid = $_[0];
	my $statline = $_[1];
	my $ppid;

	if ($pid == 0) {
		return "NOPARENT-0";
	}

	if ($statline !~ /$regex_statppid/o) {
		die("Failed to match stat line process ppid:: $statline");
	}

	# Read the ppid stat line
	$ppid = $1;
	return guess_process_pid($ppid, read_statline($ppid));
}

sub process_events {
	my $traceevent;
	my $process_pid;
	my $cpus;
	my $timestamp;
	my $tracepoint;
	my $details;
	my $statline;

	# Read each line of the event log
EVENT_PROCESS:
	while ($traceevent = <STDIN>) {
		if ($traceevent =~ /$regex_traceevent/o) {
			$process_pid = $1;
			$tracepoint = $4;

			if ($opt_read_procstat || $opt_prepend_parent) {
				$process_pid =~ /(.*)-([0-9]*)$/;
				my $process = $1;
				my $pid = $2;

				$statline = read_statline($pid);

				if ($opt_read_procstat && $process eq '') {
					$process_pid = guess_process_pid($pid, $statline);
				}

				if ($opt_prepend_parent) {
					$process_pid = parent_info($pid, $statline) . " :: $process_pid";
				}
			}

			# Unnecessary in this script. Uncomment if required
			# $cpus = $2;
			# $timestamp = $3;
		} else {
			next;
		}

		# Perl Switch() sucks majorly
		if ($tracepoint eq "mm_page_alloc") {
			$perprocesspid{$process_pid}->{MM_PAGE_ALLOC}++;
		} elsif ($tracepoint eq "mm_page_free") {
			$perprocesspid{$process_pid}->{MM_PAGE_FREE}++
		} elsif ($tracepoint eq "mm_page_free_batched") {
			$perprocesspid{$process_pid}->{MM_PAGE_FREE_BATCHED}++;
		} elsif ($tracepoint eq "mm_page_pcpu_drain") {
			$perprocesspid{$process_pid}->{MM_PAGE_PCPU_DRAIN}++;
			$perprocesspid{$process_pid}->{STATE_PCPU_PAGES_DRAINED}++;
		} elsif ($tracepoint eq "mm_page_alloc_zone_locked") {
			$perprocesspid{$process_pid}->{MM_PAGE_ALLOC_ZONE_LOCKED}++;
			$perprocesspid{$process_pid}->{STATE_PCPU_PAGES_REFILLED}++;
		} elsif ($tracepoint eq "mm_page_alloc_extfrag") {

			# Extract the details of the event now
			$details = $5;

			my ($page, $pfn);
			my ($alloc_order, $fallback_order, $pageblock_order);
			my ($alloc_migratetype, $fallback_migratetype);
			my ($fragmenting, $change_ownership);

			if ($details !~ /$regex_fragdetails/o) {
				print "WARNING: Failed to parse mm_page_alloc_extfrag as expected\n";
				next;
			}

			$perprocesspid{$process_pid}->{MM_PAGE_ALLOC_EXTFRAG}++;
			$page = $1;
			$pfn = $2;
			$alloc_order = $3;
			$fallback_order = $4;
			$pageblock_order = $5;
			$alloc_migratetype = $6;
			$fallback_migratetype = $7;
			$fragmenting = $8;
			$change_ownership = $9;

			if ($fragmenting) {
				$perprocesspid{$process_pid}->{HIGH_EXT_FRAG}++;
				if ($fallback_order <= 3) {
					$perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_SEVERE}++;
				} else {
					$perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_MODERATE}++;
				}
			}
			if ($change_ownership) {
				$perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_CHANGED}++;
			}
		} else {
			$perprocesspid{$process_pid}->{EVENT_UNKNOWN}++;
		}

		# Catch a full pcpu drain event
		if ($perprocesspid{$process_pid}->{STATE_PCPU_PAGES_DRAINED} &&
				$tracepoint ne "mm_page_pcpu_drain") {

			$perprocesspid{$process_pid}->{HIGH_PCPU_DRAINS}++;
			$perprocesspid{$process_pid}->{STATE_PCPU_PAGES_DRAINED} = 0;
		}

		# Catch a full pcpu refill event
		if ($perprocesspid{$process_pid}->{STATE_PCPU_PAGES_REFILLED} &&
				$tracepoint ne "mm_page_alloc_zone_locked") {
			$perprocesspid{$process_pid}->{HIGH_PCPU_REFILLS}++;
			$perprocesspid{$process_pid}->{STATE_PCPU_PAGES_REFILLED} = 0;
		}

		if ($sigint_pending) {
			last EVENT_PROCESS;
		}
	}
}

sub dump_stats {
	my $hashref = shift;
	my %stats = %$hashref;

	# Dump per-process stats
	my $process_pid;
	my $max_strlen = 0;

	# Get the maximum process name
	foreach $process_pid (keys %perprocesspid) {
		my $len = length($process_pid);
		if ($len > $max_strlen) {
			$max_strlen = $len;
		}
	}
	$max_strlen += 2;

	printf("\n");
	printf("%-" . $max_strlen . "s %8s %10s   %8s %8s   %8s %8s %8s   %8s %8s %8s %8s %8s %8s\n",
		"Process", "Pages",  "Pages",      "Pages", "Pages", "PCPU",  "PCPU",   "PCPU",    "Fragment",  "Fragment", "MigType", "Fragment", "Fragment", "Unknown");
	printf("%-" . $max_strlen . "s %8s %10s   %8s %8s   %8s %8s %8s   %8s %8s %8s %8s %8s %8s\n",
		"details", "allocd", "allocd",     "freed", "freed", "pages", "drains", "refills", "Fallback", "Causing",   "Changed", "Severe", "Moderate", "");

	printf("%-" . $max_strlen . "s %8s %10s   %8s %8s   %8s %8s %8s   %8s %8s %8s %8s %8s %8s\n",
		"",        "",       "under lock", "direct", "pagevec", "drain", "", "", "", "", "", "", "", "");

	foreach $process_pid (keys %stats) {
		# Dump final aggregates
		if ($stats{$process_pid}->{STATE_PCPU_PAGES_DRAINED}) {
			$stats{$process_pid}->{HIGH_PCPU_DRAINS}++;
			$stats{$process_pid}->{STATE_PCPU_PAGES_DRAINED} = 0;
		}
		if ($stats{$process_pid}->{STATE_PCPU_PAGES_REFILLED}) {
			$stats{$process_pid}->{HIGH_PCPU_REFILLS}++;
			$stats{$process_pid}->{STATE_PCPU_PAGES_REFILLED} = 0;
		}

		printf("%-" . $max_strlen . "s %8d %10d   %8d %8d   %8d %8d %8d   %8d %8d %8d %8d %8d %8d\n",
			$process_pid,
			$stats{$process_pid}->{MM_PAGE_ALLOC},
			$stats{$process_pid}->{MM_PAGE_ALLOC_ZONE_LOCKED},
			$stats{$process_pid}->{MM_PAGE_FREE},
			$stats{$process_pid}->{MM_PAGE_FREE_BATCHED},
			$stats{$process_pid}->{MM_PAGE_PCPU_DRAIN},
			$stats{$process_pid}->{HIGH_PCPU_DRAINS},
			$stats{$process_pid}->{HIGH_PCPU_REFILLS},
			$stats{$process_pid}->{MM_PAGE_ALLOC_EXTFRAG},
			$stats{$process_pid}->{HIGH_EXT_FRAG},
			$stats{$process_pid}->{HIGH_EXT_FRAGMENT_CHANGED},
			$stats{$process_pid}->{HIGH_EXT_FRAGMENT_SEVERE},
			$stats{$process_pid}->{HIGH_EXT_FRAGMENT_MODERATE},
			$stats{$process_pid}->{EVENT_UNKNOWN});
	}
}

sub aggregate_perprocesspid() {
	my $process_pid;
	my $process;
	undef %perprocess;

	foreach $process_pid (keys %perprocesspid) {
		$process = $process_pid;
		$process =~ s/-([0-9])*$//;
		if ($process eq '') {
			$process = "NO_PROCESS_NAME";
		}

		$perprocess{$process}->{MM_PAGE_ALLOC} += $perprocesspid{$process_pid}->{MM_PAGE_ALLOC};
		$perprocess{$process}->{MM_PAGE_ALLOC_ZONE_LOCKED} += $perprocesspid{$process_pid}->{MM_PAGE_ALLOC_ZONE_LOCKED};
		$perprocess{$process}->{MM_PAGE_FREE} += $perprocesspid{$process_pid}->{MM_PAGE_FREE};
		$perprocess{$process}->{MM_PAGE_FREE_BATCHED} += $perprocesspid{$process_pid}->{MM_PAGE_FREE_BATCHED};
		$perprocess{$process}->{MM_PAGE_PCPU_DRAIN} += $perprocesspid{$process_pid}->{MM_PAGE_PCPU_DRAIN};
		$perprocess{$process}->{HIGH_PCPU_DRAINS} += $perprocesspid{$process_pid}->{HIGH_PCPU_DRAINS};
		$perprocess{$process}->{HIGH_PCPU_REFILLS} += $perprocesspid{$process_pid}->{HIGH_PCPU_REFILLS};
		$perprocess{$process}->{MM_PAGE_ALLOC_EXTFRAG} += $perprocesspid{$process_pid}->{MM_PAGE_ALLOC_EXTFRAG};
		$perprocess{$process}->{HIGH_EXT_FRAG} += $perprocesspid{$process_pid}->{HIGH_EXT_FRAG};
		$perprocess{$process}->{HIGH_EXT_FRAGMENT_CHANGED} += $perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_CHANGED};
		$perprocess{$process}->{HIGH_EXT_FRAGMENT_SEVERE} += $perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_SEVERE};
		$perprocess{$process}->{HIGH_EXT_FRAGMENT_MODERATE} += $perprocesspid{$process_pid}->{HIGH_EXT_FRAGMENT_MODERATE};
		$perprocess{$process}->{EVENT_UNKNOWN} += $perprocesspid{$process_pid}->{EVENT_UNKNOWN};
	}
}

sub report() {
	if (!$opt_ignorepid) {
		dump_stats(\%perprocesspid);
	} else {
		aggregate_perprocesspid();
		dump_stats(\%perprocess);
	}
}

# Process events or signals until neither is available
sub signal_loop() {
	my $sigint_processed;
	do {
		$sigint_processed = 0;
		process_events();

		# Handle pending signals if any
		if ($sigint_pending) {
			my $current_time = time;

			if ($sigint_exit) {
				print "Received exit signal\n";
				$sigint_pending = 0;
			}
			if ($sigint_report) {
				if ($current_time >= $sigint_received + 2) {
					report();
					$sigint_report = 0;
					$sigint_pending = 0;
					$sigint_processed = 1;
				}
			}
		}
	} while ($sigint_pending || $sigint_processed);
}

signal_loop();
report();