code:
==========================================
Using the RAM disk block device with Linux
==========================================

.. Contents:

	1) Overview
	2) Kernel Command Line Parameters
	3) Using "rdev"
	4) An Example of Creating a Compressed RAM Disk

1) Overview
-----------

The RAM disk driver is a way to use main system memory as a block device.  It
is required for initrd, an initial filesystem used if you need to load modules
in order to access the root filesystem (see Documentation/admin-guide/initrd.rst).  It can
also be used for a temporary filesystem for crypto work, since the contents
are erased on reboot.

The RAM disk dynamically grows as more space is required. It does this by using
RAM from the buffer cache. The driver marks the buffers it is using as dirty
so that the VM subsystem does not try to reclaim them later.

The RAM disk supports up to 16 RAM disks by default, and can be reconfigured
to support an unlimited number of RAM disks (at your own risk).  Just change
the configuration symbol BLK_DEV_RAM_COUNT in the Block drivers config menu
and (re)build the kernel.

To use RAM disk support with your system, run './MAKEDEV ram' from the /dev
directory.  RAM disks are all major number 1, and start with minor number 0
for /dev/ram0, etc.  If used, modern kernels use /dev/ram0 for an initrd.

The new RAM disk also has the ability to load compressed RAM disk images,
allowing one to squeeze more programs onto an average installation or
rescue floppy disk.

2) Parameters
---------------------------------

2a) Kernel Command Line Parameters

	ramdisk_size=N
		Size of the ramdisk.

This parameter tells the RAM disk driver to set up RAM disks of N k size.  The
default is 4096 (4 MB).

2b) Module parameters

	rd_nr
		/dev/ramX devices created.

	max_part
		Maximum partition number.

	rd_size
		See ramdisk_size.

3) Using "rdev"
---------------

"rdev" is an obsolete, deprecated, antiquated utility that could be used
to set the boot device in a Linux kernel image.

Instead of using rdev, just place the boot device information on the
kernel command line and pass it to the kernel from the bootloader.

You can also pass arguments to the kernel by setting FDARGS in
arch/x86/boot/Makefile and specify in initrd image by setting FDINITRD in
arch/x86/boot/Makefile.

Some of the kernel command line boot options that may apply here are::

  ramdisk_start=N
  ramdisk_size=M

If you make a boot disk that has LILO, then for the above, you would use::

	append = "ramdisk_start=N ramdisk_size=M"

4) An Example of Creating a Compressed RAM Disk
-----------------------------------------------

To create a RAM disk image, you will need a spare block device to
construct it on. This can be the RAM disk device itself, or an
unused disk partition (such as an unmounted swap partition). For this
example, we will use the RAM disk device, "/dev/ram0".

Note: This technique should not be done on a machine with less than 8 MB
of RAM. If using a spare disk partition instead of /dev/ram0, then this
restriction does not apply.

a) Decide on the RAM disk size that you want. Say 2 MB for this example.
   Create it by writing to the RAM disk device. (This step is not currently
   required, but may be in the future.) It is wise to zero out the
   area (esp. for disks) so that maximal compression is achieved for
   the unused blocks of the image that you are about to create::

	dd if=/dev/zero of=/dev/ram0 bs=1k count=2048

b) Make a filesystem on it. Say ext2fs for this example::

	mke2fs -vm0 /dev/ram0 2048

c) Mount it, copy the files you want to it (eg: /etc/* /dev/* ...)
   and unmount it again.

d) Compress the contents of the RAM disk. The level of compression
   will be approximately 50% of the space used by the files. Unused
   space on the RAM disk will compress to almost nothing::

	dd if=/dev/ram0 bs=1k count=2048 | gzip -v9 > /tmp/ram_image.gz

e) Put the kernel onto the floppy::

	dd if=zImage of=/dev/fd0 bs=1k

f) Put the RAM disk image onto the floppy, after the kernel. Use an offset
   that is slightly larger than the kernel, so that you can put another
   (possibly larger) kernel onto the same floppy later without overlapping
   the RAM disk image. An offset of 400 kB for kernels about 350 kB in
   size would be reasonable. Make sure offset+size of ram_image.gz is
   not larger than the total space on your floppy (usually 1440 kB)::

	dd if=/tmp/ram_image.gz of=/dev/fd0 bs=1k seek=400

g) Make sure that you have already specified the boot information in
   FDARGS and FDINITRD or that you use a bootloader to pass kernel
   command line boot options to the kernel.

That is it. You now have your boot/root compressed RAM disk floppy. Some
users may wish to combine steps (d) and (f) by using a pipe.

						Paul Gortmaker 12/95

Changelog:
----------

SEPT-2020 :

                Removed usage of "rdev"

10-22-04 :
		Updated to reflect changes in command line options, remove
		obsolete references, general cleanup.
		James Nelson (james4765@gmail.com)

12-95 :
		Original Document


code:
========================================
zram: Compressed RAM-based block devices
========================================

Introduction
============

The zram module creates RAM-based block devices named /dev/zram<id>
(<id> = 0, 1, ...). Pages written to these disks are compressed and stored
in memory itself. These disks allow very fast I/O and compression provides
good amounts of memory savings. Some of the use cases include /tmp storage,
use as swap disks, various caches under /var and maybe many more. :)

Statistics for individual zram devices are exported through sysfs nodes at
/sys/block/zram<id>/

Usage
=====

There are several ways to configure and manage zram device(-s):

a) using zram and zram_control sysfs attributes
b) using zramctl utility, provided by util-linux (util-linux@vger.kernel.org).

In this document we will describe only 'manual' zram configuration steps,
IOW, zram and zram_control sysfs attributes.

In order to get a better idea about zramctl please consult util-linux
documentation, zramctl man-page or `zramctl --help`. Please be informed
that zram maintainers do not develop/maintain util-linux or zramctl, should
you have any questions please contact util-linux@vger.kernel.org

Following shows a typical sequence of steps for using zram.

WARNING
=======

For the sake of simplicity we skip error checking parts in most of the
examples below. However, it is your sole responsibility to handle errors.

zram sysfs attributes always return negative values in case of errors.
The list of possible return codes:

========  =============================================================
-EBUSY	  an attempt to modify an attribute that cannot be changed once
	  the device has been initialised. Please reset device first.
-ENOMEM	  zram was not able to allocate enough memory to fulfil your
	  needs.
-EINVAL	  invalid input has been provided.
========  =============================================================

If you use 'echo', the returned value is set by the 'echo' utility,
and, in general case, something like::

	echo 3 > /sys/block/zram0/max_comp_streams
	if [ $? -ne 0 ]; then
		handle_error
	fi

should suffice.

1) Load Module
==============

::

	modprobe zram num_devices=4

This creates 4 devices: /dev/zram{0,1,2,3}

num_devices parameter is optional and tells zram how many devices should be
pre-created. Default: 1.

2) Set max number of compression streams
========================================

Regardless of the value passed to this attribute, ZRAM will always
allocate multiple compression streams - one per online CPU - thus
allowing several concurrent compression operations. The number of
allocated compression streams goes down when some of the CPUs
become offline. There is no single-compression-stream mode anymore,
unless you are running a UP system or have only 1 CPU online.

To find out how many streams are currently available::

	cat /sys/block/zram0/max_comp_streams

3) Select compression algorithm
===============================

Using comp_algorithm device attribute one can see available and
currently selected (shown in square brackets) compression algorithms,
or change the selected compression algorithm (once the device is initialised
there is no way to change compression algorithm).

Examples::

	#show supported compression algorithms
	cat /sys/block/zram0/comp_algorithm
	lzo [lz4]

	#select lzo compression algorithm
	echo lzo > /sys/block/zram0/comp_algorithm

For the time being, the `comp_algorithm` content does not necessarily
show every compression algorithm supported by the kernel. We keep this
list primarily to simplify device configuration and one can configure
a new device with a compression algorithm that is not listed in
`comp_algorithm`. The thing is that, internally, ZRAM uses Crypto API
and, if some of the algorithms were built as modules, it's impossible
to list all of them using, for instance, /proc/crypto or any other
method. This, however, has an advantage of permitting the usage of
custom crypto compression modules (implementing S/W or H/W compression).

4) Set Disksize
===============

Set disk size by writing the value to sysfs node 'disksize'.
The value can be either in bytes or you can use mem suffixes.
Examples::

	# Initialize /dev/zram0 with 50MB disksize
	echo $((50*1024*1024)) > /sys/block/zram0/disksize

	# Using mem suffixes
	echo 256K > /sys/block/zram0/disksize
	echo 512M > /sys/block/zram0/disksize
	echo 1G > /sys/block/zram0/disksize

Note:
There is little point creating a zram of greater than twice the size of memory
since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the
size of the disk when not in use so a huge zram is wasteful.

5) Set memory limit: Optional
=============================

Set memory limit by writing the value to sysfs node 'mem_limit'.
The value can be either in bytes or you can use mem suffixes.
In addition, you could change the value in runtime.
Examples::

	# limit /dev/zram0 with 50MB memory
	echo $((50*1024*1024)) > /sys/block/zram0/mem_limit

	# Using mem suffixes
	echo 256K > /sys/block/zram0/mem_limit
	echo 512M > /sys/block/zram0/mem_limit
	echo 1G > /sys/block/zram0/mem_limit

	# To disable memory limit
	echo 0 > /sys/block/zram0/mem_limit

6) Activate
===========

::

	mkswap /dev/zram0
	swapon /dev/zram0

	mkfs.ext4 /dev/zram1
	mount /dev/zram1 /tmp

7) Add/remove zram devices
==========================

zram provides a control interface, which enables dynamic (on-demand) device
addition and removal.

In order to add a new /dev/zramX device, perform a read operation on the hot_add
attribute. This will return either the new device's device id (meaning that you
can use /dev/zram<id>) or an error code.

Example::

	cat /sys/class/zram-control/hot_add
	1

To remove the existing /dev/zramX device (where X is a device id)
execute::

	echo X > /sys/class/zram-control/hot_remove

8) Stats
========

Per-device statistics are exported as various nodes under /sys/block/zram<id>/

A brief description of exported device attributes follows. For more details
please read Documentation/ABI/testing/sysfs-block-zram.

======================  ======  ===============================================
Name            	access            description
======================  ======  ===============================================
disksize          	RW	show and set the device's disk size
initstate         	RO	shows the initialization state of the device
reset             	WO	trigger device reset
mem_used_max      	WO	reset the `mem_used_max` counter (see later)
mem_limit         	WO	specifies the maximum amount of memory ZRAM can
				use to store the compressed data
writeback_limit   	WO	specifies the maximum amount of write IO zram
				can write out to backing device as 4KB unit
writeback_limit_enable  RW	show and set writeback_limit feature
max_comp_streams  	RW	the number of possible concurrent compress
				operations
comp_algorithm    	RW	show and change the compression algorithm
compact           	WO	trigger memory compaction
debug_stat        	RO	this file is used for zram debugging purposes
backing_dev	  	RW	set up backend storage for zram to write out
idle		  	WO	mark allocated slot as idle
======================  ======  ===============================================

User space is advised to use the following files to read the device statistics.

File /sys/block/zram<id>/stat

Represents block layer statistics. Read Documentation/block/stat.rst for
details.

File /sys/block/zram<id>/io_stat

The stat file represents device's I/O statistics not accounted by block
layer and, thus, not available in zram<id>/stat file. It consists of a
single line of text and contains the following stats separated by
whitespace:

 =============    =============================================================
 failed_reads     The number of failed reads
 failed_writes    The number of failed writes
 invalid_io       The number of non-page-size-aligned I/O requests
 notify_free      Depending on device usage scenario it may account

                  a) the number of pages freed because of swap slot free
                     notifications
                  b) the number of pages freed because of
                     REQ_OP_DISCARD requests sent by bio. The former ones are
                     sent to a swap block device when a swap slot is freed,
                     which implies that this disk is being used as a swap disk.

                  The latter ones are sent by filesystem mounted with
                  discard option, whenever some data blocks are getting
                  discarded.
 =============    =============================================================

File /sys/block/zram<id>/mm_stat

The mm_stat file represents the device's mm statistics. It consists of a single
line of text and contains the following stats separated by whitespace:

 ================ =============================================================
 orig_data_size   uncompressed size of data stored in this disk.
                  Unit: bytes
 compr_data_size  compressed size of data stored in this disk
 mem_used_total   the amount of memory allocated for this disk. This
                  includes allocator fragmentation and metadata overhead,
                  allocated for this disk. So, allocator space efficiency
                  can be calculated using compr_data_size and this statistic.
                  Unit: bytes
 mem_limit        the maximum amount of memory ZRAM can use to store
                  the compressed data
 mem_used_max     the maximum amount of memory zram has consumed to
                  store the data
 same_pages       the number of same element filled pages written to this disk.
                  No memory is allocated for such pages.
 pages_compacted  the number of pages freed during compaction
 huge_pages	  the number of incompressible pages
 huge_pages_since the number of incompressible pages since zram set up
 ================ =============================================================

File /sys/block/zram<id>/bd_stat

The bd_stat file represents a device's backing device statistics. It consists of
a single line of text and contains the following stats separated by whitespace:

 ============== =============================================================
 bd_count	size of data written in backing device.
		Unit: 4K bytes
 bd_reads	the number of reads from backing device
		Unit: 4K bytes
 bd_writes	the number of writes to backing device
		Unit: 4K bytes
 ============== =============================================================

9) Deactivate
=============

::

	swapoff /dev/zram0
	umount /dev/zram1

10) Reset
=========

	Write any positive value to 'reset' sysfs node::

		echo 1 > /sys/block/zram0/reset
		echo 1 > /sys/block/zram1/reset

	This frees all the memory allocated for the given device and
	resets the disksize to zero. You must set the disksize again
	before reusing the device.

Optional Feature
================

writeback
---------

With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page
to backing storage rather than keeping it in memory.
To use the feature, admin should set up backing device via::

	echo /dev/sda5 > /sys/block/zramX/backing_dev

before disksize setting. It supports only partitions at this moment.
If admin wants to use incompressible page writeback, they could do it via::

	echo huge > /sys/block/zramX/writeback

To use idle page writeback, first, user need to declare zram pages
as idle::

	echo all > /sys/block/zramX/idle

From now on, any pages on zram are idle pages. The idle mark
will be removed until someone requests access of the block.
IOW, unless there is access request, those pages are still idle pages.
Additionally, when CONFIG_ZRAM_MEMORY_TRACKING is enabled pages can be
marked as idle based on how long (in seconds) it's been since they were
last accessed::

        echo 86400 > /sys/block/zramX/idle

In this example all pages which haven't been accessed in more than 86400
seconds (one day) will be marked idle.

Admin can request writeback of those idle pages at right timing via::

	echo idle > /sys/block/zramX/writeback

With the command, zram will writeback idle pages from memory to the storage.

Additionally, if a user choose to writeback only huge and idle pages
this can be accomplished with::

        echo huge_idle > /sys/block/zramX/writeback

If a user chooses to writeback only incompressible pages (pages that none of
algorithms can compress) this can be accomplished with::

	echo incompressible > /sys/block/zramX/writeback

If an admin wants to write a specific page in zram device to the backing device,
they could write a page index into the interface::

	echo "page_index=1251" > /sys/block/zramX/writeback

If there are lots of write IO with flash device, potentially, it has
flash wearout problem so that admin needs to design write limitation
to guarantee storage health for entire product life.

To overcome the concern, zram supports "writeback_limit" feature.
The "writeback_limit_enable"'s default value is 0 so that it doesn't limit
any writeback. IOW, if admin wants to apply writeback budget, they should
enable writeback_limit_enable via::

	$ echo 1 > /sys/block/zramX/writeback_limit_enable

Once writeback_limit_enable is set, zram doesn't allow any writeback
until admin sets the budget via /sys/block/zramX/writeback_limit.

(If admin doesn't enable writeback_limit_enable, writeback_limit's value
assigned via /sys/block/zramX/writeback_limit is meaningless.)

If admin wants to limit writeback as per-day 400M, they could do it
like below::

	$ MB_SHIFT=20
	$ 4K_SHIFT=12
	$ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
		/sys/block/zram0/writeback_limit.
	$ echo 1 > /sys/block/zram0/writeback_limit_enable

If admins want to allow further write again once the budget is exhausted,
they could do it like below::

	$ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
		/sys/block/zram0/writeback_limit

If an admin wants to see the remaining writeback budget since last set::

	$ cat /sys/block/zramX/writeback_limit

If an admin wants to disable writeback limit, they could do::

	$ echo 0 > /sys/block/zramX/writeback_limit_enable

The writeback_limit count will reset whenever you reset zram (e.g.,
system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of
writeback happened until you reset the zram to allocate extra writeback
budget in next setting is user's job.

If admin wants to measure writeback count in a certain period, they could
know it via /sys/block/zram0/bd_stat's 3rd column.

recompression
-------------

With CONFIG_ZRAM_MULTI_COMP, zram can recompress pages using alternative
(secondary) compression algorithms. The basic idea is that alternative
compression algorithm can provide better compression ratio at a price of
(potentially) slower compression/decompression speeds. Alternative compression
algorithm can, for example, be more successful compressing huge pages (those
that default algorithm failed to compress). Another application is idle pages
recompression - pages that are cold and sit in the memory can be recompressed
using more effective algorithm and, hence, reduce zsmalloc memory usage.

With CONFIG_ZRAM_MULTI_COMP, zram supports up to 4 compression algorithms:
one primary and up to 3 secondary ones. Primary zram compressor is explained
in "3) Select compression algorithm", secondary algorithms are configured
using recomp_algorithm device attribute.

Example:::

	#show supported recompression algorithms
	cat /sys/block/zramX/recomp_algorithm
	#1: lzo lzo-rle lz4 lz4hc [zstd]
	#2: lzo lzo-rle lz4 [lz4hc] zstd

Alternative compression algorithms are sorted by priority. In the example
above, zstd is used as the first alternative algorithm, which has priority
of 1, while lz4hc is configured as a compression algorithm with priority 2.
Alternative compression algorithm's priority is provided during algorithms
configuration:::

	#select zstd recompression algorithm, priority 1
	echo "algo=zstd priority=1" > /sys/block/zramX/recomp_algorithm

	#select deflate recompression algorithm, priority 2
	echo "algo=deflate priority=2" > /sys/block/zramX/recomp_algorithm

Another device attribute that CONFIG_ZRAM_MULTI_COMP enables is recompress,
which controls recompression.

Examples:::

	#IDLE pages recompression is activated by `idle` mode
	echo "type=idle" > /sys/block/zramX/recompress

	#HUGE pages recompression is activated by `huge` mode
	echo "type=huge" > /sys/block/zram0/recompress

	#HUGE_IDLE pages recompression is activated by `huge_idle` mode
	echo "type=huge_idle" > /sys/block/zramX/recompress

The number of idle pages can be significant, so user-space can pass a size
threshold (in bytes) to the recompress knob: zram will recompress only pages
of equal or greater size:::

	#recompress all pages larger than 3000 bytes
	echo "threshold=3000" > /sys/block/zramX/recompress

	#recompress idle pages larger than 2000 bytes
	echo "type=idle threshold=2000" > /sys/block/zramX/recompress

Recompression of idle pages requires memory tracking.

During re-compression for every page, that matches re-compression criteria,
ZRAM iterates the list of registered alternative compression algorithms in
order of their priorities. ZRAM stops either when re-compression was
successful (re-compressed object is smaller in size than the original one)
and matches re-compression criteria (e.g. size threshold) or when there are
no secondary algorithms left to try. If none of the secondary algorithms can
successfully re-compressed the page such a page is marked as incompressible,
so ZRAM will not attempt to re-compress it in the future.

This re-compression behaviour, when it iterates through the list of
registered compression algorithms, increases our chances of finding the
algorithm that successfully compresses a particular page. Sometimes, however,
it is convenient (and sometimes even necessary) to limit recompression to
only one particular algorithm so that it will not try any other algorithms.
This can be achieved by providing a algo=NAME parameter:::

	#use zstd algorithm only (if registered)
	echo "type=huge algo=zstd" > /sys/block/zramX/recompress

memory tracking
===============

With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the
zram block. It could be useful to catch cold or incompressible
pages of the process with*pagemap.

If you enable the feature, you could see block state via
/sys/kernel/debug/zram/zram0/block_state". The output is as follows::

	  300    75.033841 .wh...
	  301    63.806904 s.....
	  302    63.806919 ..hi..
	  303    62.801919 ....r.
	  304   146.781902 ..hi.n

First column
	zram's block index.
Second column
	access time since the system was booted
Third column
	state of the block:

	s:
		same page
	w:
		written page to backing store
	h:
		huge page
	i:
		idle page
	r:
		recompressed page (secondary compression algorithm)
	n:
		none (including secondary) of algorithms could compress it

First line of above example says 300th block is accessed at 75.033841sec
and the block's state is huge so it is written back to the backing
storage. It's a debugging feature so anyone shouldn't rely on it to work
properly.

Nitin Gupta
ngupta@vflare.org

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

.. _bootconfig:

==================
Boot Configuration
==================

:Author: Masami Hiramatsu <mhiramat@kernel.org>

Overview
========

The boot configuration expands the current kernel command line to support
additional key-value data when booting the kernel in an efficient way.
This allows administrators to pass a structured-Key config file.

Config File Syntax
==================

The boot config syntax is a simple structured key-value. Each key consists
of dot-connected-words, and key and value are connected by ``=``. The value
has to be terminated by semi-colon (``;``) or newline (``\n``).
For array value, array entries are separated by comma (``,``). ::

  KEY[.WORD[...]] = VALUE[, VALUE2[...]][;]

Unlike the kernel command line syntax, spaces are OK around the comma and ``=``.

Each key word must contain only alphabets, numbers, dash (``-``) or underscore
(``_``). And each value only contains printable characters or spaces except
for delimiters such as semi-colon (``;``), new-line (``\n``), comma (``,``),
hash (``#``) and closing brace (``}``).

If you want to use those delimiters in a value, you can use either double-
quotes (``"VALUE"``) or single-quotes (``'VALUE'``) to quote it. Note that
you can not escape these quotes.

There can be a key which doesn't have value or has an empty value. Those keys
are used for checking if the key exists or not (like a boolean).

Key-Value Syntax
----------------

The boot config file syntax allows user to merge partially same word keys
by brace. For example::

 foo.bar.baz = value1
 foo.bar.qux.quux = value2

These can be written also in::

 foo.bar {
    baz = value1
    qux.quux = value2
 }

Or more shorter, written as following::

 foo.bar { baz = value1; qux.quux = value2 }

In both styles, same key words are automatically merged when parsing it
at boot time. So you can append similar trees or key-values.

Same-key Values
---------------

It is prohibited that two or more values or arrays share a same-key.
For example,::

 foo = bar, baz
 foo = qux  # !ERROR! we can not re-define same key

If you want to update the value, you must use the override operator
``:=`` explicitly. For example::

 foo = bar, baz
 foo := qux

then, the ``qux`` is assigned to ``foo`` key. This is useful for
overriding the default value by adding (partial) custom bootconfigs
without parsing the default bootconfig.

If you want to append the value to existing key as an array member,
you can use ``+=`` operator. For example::

 foo = bar, baz
 foo += qux

In this case, the key ``foo`` has ``bar``, ``baz`` and ``qux``.

Moreover, sub-keys and a value can coexist under a parent key.
For example, following config is allowed.::

 foo = value1
 foo.bar = value2
 foo := value3 # This will update foo's value.

Note, since there is no syntax to put a raw value directly under a
structured key, you have to define it outside of the brace. For example::

 foo {
     bar = value1
     bar {
         baz = value2
         qux = value3
     }
 }

Also, the order of the value node under a key is fixed. If there
are a value and subkeys, the value is always the first child node
of the key. Thus if user specifies subkeys first, e.g.::

 foo.bar = value1
 foo = value2

In the program (and /proc/bootconfig), it will be shown as below::

 foo = value2
 foo.bar = value1

Comments
--------

The config syntax accepts shell-script style comments. The comments starting
with hash ("#") until newline ("\n") will be ignored.

::

 # comment line
 foo = value # value is set to foo.
 bar = 1, # 1st element
       2, # 2nd element
       3  # 3rd element

This is parsed as below::

 foo = value
 bar = 1, 2, 3

Note that you can not put a comment between value and delimiter(``,`` or
``;``). This means following config has a syntax error ::

 key = 1 # comment
       ,2

/proc/bootconfig
================

/proc/bootconfig is a user-space interface of the boot config.
Unlike /proc/cmdline, this file shows the key-value style list.
Each key-value pair is shown in each line with following style::

 KEY[.WORDS...] = "[VALUE]"[,"VALUE2"...]

Boot Kernel With a Boot Config
==============================

There are two options to boot the kernel with bootconfig: attaching the
bootconfig to the initrd image or embedding it in the kernel itself.

Attaching a Boot Config to Initrd
---------------------------------

Since the boot configuration file is loaded with initrd by default,
it will be added to the end of the initrd (initramfs) image file with
padding, size, checksum and 12-byte magic word as below.

[initrd][bootconfig][padding][size(le32)][checksum(le32)][#BOOTCONFIG\n]

The size and checksum fields are unsigned 32bit little endian value.

When the boot configuration is added to the initrd image, the total
file size is aligned to 4 bytes. To fill the gap, null characters
(``\0``) will be added. Thus the ``size`` is the length of the bootconfig
file + padding bytes.

The Linux kernel decodes the last part of the initrd image in memory to
get the boot configuration data.
Because of this "piggyback" method, there is no need to change or
update the boot loader and the kernel image itself as long as the boot
loader passes the correct initrd file size. If by any chance, the boot
loader passes a longer size, the kernel fails to find the bootconfig data.

To do this operation, Linux kernel provides ``bootconfig`` command under
tools/bootconfig, which allows admin to apply or delete the config file
to/from initrd image. You can build it by the following command::

 # make -C tools/bootconfig

To add your boot config file to initrd image, run bootconfig as below
(Old data is removed automatically if exists)::

 # tools/bootconfig/bootconfig -a your-config /boot/initrd.img-X.Y.Z

To remove the config from the image, you can use -d option as below::

 # tools/bootconfig/bootconfig -d /boot/initrd.img-X.Y.Z

Then add "bootconfig" on the normal kernel command line to tell the
kernel to look for the bootconfig at the end of the initrd file.

Embedding a Boot Config into Kernel
-----------------------------------

If you can not use initrd, you can also embed the bootconfig file in the
kernel by Kconfig options. In this case, you need to recompile the kernel
with the following configs::

 CONFIG_BOOT_CONFIG_EMBED=y
 CONFIG_BOOT_CONFIG_EMBED_FILE="/PATH/TO/BOOTCONFIG/FILE"

``CONFIG_BOOT_CONFIG_EMBED_FILE`` requires an absolute path or a relative
path to the bootconfig file from source tree or object tree.
The kernel will embed it as the default bootconfig.

Just as when attaching the bootconfig to the initrd, you need ``bootconfig``
option on the kernel command line to enable the embedded bootconfig.

Note that even if you set this option, you can override the embedded
bootconfig by another bootconfig which attached to the initrd.

Kernel parameters via Boot Config
=================================

In addition to the kernel command line, the boot config can be used for
passing the kernel parameters. All the key-value pairs under ``kernel``
key will be passed to kernel cmdline directly. Moreover, the key-value
pairs under ``init`` will be passed to init process via the cmdline.
The parameters are concatenated with user-given kernel cmdline string
as the following order, so that the command line parameter can override
bootconfig parameters (this depends on how the subsystem handles parameters
but in general, earlier parameter will be overwritten by later one.)::

 [bootconfig params][cmdline params] -- [bootconfig init params][cmdline init params]

Here is an example of the bootconfig file for kernel/init parameters.::

 kernel {
   root = 01234567-89ab-cdef-0123-456789abcd
 }
 init {
  splash
 }

This will be copied into the kernel cmdline string as the following::

 root="01234567-89ab-cdef-0123-456789abcd" -- splash

If user gives some other command line like,::

 ro bootconfig -- quiet

The final kernel cmdline will be the following::

 root="01234567-89ab-cdef-0123-456789abcd" ro bootconfig -- splash quiet

Config File Limitation
======================

Currently the maximum config size size is 32KB and the total key-words (not
key-value entries) must be under 1024 nodes.
Note: this is not the number of entries but nodes, an entry must consume
more than 2 nodes (a key-word and a value). So theoretically, it will be
up to 512 key-value pairs. If keys contains 3 words in average, it can
contain 256 key-value pairs. In most cases, the number of config items
will be under 100 entries and smaller than 8KB, so it would be enough.
If the node number exceeds 1024, parser returns an error even if the file
size is smaller than 32KB. (Note that this maximum size is not including
the padding null characters.)
Anyway, since bootconfig command verifies it when appending a boot config
to initrd image, user can notice it before boot.

Bootconfig APIs
===============

User can query or loop on key-value pairs, also it is possible to find
a root (prefix) key node and find key-values under that node.

If you have a key string, you can query the value directly with the key
using xbc_find_value(). If you want to know what keys exist in the boot
config, you can use xbc_for_each_key_value() to iterate key-value pairs.
Note that you need to use xbc_array_for_each_value() for accessing
each array's value, e.g.::

 vnode = NULL;
 xbc_find_value("key.word", &vnode);
 if (vnode && xbc_node_is_array(vnode))
    xbc_array_for_each_value(vnode, value) {
      printk("%s ", value);
    }

If you want to focus on keys which have a prefix string, you can use
xbc_find_node() to find a node by the prefix string, and iterate
keys under the prefix node with xbc_node_for_each_key_value().

But the most typical usage is to get the named value under prefix
or get the named array under prefix as below::

 root = xbc_find_node("key.prefix");
 value = xbc_node_find_value(root, "option", &vnode);
 ...
 xbc_node_for_each_array_value(root, "array-option", value, anode) {
    ...
 }

This accesses a value of "key.prefix.option" and an array of
"key.prefix.array-option".

Locking is not needed, since after initialization, the config becomes
read-only. All data and keys must be copied if you need to modify it.

Functions and structures
========================

.. kernel-doc:: include/linux/bootconfig.h
.. kernel-doc:: lib/bootconfig.c


code:
Linux Braille Console
=====================

To get early boot messages on a braille device (before userspace screen
readers can start), you first need to compile the support for the usual serial
console (see :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`), and
for braille device
(in :menuselection:`Device Drivers --> Accessibility support --> Console on braille device`).

Then you need to specify a ``console=brl``, option on the kernel command line, the
format is::

	console=brl,serial_options...

where ``serial_options...`` are the same as described in
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`.

So for instance you can use ``console=brl,ttyS0`` if the braille device is connected to the first serial port, and ``console=brl,ttyS0,115200`` to
override the baud rate to 115200, etc.

By default, the braille device will just show the last kernel message (console
mode).  To review previous messages, press the Insert key to switch to the VT
review mode.  In review mode, the arrow keys permit to browse in the VT content,
:kbd:`PAGE-UP`/:kbd:`PAGE-DOWN` keys go at the top/bottom of the screen, and
the :kbd:`HOME` key goes back
to the cursor, hence providing very basic screen reviewing facility.

Sound feedback can be obtained by adding the ``braille_console.sound=1`` kernel
parameter.

For simplicity, only one braille console can be enabled, other uses of
``console=brl,...`` will be discarded.  Also note that it does not interfere with
the console selection mechanism described in
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`.

For now, only the VisioBraille device is supported.

Samuel Thibault <samuel.thibault@ens-lyon.org>


code:
=============
btmrvl driver
=============

All commands are used via debugfs interface.

Set/get driver configurations
=============================

Path:	/debug/btmrvl/config/

gpiogap=[n], hscfgcmd
	These commands are used to configure the host sleep parameters::
	bit 8:0  -- Gap
	bit 16:8 -- GPIO

	where GPIO is the pin number of GPIO used to wake up the host.
	It could be any valid GPIO pin# (e.g. 0-7) or 0xff (SDIO interface
	wakeup will be used instead).

	where Gap is the gap in milli seconds between wakeup signal and
	wakeup event, or 0xff for special host sleep setting.

	Usage::

		# Use SDIO interface to wake up the host and set GAP to 0x80:
		echo 0xff80 > /debug/btmrvl/config/gpiogap
		echo 1 > /debug/btmrvl/config/hscfgcmd

		# Use GPIO pin #3 to wake up the host and set GAP to 0xff:
		echo 0x03ff >  /debug/btmrvl/config/gpiogap
		echo 1 > /debug/btmrvl/config/hscfgcmd

psmode=[n], pscmd
	These commands are used to enable/disable auto sleep mode

	where the option is::

			1 	-- Enable auto sleep mode
			0 	-- Disable auto sleep mode

	Usage::

		# Enable auto sleep mode
		echo 1 > /debug/btmrvl/config/psmode
		echo 1 > /debug/btmrvl/config/pscmd

		# Disable auto sleep mode
		echo 0 > /debug/btmrvl/config/psmode
		echo 1 > /debug/btmrvl/config/pscmd

hsmode=[n], hscmd
	These commands are used to enable host sleep or wake up firmware

	where the option is::

			1	-- Enable host sleep
			0	-- Wake up firmware

	Usage::

		# Enable host sleep
		echo 1 > /debug/btmrvl/config/hsmode
		echo 1 > /debug/btmrvl/config/hscmd

		# Wake up firmware
		echo 0 > /debug/btmrvl/config/hsmode
		echo 1 > /debug/btmrvl/config/hscmd

Get driver status
=================

Path:	/debug/btmrvl/status/

Usage::

	cat /debug/btmrvl/status/<args>

where the args are:

curpsmode
	This command displays current auto sleep status.

psstate
	This command display the power save state.

hsstate
	This command display the host sleep state.

txdnldrdy
	This command displays the value of Tx download ready flag.

Issuing a raw hci command
=========================

Use hcitool to issue raw hci command, refer to hcitool manual

Usage::

	Hcitool cmd <ogf> <ocf> [Parameters]

Interface Control Command::

	hcitool cmd 0x3f 0x5b 0xf5 0x01 0x00    --Enable All interface
	hcitool cmd 0x3f 0x5b 0xf5 0x01 0x01    --Enable Wlan interface
	hcitool cmd 0x3f 0x5b 0xf5 0x01 0x02    --Enable BT interface
	hcitool cmd 0x3f 0x5b 0xf5 0x00 0x00    --Disable All interface
	hcitool cmd 0x3f 0x5b 0xf5 0x00 0x01    --Disable Wlan interface
	hcitool cmd 0x3f 0x5b 0xf5 0x00 0x02    --Disable BT interface

SD8688 firmware
===============

Images:

- /lib/firmware/sd8688_helper.bin
- /lib/firmware/sd8688.bin

The images can be downloaded from:

git.infradead.org/users/dwmw2/linux-firmware.git/libertas/


code:
Bisecting a bug
+++++++++++++++

Last updated: 28 October 2016

Introduction
============

Always try the latest kernel from kernel.org and build from source. If you are
not confident in doing that please report the bug to your distribution vendor
instead of to a kernel developer.

Finding bugs is not always easy. Have a go though. If you can't find it don't
give up. Report as much as you have found to the relevant maintainer. See
MAINTAINERS for who that is for the subsystem you have worked on.

Before you submit a bug report read
'Documentation/admin-guide/reporting-issues.rst'.

Devices not appearing
=====================

Often this is caused by udev/systemd. Check that first before blaming it
on the kernel.

Finding patch that caused a bug
===============================

Using the provided tools with ``git`` makes finding bugs easy provided the bug
is reproducible.

Steps to do it:

- build the Kernel from its git source
- start bisect with [#f1]_::

	$ git bisect start

- mark the broken changeset with::

	$ git bisect bad [commit]

- mark a changeset where the code is known to work with::

	$ git bisect good [commit]

- rebuild the Kernel and test
- interact with git bisect by using either::

	$ git bisect good

  or::

	$ git bisect bad

  depending if the bug happened on the changeset you're testing
- After some interactions, git bisect will give you the changeset that
  likely caused the bug.

- For example, if you know that the current version is bad, and version
  4.8 is good, you could do::

           $ git bisect start
           $ git bisect bad                 # Current version is bad
           $ git bisect good v4.8

.. [#f1] You can, optionally, provide both good and bad arguments at git
	 start with ``git bisect start [BAD] [GOOD]``

For further references, please read:

- The man page for ``git-bisect``
- `Fighting regressions with git bisect <https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html>`_
- `Fully automated bisecting with "git bisect run" <https://lwn.net/Articles/317154>`_
- `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_


code:
Bug hunting
===========

Kernel bug reports often come with a stack dump like the one below::

	------------[ cut here ]------------
	WARNING: CPU: 1 PID: 28102 at kernel/module.c:1108 module_put+0x57/0x70
	Modules linked in: dvb_usb_gp8psk(-) dvb_usb dvb_core nvidia_drm(PO) nvidia_modeset(PO) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd soundcore nvidia(PO) [last unloaded: rc_core]
	CPU: 1 PID: 28102 Comm: rmmod Tainted: P        WC O 4.8.4-build.1 #1
	Hardware name: MSI MS-7309/MS-7309, BIOS V1.12 02/23/2009
	 00000000 c12ba080 00000000 00000000 c103ed6a c1616014 00000001 00006dc6
	 c1615862 00000454 c109e8a7 c109e8a7 00000009 ffffffff 00000000 f13f6a10
	 f5f5a600 c103ee33 00000009 00000000 00000000 c109e8a7 f80ca4d0 c109f617
	Call Trace:
	 [<c12ba080>] ? dump_stack+0x44/0x64
	 [<c103ed6a>] ? __warn+0xfa/0x120
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<c103ee33>] ? warn_slowpath_null+0x23/0x30
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<f80ca4d0>] ? gp8psk_fe_set_frontend+0x460/0x460 [dvb_usb_gp8psk]
	 [<c109f617>] ? symbol_put_addr+0x27/0x50
	 [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
	 [<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0 [dvb_usb]
	 [<c13d03bc>] ? usb_disable_endpoint+0x7c/0xb0
	 [<f80bb48a>] ? dvb_usb_device_exit+0x2a/0x50 [dvb_usb]
	 [<c13d2882>] ? usb_unbind_interface+0x62/0x250
	 [<c136b514>] ? __pm_runtime_idle+0x44/0x70
	 [<c13620d8>] ? __device_release_driver+0x78/0x120
	 [<c1362907>] ? driver_detach+0x87/0x90
	 [<c1361c48>] ? bus_remove_driver+0x38/0x90
	 [<c13d1c18>] ? usb_deregister+0x58/0xb0
	 [<c109fbb0>] ? SyS_delete_module+0x130/0x1f0
	 [<c1055654>] ? task_work_run+0x64/0x80
	 [<c1000fa5>] ? exit_to_usermode_loop+0x85/0x90
	 [<c10013f0>] ? do_fast_syscall_32+0x80/0x130
	 [<c1549f43>] ? sysenter_past_esp+0x40/0x6a
	---[ end trace 6ebc60ef3981792f ]---

Such stack traces provide enough information to identify the line inside the
Kernel's source code where the bug happened. Depending on the severity of
the issue, it may also contain the word **Oops**, as on this one::

	BUG: unable to handle kernel NULL pointer dereference at   (null)
	IP: [<c06969d4>] iret_exc+0x7d0/0xa59
	*pdpt = 000000002258a001 *pde = 0000000000000000
	Oops: 0002 [#1] PREEMPT SMP
	...

Despite being an **Oops** or some other sort of stack trace, the offended
line is usually required to identify and handle the bug. Along this chapter,
we'll refer to "Oops" for all kinds of stack traces that need to be analyzed.

If the kernel is compiled with ``CONFIG_DEBUG_INFO``, you can enhance the
quality of the stack trace by using file:`scripts/decode_stacktrace.sh`.

Modules linked in
-----------------

Modules that are tainted or are being loaded or unloaded are marked with
"(...)", where the taint flags are described in
file:`Documentation/admin-guide/tainted-kernels.rst`, "being loaded" is
annotated with "+", and "being unloaded" is annotated with "-".

Where is the Oops message is located?
-------------------------------------

Normally the Oops text is read from the kernel buffers by klogd and
handed to ``syslogd`` which writes it to a syslog file, typically
``/var/log/messages`` (depends on ``/etc/syslog.conf``). On systems with
systemd, it may also be stored by the ``journald`` daemon, and accessed
by running ``journalctl`` command.

Sometimes ``klogd`` dies, in which case you can run ``dmesg > file`` to
read the data from the kernel buffers and save it.  Or you can
``cat /proc/kmsg > file``, however you have to break in to stop the transfer,
since ``kmsg`` is a "never ending file".

If the machine has crashed so badly that you cannot enter commands or
the disk is not available then you have three options:

(1) Hand copy the text from the screen and type it in after the machine
    has restarted.  Messy but it is the only option if you have not
    planned for a crash. Alternatively, you can take a picture of
    the screen with a digital camera - not nice, but better than
    nothing.  If the messages scroll off the top of the console, you
    may find that booting with a higher resolution (e.g., ``vga=791``)
    will allow you to read more of the text. (Caveat: This needs ``vesafb``,
    so won't help for 'early' oopses.)

(2) Boot with a serial console (see
    :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`),
    run a null modem to a second machine and capture the output there
    using your favourite communication program.  Minicom works well.

(3) Use Kdump (see Documentation/admin-guide/kdump/kdump.rst),
    extract the kernel ring buffer from old memory with using dmesg
    gdbmacro in Documentation/admin-guide/kdump/gdbmacros.txt.

Finding the bug's location
--------------------------

Reporting a bug works best if you point the location of the bug at the
Kernel source file. There are two methods for doing that. Usually, using
``gdb`` is easier, but the Kernel should be pre-compiled with debug info.

gdb
^^^

The GNU debugger (``gdb``) is the best way to figure out the exact file and line
number of the OOPS from the ``vmlinux`` file.

The usage of gdb works best on a kernel compiled with ``CONFIG_DEBUG_INFO``.
This can be set by running::

  $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO

On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
EIP value from the OOPS::

 EIP:    0060:[<c021e50e>]    Not tainted VLI

And use GDB to translate that to human-readable form::

  $ gdb vmlinux
  (gdb) l *0xc021e50e

If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
offset from the OOPS::

 EIP is at vt_ioctl+0xda8/0x1482

And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::

  $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
  $ make vmlinux
  $ gdb vmlinux
  (gdb) l *vt_ioctl+0xda8
  0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
  288	{
  289		struct vc_data *vc = NULL;
  290		int ret = 0;
  291
  292		console_lock();
  293		if (VT_BUSY(vc_num))
  294			ret = -EBUSY;
  295		else if (vc_num)
  296			vc = vc_deallocate(vc_num);
  297		console_unlock();

or, if you want to be more verbose::

  (gdb) p vt_ioctl
  $1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
  (gdb) l *0xae0+0xda8

You could, instead, use the object file::

  $ make drivers/tty/
  $ gdb drivers/tty/vt/vt_ioctl.o
  (gdb) l *vt_ioctl+0xda8

If you have a call trace, such as::

     Call Trace:
      [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
      [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
      [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
      ...

this shows the problem likely is in the :jbd: module. You can load that module
in gdb and list the relevant code::

  $ gdb fs/jbd/jbd.ko
  (gdb) l *log_wait_commit+0xa3

.. note::

     You can also do the same for any function call at the stack trace,
     like this one::

	 [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]

     The position where the above call happened can be seen with::

	$ gdb drivers/media/usb/dvb-usb/dvb-usb.o
	(gdb) l *dvb_usb_adapter_frontend_exit+0x3a

objdump
^^^^^^^

To debug a kernel, use objdump and look for the hex offset from the crash
output to find the valid line of code/assembler. Without debug symbols, you
will see the assembler code for the routine shown, but if your kernel has
debug symbols the C code will also be available. (Debug symbols can be enabled
in the kernel hacking menu of the menu configuration.) For example::

    $ objdump -r -S -l --disassemble net/dccp/ipv4.o

.. note::

   You need to be at the top level of the kernel tree for this to pick up
   your C files.

If you don't have access to the source code you can still debug some crash
dumps using the following method (example crash dump output as shown by
Dave Miller)::

     EIP is at 	+0x14/0x4c0
      ...
     Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
     00 00 55 57  56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
     <8b> 83 3c 01 00 00 89 44  24 14 8b 45 28 85 c0 89 44 24 18 0f 85

     Put the bytes into a "foo.s" file like this:

            .text
            .globl foo
     foo:
            .byte  .... /* bytes from Code: part of OOPS dump */

     Compile it with "gcc -c -o foo.o foo.s" then look at the output of
     "objdump --disassemble foo.o".

     Output:

     ip_queue_xmit:
         push       %ebp
         push       %edi
         push       %esi
         push       %ebx
         sub        $0xbc, %esp
         mov        0xd0(%esp), %ebp        ! %ebp = arg0 (skb)
         mov        0x8(%ebp), %ebx         ! %ebx = skb->sk
         mov        0x13c(%ebx), %eax       ! %eax = inet_sk(sk)->opt

file:`scripts/decodecode` can be used to automate most of this, depending
on what CPU architecture is being debugged.

Reporting the bug
-----------------

Once you find where the bug happened, by inspecting its location,
you could either try to fix it yourself or report it upstream.

In order to report it upstream, you should identify the mailing list
used for the development of the affected code. This can be done by using
the ``get_maintainer.pl`` script.

For example, if you find a bug at the gspca's sonixj.c file, you can get
its maintainers with::

	$ ./scripts/get_maintainer.pl -f drivers/media/usb/gspca/sonixj.c
	Hans Verkuil <hverkuil@xs4all.nl> (odd fixer:GSPCA USB WEBCAM DRIVER,commit_signer:1/1=100%)
	Mauro Carvalho Chehab <mchehab@kernel.org> (maintainer:MEDIA INPUT INFRASTRUCTURE (V4L/DVB),commit_signer:1/1=100%)
	Tejun Heo <tj@kernel.org> (commit_signer:1/1=100%)
	Bhaktipriya Shridhar <bhaktipriya96@gmail.com> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:4/4=100%,removed_lines:9/9=100%)
	linux-media@vger.kernel.org (open list:GSPCA USB WEBCAM DRIVER)
	linux-kernel@vger.kernel.org (open list)

Please notice that it will point to:

- The last developers that touched the source code (if this is done inside
  a git tree). On the above example, Tejun and Bhaktipriya (in this
  specific case, none really involved on the development of this file);
- The driver maintainer (Hans Verkuil);
- The subsystem maintainer (Mauro Carvalho Chehab);
- The driver and/or subsystem mailing list (linux-media@vger.kernel.org);
- the Linux Kernel mailing list (linux-kernel@vger.kernel.org).

Usually, the fastest way to have your bug fixed is to report it to mailing
list used for the development of the code (linux-media ML) copying the
driver maintainer (Hans).

If you are totally stumped as to whom to send the report, and
``get_maintainer.pl`` didn't provide you anything useful, send it to
linux-kernel@vger.kernel.org.

Thanks for your help in making Linux as stable as humanly possible.

Fixing the bug
--------------

If you know programming, you could help us by not only reporting the bug,
but also providing us with a solution. After all, open source is about
sharing what you do and don't you want to be recognised for your genius?

If you decide to take this way, once you have worked out a fix please submit
it upstream.

Please do read
:ref:`Documentation/process/submitting-patches.rst <submittingpatches>` though
to help your code get accepted.

---------------------------------------------------------------------------

Notes on Oops tracing with ``klogd``
------------------------------------

In order to help Linus and the other kernel developers there has been
substantial support incorporated into ``klogd`` for processing protection
faults.  In order to have full support for address resolution at least
version 1.3-pl3 of the ``sysklogd`` package should be used.

When a protection fault occurs the ``klogd`` daemon automatically
translates important addresses in the kernel log messages to their
symbolic equivalents.  This translated kernel message is then
forwarded through whatever reporting mechanism ``klogd`` is using.  The
protection fault message can be simply cut out of the message files
and forwarded to the kernel developers.

Two types of address resolution are performed by ``klogd``.  The first is
static translation and the second is dynamic translation.
Static translation uses the System.map file.
In order to do static translation the ``klogd`` daemon
must be able to find a system map file at daemon initialization time.
See the klogd man page for information on how ``klogd`` searches for map
files.

Dynamic address translation is important when kernel loadable modules
are being used.  Since memory for kernel modules is allocated from the
kernel's dynamic memory pools there are no fixed locations for either
the start of the module or for functions and symbols in the module.

The kernel supports system calls which allow a program to determine
which modules are loaded and their location in memory.  Using these
system calls the klogd daemon builds a symbol table which can be used
to debug a protection fault which occurs in a loadable kernel module.

At the very minimum klogd will provide the name of the module which
generated the protection fault.  There may be additional symbolic
information available if the developer of the loadable module chose to
export symbol information from the module.

Since the kernel module environment can be dynamic there must be a
mechanism for notifying the ``klogd`` daemon when a change in module
environment occurs.  There are command line options available which
allow klogd to signal the currently executing daemon that symbol
information should be refreshed.  See the ``klogd`` manual page for more
information.

A patch is included with the sysklogd distribution which modifies the
``modules-2.0.0`` package to automatically signal klogd whenever a module
is loaded or unloaded.  Applying this patch provides essentially
seamless support for debugging protection faults which occur with
kernel loadable modules.

The following is an example of a protection fault in a loadable module
processed by ``klogd``::

	Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
	Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
	Aug 29 09:51:01 blizard kernel: *pde = 00000000
	Aug 29 09:51:01 blizard kernel: Oops: 0002
	Aug 29 09:51:01 blizard kernel: CPU:    0
	Aug 29 09:51:01 blizard kernel: EIP:    0010:[oops:_oops+16/3868]
	Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
	Aug 29 09:51:01 blizard kernel: eax: 315e97cc   ebx: 003a6f80   ecx: 001be77b   edx: 00237c0c
	Aug 29 09:51:01 blizard kernel: esi: 00000000   edi: bffffdb3   ebp: 00589f90   esp: 00589f8c
	Aug 29 09:51:01 blizard kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018
	Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
	Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
	Aug 29 09:51:01 blizard kernel:        00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
	Aug 29 09:51:01 blizard kernel:        bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
	Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
	Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3

---------------------------------------------------------------------------

::

  Dr. G.W. Wettstein           Oncology Research Div. Computing Facility
  Roger Maris Cancer Center    INTERNET: greg@wind.rmcc.com
  820 4th St. N.
  Fargo, ND  58122
  Phone: 701-234-7556


code:
===================
Block IO Controller
===================

Overview
========
cgroup subsys "blkio" implements the block io controller. There seems to be
a need of various kinds of IO control policies (like proportional BW, max BW)
both at leaf nodes as well as at intermediate nodes in a storage hierarchy.
Plan is to use the same cgroup based management interface for blkio controller
and based on user options switch IO policies in the background.

One IO control policy is throttling policy which can be used to
specify upper IO rate limits on devices. This policy is implemented in
generic block layer and can be used on leaf nodes as well as higher
level logical devices like device mapper.

HOWTO
=====

Throttling/Upper Limit policy
-----------------------------
Enable Block IO controller::

	CONFIG_BLK_CGROUP=y

Enable throttling in block layer::

	CONFIG_BLK_DEV_THROTTLING=y

Mount blkio controller (see cgroups.txt, Why are cgroups needed?)::

        mount -t cgroup -o blkio none /sys/fs/cgroup/blkio

Specify a bandwidth rate on particular device for root group. The format
for policy is "<major>:<minor>  <bytes_per_second>"::

        echo "8:16  1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device

This will put a limit of 1MB/second on reads happening for root group
on device having major/minor number 8:16.

Run dd to read a file and see if rate is throttled to 1MB/s or not::

        # dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024
        1024+0 records in
        1024+0 records out
        4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s

Limits for writes can be put using blkio.throttle.write_bps_device file.

Hierarchical Cgroups
====================

Throttling implements hierarchy support; however,
throttling's hierarchy support is enabled iff "sane_behavior" is
enabled from cgroup side, which currently is a development option and
not publicly available.

If somebody created a hierarchy like as follows::

			root
			/  \
		     test1 test2
			|
		     test3

Throttling with "sane_behavior" will handle the
hierarchy correctly. For throttling, all limits apply
to the whole subtree while all statistics are local to the IOs
directly generated by tasks in that cgroup.

Throttling without "sane_behavior" enabled from cgroup side will
practically treat all groups at same level as if it looks like the
following::

				pivot
			     /  /   \  \
			root  test1 test2  test3

Various user visible config options
===================================

  CONFIG_BLK_CGROUP
	  Block IO controller.

  CONFIG_BFQ_CGROUP_DEBUG
	  Debug help. Right now some additional stats file show up in cgroup
	  if this option is enabled.

  CONFIG_BLK_DEV_THROTTLING
	  Enable block device throttling support in block layer.

Details of cgroup files
=======================

Proportional weight policy files
--------------------------------

  blkio.bfq.weight
	  Specifies per cgroup weight. This is default weight of the group
	  on all the devices until and unless overridden by per device rule
	  (see `blkio.bfq.weight_device` below).

	  Currently allowed range of weights is from 1 to 1000. For more details,
          see Documentation/block/bfq-iosched.rst.

  blkio.bfq.weight_device
          Specifes per cgroup per device weights, overriding the default group
          weight. For more details, see Documentation/block/bfq-iosched.rst.

	  Following is the format::

	    # echo dev_maj:dev_minor weight > blkio.bfq.weight_device

	  Configure weight=300 on /dev/sdb (8:16) in this cgroup::

	    # echo 8:16 300 > blkio.bfq.weight_device
	    # cat blkio.bfq.weight_device
	    dev     weight
	    8:16    300

	  Configure weight=500 on /dev/sda (8:0) in this cgroup::

	    # echo 8:0 500 > blkio.bfq.weight_device
	    # cat blkio.bfq.weight_device
	    dev     weight
	    8:0     500
	    8:16    300

	  Remove specific weight for /dev/sda in this cgroup::

	    # echo 8:0 0 > blkio.bfq.weight_device
	    # cat blkio.bfq.weight_device
	    dev     weight
	    8:16    300

  blkio.time
	  Disk time allocated to cgroup per device in milliseconds. First
	  two fields specify the major and minor number of the device and
	  third field specifies the disk time allocated to group in
	  milliseconds.

  blkio.sectors
	  Number of sectors transferred to/from disk by the group. First
	  two fields specify the major and minor number of the device and
	  third field specifies the number of sectors transferred by the
	  group to/from the device.

  blkio.io_service_bytes
	  Number of bytes transferred to/from the disk by the group. These
	  are further divided by the type of operation - read or write, sync
	  or async. First two fields specify the major and minor number of the
	  device, third field specifies the operation type and the fourth field
	  specifies the number of bytes.

  blkio.io_serviced
	  Number of IOs (bio) issued to the disk by the group. These
	  are further divided by the type of operation - read or write, sync
	  or async. First two fields specify the major and minor number of the
	  device, third field specifies the operation type and the fourth field
	  specifies the number of IOs.

  blkio.io_service_time
	  Total amount of time between request dispatch and request completion
	  for the IOs done by this cgroup. This is in nanoseconds to make it
	  meaningful for flash devices too. For devices with queue depth of 1,
	  this time represents the actual service time. When queue_depth > 1,
	  that is no longer true as requests may be served out of order. This
	  may cause the service time for a given IO to include the service time
	  of multiple IOs when served out of order which may result in total
	  io_service_time > actual time elapsed. This time is further divided by
	  the type of operation - read or write, sync or async. First two fields
	  specify the major and minor number of the device, third field
	  specifies the operation type and the fourth field specifies the
	  io_service_time in ns.

  blkio.io_wait_time
	  Total amount of time the IOs for this cgroup spent waiting in the
	  scheduler queues for service. This can be greater than the total time
	  elapsed since it is cumulative io_wait_time for all IOs. It is not a
	  measure of total time the cgroup spent waiting but rather a measure of
	  the wait_time for its individual IOs. For devices with queue_depth > 1
	  this metric does not include the time spent waiting for service once
	  the IO is dispatched to the device but till it actually gets serviced
	  (there might be a time lag here due to re-ordering of requests by the
	  device). This is in nanoseconds to make it meaningful for flash
	  devices too. This time is further divided by the type of operation -
	  read or write, sync or async. First two fields specify the major and
	  minor number of the device, third field specifies the operation type
	  and the fourth field specifies the io_wait_time in ns.

  blkio.io_merged
	  Total number of bios/requests merged into requests belonging to this
	  cgroup. This is further divided by the type of operation - read or
	  write, sync or async.

  blkio.io_queued
	  Total number of requests queued up at any given instant for this
	  cgroup. This is further divided by the type of operation - read or
	  write, sync or async.

  blkio.avg_queue_size
	  Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
	  The average queue size for this cgroup over the entire time of this
	  cgroup's existence. Queue size samples are taken each time one of the
	  queues of this cgroup gets a timeslice.

  blkio.group_wait_time
	  Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
	  This is the amount of time the cgroup had to wait since it became busy
	  (i.e., went from 0 to 1 request queued) to get a timeslice for one of
	  its queues. This is different from the io_wait_time which is the
	  cumulative total of the amount of time spent by each IO in that cgroup
	  waiting in the scheduler queue. This is in nanoseconds. If this is
	  read when the cgroup is in a waiting (for timeslice) state, the stat
	  will only report the group_wait_time accumulated till the last time it
	  got a timeslice and will not include the current delta.

  blkio.empty_time
	  Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
	  This is the amount of time a cgroup spends without any pending
	  requests when not being served, i.e., it does not include any time
	  spent idling for one of the queues of the cgroup. This is in
	  nanoseconds. If this is read when the cgroup is in an empty state,
	  the stat will only report the empty_time accumulated till the last
	  time it had a pending request and will not include the current delta.

  blkio.idle_time
	  Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
	  This is the amount of time spent by the IO scheduler idling for a
	  given cgroup in anticipation of a better request than the existing ones
	  from other queues/cgroups. This is in nanoseconds. If this is read
	  when the cgroup is in an idling state, the stat will only report the
	  idle_time accumulated till the last idle period and will not include
	  the current delta.

  blkio.dequeue
	  Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This
	  gives the statistics about how many a times a group was dequeued
	  from service tree of the device. First two fields specify the major
	  and minor number of the device and third field specifies the number
	  of times a group was dequeued from a particular device.

  blkio.*_recursive
	  Recursive version of various stats. These files show the
          same information as their non-recursive counterparts but
          include stats from all the descendant cgroups.

Throttling/Upper limit policy files
-----------------------------------
  blkio.throttle.read_bps_device
	  Specifies upper limit on READ rate from the device. IO rate is
	  specified in bytes per second. Rules are per device. Following is
	  the format::

	    echo "<major>:<minor>  <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device

  blkio.throttle.write_bps_device
	  Specifies upper limit on WRITE rate to the device. IO rate is
	  specified in bytes per second. Rules are per device. Following is
	  the format::

	    echo "<major>:<minor>  <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device

  blkio.throttle.read_iops_device
	  Specifies upper limit on READ rate from the device. IO rate is
	  specified in IO per second. Rules are per device. Following is
	  the format::

	   echo "<major>:<minor>  <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device

  blkio.throttle.write_iops_device
	  Specifies upper limit on WRITE rate to the device. IO rate is
	  specified in io per second. Rules are per device. Following is
	  the format::

	    echo "<major>:<minor>  <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_device

          Note: If both BW and IOPS rules are specified for a device, then IO is
          subjected to both the constraints.

  blkio.throttle.io_serviced
	  Number of IOs (bio) issued to the disk by the group. These
	  are further divided by the type of operation - read or write, sync
	  or async. First two fields specify the major and minor number of the
	  device, third field specifies the operation type and the fourth field
	  specifies the number of IOs.

  blkio.throttle.io_service_bytes
	  Number of bytes transferred to/from the disk by the group. These
	  are further divided by the type of operation - read or write, sync
	  or async. First two fields specify the major and minor number of the
	  device, third field specifies the operation type and the fourth field
	  specifies the number of bytes.

Common files among various policies
-----------------------------------
  blkio.reset_stats
	  Writing an int to this file will result in resetting all the stats
	  for that cgroup.


code:
==============
Control Groups
==============

Written by Paul Menage <menage@google.com> based on
Documentation/admin-guide/cgroup-v1/cpusets.rst

Original copyright statements from cpusets.txt:

Portions Copyright (C) 2004 BULL SA.

Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.

Modified by Paul Jackson <pj@sgi.com>

Modified by Christoph Lameter <cl@linux.com>

.. CONTENTS:

	1. Control Groups
	1.1 What are cgroups ?
	1.2 Why are cgroups needed ?
	1.3 How are cgroups implemented ?
	1.4 What does notify_on_release do ?
	1.5 What does clone_children do ?
	1.6 How do I use cgroups ?
	2. Usage Examples and Syntax
	2.1 Basic Usage
	2.2 Attaching processes
	2.3 Mounting hierarchies by name
	3. Kernel API
	3.1 Overview
	3.2 Synchronization
	3.3 Subsystem API
	4. Extended attributes usage
	5. Questions

1. Control Groups
=================

1.1 What are cgroups ?
----------------------

Control Groups provide a mechanism for aggregating/partitioning sets of
tasks, and all their future children, into hierarchical groups with
specialized behaviour.

Definitions:

A *cgroup* associates a set of tasks with a set of parameters for one
or more subsystems.

A *subsystem* is a module that makes use of the task grouping
facilities provided by cgroups to treat groups of tasks in
particular ways. A subsystem is typically a "resource controller" that
schedules a resource or applies per-cgroup limits, but it may be
anything that wants to act on a group of processes, e.g. a
virtualization subsystem.

A *hierarchy* is a set of cgroups arranged in a tree, such that
every task in the system is in exactly one of the cgroups in the
hierarchy, and a set of subsystems; each subsystem has system-specific
state attached to each cgroup in the hierarchy.  Each hierarchy has
an instance of the cgroup virtual filesystem associated with it.

At any one time there may be multiple active hierarchies of task
cgroups. Each hierarchy is a partition of all tasks in the system.

User-level code may create and destroy cgroups by name in an
instance of the cgroup virtual file system, specify and query to
which cgroup a task is assigned, and list the task PIDs assigned to
a cgroup. Those creations and assignments only affect the hierarchy
associated with that instance of the cgroup file system.

On their own, the only use for cgroups is for simple job
tracking. The intention is that other subsystems hook into the generic
cgroup support to provide new attributes for cgroups, such as
accounting/limiting the resources which processes in a cgroup can
access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow
you to associate a set of CPUs and a set of memory nodes with the
tasks in each cgroup.

1.2 Why are cgroups needed ?
----------------------------

There are multiple efforts to provide process aggregations in the
Linux kernel, mainly for resource-tracking purposes. Such efforts
include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server
namespaces. These all require the basic notion of a
grouping/partitioning of processes, with newly forked processes ending
up in the same group (cgroup) as their parent process.

The kernel cgroup patch provides the minimum essential kernel
mechanisms required to efficiently implement such groups. It has
minimal impact on the system fast paths, and provides hooks for
specific subsystems such as cpusets to provide additional behaviour as
desired.

Multiple hierarchy support is provided to allow for situations where
the division of tasks into cgroups is distinctly different for
different subsystems - having parallel hierarchies allows each
hierarchy to be a natural division of tasks, without having to handle
complex combinations of tasks that would be present if several
unrelated subsystems needed to be forced into the same tree of
cgroups.

At one extreme, each resource controller or subsystem could be in a
separate hierarchy; at the other extreme, all subsystems
would be attached to the same hierarchy.

As an example of a scenario (originally proposed by vatsa@in.ibm.com)
that can benefit from multiple hierarchies, consider a large
university server with various users - students, professors, system
tasks etc. The resource planning for this server could be along the
following lines::

       CPU :          "Top cpuset"
                       /       \
               CPUSet1         CPUSet2
                  |               |
               (Professors)    (Students)

               In addition (system tasks) are attached to topcpuset (so
               that they can run anywhere) with a limit of 20%

       Memory : Professors (50%), Students (30%), system (20%)

       Disk : Professors (50%), Students (30%), system (20%)

       Network : WWW browsing (20%), Network File System (60%), others (20%)
                               / \
               Professors (15%)  students (5%)

Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes
into the NFS network class.

At the same time Firefox/Lynx will share an appropriate CPU/Memory class
depending on who launched it (prof/student).

With the ability to classify tasks differently for different resources
(by putting those resource subsystems in different hierarchies),
the admin can easily set up a script which receives exec notifications
and depending on who is launching the browser he can::

    # echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks

With only a single hierarchy, he now would potentially have to create
a separate cgroup for every browser launched and associate it with
appropriate network and other resource class.  This may lead to
proliferation of such cgroups.

Also let's say that the administrator would like to give enhanced network
access temporarily to a student's browser (since it is night and the user
wants to do online gaming :))  OR give one of the student's simulation
apps enhanced CPU power.

With ability to write PIDs directly to resource classes, it's just a
matter of::

       # echo pid > /sys/fs/cgroup/network/<new_class>/tasks
       (after some time)
       # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks

Without this ability, the administrator would have to split the cgroup into
multiple separate ones and then associate the new cgroups with the
new resource classes.

1.3 How are cgroups implemented ?
---------------------------------

Control Groups extends the kernel as follows:

 - Each task in the system has a reference-counted pointer to a
   css_set.

 - A css_set contains a set of reference-counted pointers to
   cgroup_subsys_state objects, one for each cgroup subsystem
   registered in the system. There is no direct link from a task to
   the cgroup of which it's a member in each hierarchy, but this
   can be determined by following pointers through the
   cgroup_subsys_state objects. This is because accessing the
   subsystem state is something that's expected to happen frequently
   and in performance-critical code, whereas operations that require a
   task's actual cgroup assignments (in particular, moving between
   cgroups) are less common. A linked list runs through the cg_list
   field of each task_struct using the css_set, anchored at
   css_set->tasks.

 - A cgroup hierarchy filesystem can be mounted for browsing and
   manipulation from user space.

 - You can list all the tasks (by PID) attached to any cgroup.

The implementation of cgroups requires a few, simple hooks
into the rest of the kernel, none in performance-critical paths:

 - in init/main.c, to initialize the root cgroups and initial
   css_set at system boot.

 - in fork and exit, to attach and detach a task from its css_set.

In addition, a new file system of type "cgroup" may be mounted, to
enable browsing and modifying the cgroups presently known to the
kernel.  When mounting a cgroup hierarchy, you may specify a
comma-separated list of subsystems to mount as the filesystem mount
options.  By default, mounting the cgroup filesystem attempts to
mount a hierarchy containing all registered subsystems.

If an active hierarchy with exactly the same set of subsystems already
exists, it will be reused for the new mount. If no existing hierarchy
matches, and any of the requested subsystems are in use in an existing
hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy
is activated, associated with the requested subsystems.

It's not currently possible to bind a new subsystem to an active
cgroup hierarchy, or to unbind a subsystem from an active cgroup
hierarchy. This may be possible in future, but is fraught with nasty
error-recovery issues.

When a cgroup filesystem is unmounted, if there are any
child cgroups created below the top-level cgroup, that hierarchy
will remain active even though unmounted; if there are no
child cgroups then the hierarchy will be deactivated.

No new system calls are added for cgroups - all support for
querying and modifying cgroups is via this cgroup file system.

Each task under /proc has an added file named 'cgroup' displaying,
for each active hierarchy, the subsystem names and the cgroup name
as the path relative to the root of the cgroup file system.

Each cgroup is represented by a directory in the cgroup file system
containing the following files describing that cgroup:

 - tasks: list of tasks (by PID) attached to that cgroup.  This list
   is not guaranteed to be sorted.  Writing a thread ID into this file
   moves the thread into this cgroup.
 - cgroup.procs: list of thread group IDs in the cgroup.  This list is
   not guaranteed to be sorted or free of duplicate TGIDs, and userspace
   should sort/uniquify the list if this property is required.
   Writing a thread group ID into this file moves all threads in that
   group into this cgroup.
 - notify_on_release flag: run the release agent on exit?
 - release_agent: the path to use for release notifications (this file
   exists in the top cgroup only)

Other subsystems such as cpusets may add additional files in each
cgroup dir.

New cgroups are created using the mkdir system call or shell
command.  The properties of a cgroup, such as its flags, are
modified by writing to the appropriate file in that cgroups
directory, as listed above.

The named hierarchical structure of nested cgroups allows partitioning
a large system into nested, dynamically changeable, "soft-partitions".

The attachment of each task, automatically inherited at fork by any
children of that task, to a cgroup allows organizing the work load
on a system into related sets of tasks.  A task may be re-attached to
any other cgroup, if allowed by the permissions on the necessary
cgroup file system directories.

When a task is moved from one cgroup to another, it gets a new
css_set pointer - if there's an already existing css_set with the
desired collection of cgroups then that group is reused, otherwise a new
css_set is allocated. The appropriate existing css_set is located by
looking into a hash table.

To allow access from a cgroup to the css_sets (and hence tasks)
that comprise it, a set of cg_cgroup_link objects form a lattice;
each cg_cgroup_link is linked into a list of cg_cgroup_links for
a single cgroup on its cgrp_link_list field, and a list of
cg_cgroup_links for a single css_set on its cg_link_list.

Thus the set of tasks in a cgroup can be listed by iterating over
each css_set that references the cgroup, and sub-iterating over
each css_set's task set.

The use of a Linux virtual file system (vfs) to represent the
cgroup hierarchy provides for a familiar permission and name space
for cgroups, with a minimum of additional kernel code.

1.4 What does notify_on_release do ?
------------------------------------

If the notify_on_release flag is enabled (1) in a cgroup, then
whenever the last task in the cgroup leaves (exits or attaches to
some other cgroup) and the last child cgroup of that cgroup
is removed, then the kernel runs the command specified by the contents
of the "release_agent" file in that hierarchy's root directory,
supplying the pathname (relative to the mount point of the cgroup
file system) of the abandoned cgroup.  This enables automatic
removal of abandoned cgroups.  The default value of
notify_on_release in the root cgroup at system boot is disabled
(0).  The default value of other cgroups at creation is the current
value of their parents' notify_on_release settings. The default value of
a cgroup hierarchy's release_agent path is empty.

1.5 What does clone_children do ?
---------------------------------

This flag only affects the cpuset controller. If the clone_children
flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its
configuration from the parent during initialization.

1.6 How do I use cgroups ?
--------------------------

To start a new job that is to be contained within a cgroup, using
the "cpuset" cgroup subsystem, the steps are something like::

 1) mount -t tmpfs cgroup_root /sys/fs/cgroup
 2) mkdir /sys/fs/cgroup/cpuset
 3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
 4) Create the new cgroup by doing mkdir's and write's (or echo's) in
    the /sys/fs/cgroup/cpuset virtual file system.
 5) Start a task that will be the "founding father" of the new job.
 6) Attach that task to the new cgroup by writing its PID to the
    /sys/fs/cgroup/cpuset tasks file for that cgroup.
 7) fork, exec or clone the job tasks from this founding father task.

For example, the following sequence of commands will setup a cgroup
named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
and then start a subshell 'sh' in that cgroup::

  mount -t tmpfs cgroup_root /sys/fs/cgroup
  mkdir /sys/fs/cgroup/cpuset
  mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset
  cd /sys/fs/cgroup/cpuset
  mkdir Charlie
  cd Charlie
  /bin/echo 2-3 > cpuset.cpus
  /bin/echo 1 > cpuset.mems
  /bin/echo $$ > tasks
  sh
  # The subshell 'sh' is now running in cgroup Charlie
  # The next line should display '/Charlie'
  cat /proc/self/cgroup

2. Usage Examples and Syntax
============================

2.1 Basic Usage
---------------

Creating, modifying, using cgroups can be done through the cgroup
virtual filesystem.

To mount a cgroup hierarchy with all available subsystems, type::

  # mount -t cgroup xxx /sys/fs/cgroup

The "xxx" is not interpreted by the cgroup code, but will appear in
/proc/mounts so may be any useful identifying string that you like.

Note: Some subsystems do not work without some user input first.  For instance,
if cpusets are enabled the user will have to populate the cpus and mems files
for each new cgroup created before that group can be used.

As explained in section `1.2 Why are cgroups needed?` you should create
different hierarchies of cgroups for each single resource or group of
resources you want to control. Therefore, you should mount a tmpfs on
/sys/fs/cgroup and create directories for each cgroup resource or resource
group::

  # mount -t tmpfs cgroup_root /sys/fs/cgroup
  # mkdir /sys/fs/cgroup/rg1

To mount a cgroup hierarchy with just the cpuset and memory
subsystems, type::

  # mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1

While remounting cgroups is currently supported, it is not recommend
to use it. Remounting allows changing bound subsystems and
release_agent. Rebinding is hardly useful as it only works when the
hierarchy is empty and release_agent itself should be replaced with
conventional fsnotify. The support for remounting will be removed in
the future.

To Specify a hierarchy's release_agent::

  # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \
    xxx /sys/fs/cgroup/rg1

Note that specifying 'release_agent' more than once will return failure.

Note that changing the set of subsystems is currently only supported
when the hierarchy consists of a single (root) cgroup. Supporting
the ability to arbitrarily bind/unbind subsystems from an existing
cgroup hierarchy is intended to be implemented in the future.

Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the
tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1
is the cgroup that holds the whole system.

If you want to change the value of release_agent::

  # echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent

It can also be changed via remount.

If you want to create a new cgroup under /sys/fs/cgroup/rg1::

  # cd /sys/fs/cgroup/rg1
  # mkdir my_cgroup

Now you want to do something with this cgroup:

  # cd my_cgroup

In this directory you can find several files::

  # ls
  cgroup.procs notify_on_release tasks
  (plus whatever files added by the attached subsystems)

Now attach your shell to this cgroup::

  # /bin/echo $$ > tasks

You can also create cgroups inside your cgroup by using mkdir in this
directory::

  # mkdir my_sub_cs

To remove a cgroup, just use rmdir::

  # rmdir my_sub_cs

This will fail if the cgroup is in use (has cgroups inside, or
has processes attached, or is held alive by other subsystem-specific
reference).

2.2 Attaching processes
-----------------------

::

  # /bin/echo PID > tasks

Note that it is PID, not PIDs. You can only attach ONE task at a time.
If you have several tasks to attach, you have to do it one after another::

  # /bin/echo PID1 > tasks
  # /bin/echo PID2 > tasks
	  ...
  # /bin/echo PIDn > tasks

You can attach the current shell task by echoing 0::

  # echo 0 > tasks

You can use the cgroup.procs file instead of the tasks file to move all
threads in a threadgroup at once. Echoing the PID of any task in a
threadgroup to cgroup.procs causes all tasks in that threadgroup to be
attached to the cgroup. Writing 0 to cgroup.procs moves all tasks
in the writing task's threadgroup.

Note: Since every task is always a member of exactly one cgroup in each
mounted hierarchy, to remove a task from its current cgroup you must
move it into a new cgroup (possibly the root cgroup) by writing to the
new cgroup's tasks file.

Note: Due to some restrictions enforced by some cgroup subsystems, moving
a process to another cgroup can fail.

2.3 Mounting hierarchies by name
--------------------------------

Passing the name=<x> option when mounting a cgroups hierarchy
associates the given name with the hierarchy.  This can be used when
mounting a pre-existing hierarchy, in order to refer to it by name
rather than by its set of active subsystems.  Each hierarchy is either
nameless, or has a unique name.

The name should match [\w.-]+

When passing a name=<x> option for a new hierarchy, you need to
specify subsystems manually; the legacy behaviour of mounting all
subsystems when none are explicitly specified is not supported when
you give a subsystem a name.

The name of the subsystem appears as part of the hierarchy description
in /proc/mounts and /proc/<pid>/cgroups.

3. Kernel API
=============

3.1 Overview
------------

Each kernel subsystem that wants to hook into the generic cgroup
system needs to create a cgroup_subsys object. This contains
various methods, which are callbacks from the cgroup system, along
with a subsystem ID which will be assigned by the cgroup system.

Other fields in the cgroup_subsys object include:

- subsys_id: a unique array index for the subsystem, indicating which
  entry in cgroup->subsys[] this subsystem should be managing.

- name: should be initialized to a unique subsystem name. Should be
  no longer than MAX_CGROUP_TYPE_NAMELEN.

- early_init: indicate if the subsystem needs early initialization
  at system boot.

Each cgroup object created by the system has an array of pointers,
indexed by subsystem ID; this pointer is entirely managed by the
subsystem; the generic cgroup code will never touch this pointer.

3.2 Synchronization
-------------------

There is a global mutex, cgroup_mutex, used by the cgroup
system. This should be taken by anything that wants to modify a
cgroup. It may also be taken to prevent cgroups from being
modified, but more specific locks may be more appropriate in that
situation.

See kernel/cgroup.c for more details.

Subsystems can take/release the cgroup_mutex via the functions
cgroup_lock()/cgroup_unlock().

Accessing a task's cgroup pointer may be done in the following ways:
- while holding cgroup_mutex
- while holding the task's alloc_lock (via task_lock())
- inside an rcu_read_lock() section via rcu_dereference()

3.3 Subsystem API
-----------------

Each subsystem should:

- add an entry in linux/cgroup_subsys.h
- define a cgroup_subsys object called <name>_cgrp_subsys

Each subsystem may export the following methods. The only mandatory
methods are css_alloc/free. Any others that are null are presumed to
be successful no-ops.

``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)``
(cgroup_mutex held by caller)

Called to allocate a subsystem state object for a cgroup. The
subsystem should allocate its subsystem state object for the passed
cgroup, returning a pointer to the new object on success or a
ERR_PTR() value. On success, the subsystem pointer should point to
a structure of type cgroup_subsys_state (typically embedded in a
larger subsystem-specific object), which will be initialized by the
cgroup system. Note that this will be called at initialization to
create the root subsystem state for this subsystem; this case can be
identified by the passed cgroup object having a NULL parent (since
it's the root of the hierarchy) and may be an appropriate place for
initialization code.

``int css_online(struct cgroup *cgrp)``
(cgroup_mutex held by caller)

Called after @cgrp successfully completed all allocations and made
visible to cgroup_for_each_child/descendant_*() iterators. The
subsystem may choose to fail creation by returning -errno. This
callback can be used to implement reliable state sharing and
propagation along the hierarchy. See the comment on
cgroup_for_each_descendant_pre() for details.

``void css_offline(struct cgroup *cgrp);``
(cgroup_mutex held by caller)

This is the counterpart of css_online() and called iff css_online()
has succeeded on @cgrp. This signifies the beginning of the end of
@cgrp. @cgrp is being removed and the subsystem should start dropping
all references it's holding on @cgrp. When all references are dropped,
cgroup removal will proceed to the next step - css_free(). After this
callback, @cgrp should be considered dead to the subsystem.

``void css_free(struct cgroup *cgrp)``
(cgroup_mutex held by caller)

The cgroup system is about to free @cgrp; the subsystem should free
its subsystem state object. By the time this method is called, @cgrp
is completely unused; @cgrp->parent is still valid. (Note - can also
be called for a newly-created cgroup if an error occurs after this
subsystem's create() method has been called for the new cgroup).

``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)

Called prior to moving one or more tasks into a cgroup; if the
subsystem returns an error, this will abort the attach operation.
@tset contains the tasks to be attached and is guaranteed to have at
least one task in it.

If there are multiple tasks in the taskset, then:
  - it's guaranteed that all are from the same thread group
  - @tset contains all tasks from the thread group whether or not
    they're switching cgroups
  - the first task is the leader

Each @tset entry also contains the task's old cgroup and tasks which
aren't switching cgroup can be skipped easily using the
cgroup_taskset_for_each() iterator. Note that this isn't called on a
fork. If this method returns 0 (success) then this should remain valid
while the caller holds cgroup_mutex and it is ensured that either
attach() or cancel_attach() will be called in future.

``void css_reset(struct cgroup_subsys_state *css)``
(cgroup_mutex held by caller)

An optional operation which should restore @css's configuration to the
initial state.  This is currently only used on the unified hierarchy
when a subsystem is disabled on a cgroup through
"cgroup.subtree_control" but should remain enabled because other
subsystems depend on it.  cgroup core makes such a css invisible by
removing the associated interface files and invokes this callback so
that the hidden subsystem can return to the initial neutral state.
This prevents unexpected resource control from a hidden css and
ensures that the configuration is in the initial state when it is made
visible again later.

``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)

Called when a task attach operation has failed after can_attach() has succeeded.
A subsystem whose can_attach() has some side-effects should provide this
function, so that the subsystem can implement a rollback. If not, not necessary.
This will be called only about subsystems whose can_attach() operation have
succeeded. The parameters are identical to can_attach().

``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)

Called after the task has been attached to the cgroup, to allow any
post-attachment activity that requires memory allocations or blocking.
The parameters are identical to can_attach().

``void fork(struct task_struct *task)``

Called when a task is forked into a cgroup.

``void exit(struct task_struct *task)``

Called during task exit.

``void free(struct task_struct *task)``

Called when the task_struct is freed.

``void bind(struct cgroup *root)``
(cgroup_mutex held by caller)

Called when a cgroup subsystem is rebound to a different hierarchy
and root cgroup. Currently this will only involve movement between
the default hierarchy (which never has sub-cgroups) and a hierarchy
that is being created/destroyed (and hence has no sub-cgroups).

4. Extended attribute usage
===========================

cgroup filesystem supports certain types of extended attributes in its
directories and files.  The current supported types are:

	- Trusted (XATTR_TRUSTED)
	- Security (XATTR_SECURITY)

Both require CAP_SYS_ADMIN capability to set.

Like in tmpfs, the extended attributes in cgroup filesystem are stored
using kernel memory and it's advised to keep the usage at minimum.  This
is the reason why user defined extended attributes are not supported, since
any user can do it and there's no limit in the value size.

The current known users for this feature are SELinux to limit cgroup usage
in containers and systemd for assorted meta data like main PID in a cgroup
(systemd creates a cgroup per service).

5. Questions
============

::

  Q: what's up with this '/bin/echo' ?
  A: bash's builtin 'echo' command does not check calls to write() against
     errors. If you use it in the cgroup file system, you won't be
     able to tell whether a command succeeded or failed.

  Q: When I attach processes, only the first of the line gets really attached !
  A: We can only return one error code per call to write(). So you should also
     put only ONE PID.


code:
=========================
CPU Accounting Controller
=========================

The CPU accounting controller is used to group tasks using cgroups and
account the CPU usage of these groups of tasks.

The CPU accounting controller supports multi-hierarchy groups. An accounting
group accumulates the CPU usage of all of its child groups and the tasks
directly present in its group.

Accounting groups can be created by first mounting the cgroup filesystem::

  # mount -t cgroup -ocpuacct none /sys/fs/cgroup

With the above step, the initial or the parent accounting group becomes
visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.
/sys/fs/cgroup/cpuacct.usage gives the CPU time (in nanoseconds) obtained
by this group which is essentially the CPU time obtained by all the tasks
in the system.

New accounting groups can be created under the parent group /sys/fs/cgroup::

  # cd /sys/fs/cgroup
  # mkdir g1
  # echo $$ > g1/tasks

The above steps create a new group g1 and move the current shell
process (bash) into it. CPU time consumed by this bash and its children
can be obtained from g1/cpuacct.usage and the same is accumulated in
/sys/fs/cgroup/cpuacct.usage also.

cpuacct.stat file lists a few statistics which further divide the
CPU time obtained by the cgroup into user and system times. Currently
the following statistics are supported:

user: Time spent by tasks of the cgroup in user mode.
system: Time spent by tasks of the cgroup in kernel mode.

user and system are in USER_HZ unit.

cpuacct controller uses percpu_counter interface to collect user and
system times. This has two side effects:

- It is theoretically possible to see wrong values for user and system times.
  This is because percpu_counter_read() on 32bit systems isn't safe
  against concurrent writes.
- It is possible to see slightly outdated values for user and system times
  due to the batch processing nature of percpu_counter.