code:
========
dm-crypt
========

Device-Mapper's "crypt" target provides transparent encryption of block devices
using the kernel crypto API.

For a more detailed description of supported parameters see:
https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt

Parameters::

	      <cipher> <key> <iv_offset> <device path> \
	      <offset> [<#opt_params> <opt_params>]

<cipher>
    Encryption cipher, encryption mode and Initial Vector (IV) generator.

    The cipher specifications format is::

       cipher[:keycount]-chainmode-ivmode[:ivopts]

    Examples::

       aes-cbc-essiv:sha256
       aes-xts-plain64
       serpent-xts-plain64

    Cipher format also supports direct specification with kernel crypt API
    format (selected by capi: prefix). The IV specification is the same
    as for the first format type.
    This format is mainly used for specification of authenticated modes.

    The crypto API cipher specifications format is::

        capi:cipher_api_spec-ivmode[:ivopts]

    Examples::

        capi:cbc(aes)-essiv:sha256
        capi:xts(aes)-plain64

    Examples of authenticated modes::

        capi:gcm(aes)-random
        capi:authenc(hmac(sha256),xts(aes))-random
        capi:rfc7539(chacha20,poly1305)-random

    The /proc/crypto contains a list of currently loaded crypto modes.

<key>
    Key used for encryption. It is encoded either as a hexadecimal number
    or it can be passed as <key_string> prefixed with single colon
    character (':') for keys residing in kernel keyring service.
    You can only use key sizes that are valid for the selected cipher
    in combination with the selected iv mode.
    Note that for some iv modes the key string can contain additional
    keys (for example IV seed) so the key contains more parts concatenated
    into a single string.

<key_string>
    The kernel keyring key is identified by string in following format:
    <key_size>:<key_type>:<key_description>.

<key_size>
    The encryption key size in bytes. The kernel key payload size must match
    the value passed in <key_size>.

<key_type>
    Either 'logon', 'user', 'encrypted' or 'trusted' kernel key type.

<key_description>
    The kernel keyring key description crypt target should look for
    when loading key of <key_type>.

<keycount>
    Multi-key compatibility mode. You can define <keycount> keys and
    then sectors are encrypted according to their offsets (sector 0 uses key0;
    sector 1 uses key1 etc.).  <keycount> must be a power of two.

<iv_offset>
    The IV offset is a sector count that is added to the sector number
    before creating the IV.

<device path>
    This is the device that is going to be used as backend and contains the
    encrypted data.  You can specify it as a path like /dev/xxx or a device
    number <major>:<minor>.

<offset>
    Starting sector within the device where the encrypted data begins.

<#opt_params>
    Number of optional parameters. If there are no optional parameters,
    the optional parameters section can be skipped or #opt_params can be zero.
    Otherwise #opt_params is the number of following arguments.

    Example of optional parameters section:
        3 allow_discards same_cpu_crypt submit_from_crypt_cpus

allow_discards
    Block discard requests (a.k.a. TRIM) are passed through the crypt device.
    The default is to ignore discard requests.

    WARNING: Assess the specific security risks carefully before enabling this
    option.  For example, allowing discards on encrypted devices may lead to
    the leak of information about the ciphertext device (filesystem type,
    used space etc.) if the discarded blocks can be located easily on the
    device later.

same_cpu_crypt
    Perform encryption using the same cpu that IO was submitted on.
    The default is to use an unbound workqueue so that encryption work
    is automatically balanced between available CPUs.

submit_from_crypt_cpus
    Disable offloading writes to a separate thread after encryption.
    There are some situations where offloading write bios from the
    encryption threads to a single thread degrades performance
    significantly.  The default is to offload write bios to the same
    thread because it benefits CFQ to have writes submitted using the
    same context.

no_read_workqueue
    Bypass dm-crypt internal workqueue and process read requests synchronously.

no_write_workqueue
    Bypass dm-crypt internal workqueue and process write requests synchronously.
    This option is automatically enabled for host-managed zoned block devices
    (e.g. host-managed SMR hard-disks).

integrity:<bytes>:<type>
    The device requires additional <bytes> metadata per-sector stored
    in per-bio integrity structure. This metadata must by provided
    by underlying dm-integrity target.

    The <type> can be "none" if metadata is used only for persistent IV.

    For Authenticated Encryption with Additional Data (AEAD)
    the <type> is "aead". An AEAD mode additionally calculates and verifies
    integrity for the encrypted device. The additional space is then
    used for storing authentication tag (and persistent IV if needed).

sector_size:<bytes>
    Use <bytes> as the encryption unit instead of 512 bytes sectors.
    This option can be in range 512 - 4096 bytes and must be power of two.
    Virtual device will announce this size as a minimal IO and logical sector.

iv_large_sectors
   IV generators will use sector number counted in <sector_size> units
   instead of default 512 bytes sectors.

   For example, if <sector_size> is 4096 bytes, plain64 IV for the second
   sector will be 8 (without flag) and 1 if iv_large_sectors is present.
   The <iv_offset> must be multiple of <sector_size> (in 512 bytes units)
   if this flag is specified.

Example scripts
===============
LUKS (Linux Unified Key Setup) is now the preferred way to set up disk
encryption with dm-crypt using the 'cryptsetup' utility, see
https://gitlab.com/cryptsetup/cryptsetup

::

	#!/bin/sh
	# Create a crypt device using dmsetup
	dmsetup create crypt1 --table "0 `blockdev --getsz $1` crypt aes-cbc-essiv:sha256 babebabebabebabebabebabebabebabe 0 $1 0"

::

	#!/bin/sh
	# Create a crypt device using dmsetup when encryption key is stored in keyring service
	dmsetup create crypt2 --table "0 `blockdev --getsize $1` crypt aes-cbc-essiv:sha256 :32:logon:my_prefix:my_key 0 $1 0"

::

	#!/bin/sh
	# Create a crypt device using cryptsetup and LUKS header with default cipher
	cryptsetup luksFormat $1
	cryptsetup luksOpen $1 crypt1


code:
dm-dust
=======

This target emulates the behavior of bad sectors at arbitrary
locations, and the ability to enable the emulation of the failures
at an arbitrary time.

This target behaves similarly to a linear target.  At a given time,
the user can send a message to the target to start failing read
requests on specific blocks (to emulate the behavior of a hard disk
drive with bad sectors).

When the failure behavior is enabled (i.e.: when the output of
"dmsetup status" displays "fail_read_on_bad_block"), reads of blocks
in the "bad block list" will fail with EIO ("Input/output error").

Writes of blocks in the "bad block list will result in the following:

1. Remove the block from the "bad block list".
2. Successfully complete the write.

This emulates the "remapped sector" behavior of a drive with bad
sectors.

Normally, a drive that is encountering bad sectors will most likely
encounter more bad sectors, at an unknown time or location.
With dm-dust, the user can use the "addbadblock" and "removebadblock"
messages to add arbitrary bad blocks at new locations, and the
"enable" and "disable" messages to modulate the state of whether the
configured "bad blocks" will be treated as bad, or bypassed.
This allows the pre-writing of test data and metadata prior to
simulating a "failure" event where bad sectors start to appear.

Table parameters
----------------
<device_path> <offset> <blksz>

Mandatory parameters:
    <device_path>:
        Path to the block device.

    <offset>:
        Offset to data area from start of device_path

    <blksz>:
        Block size in bytes

	     (minimum 512, maximum 1073741824, must be a power of 2)

Usage instructions
------------------

First, find the size (in 512-byte sectors) of the device to be used::

        $ sudo blockdev --getsz /dev/vdb1
        33552384

Create the dm-dust device:
(For a device with a block size of 512 bytes)

::

        $ sudo dmsetup create dust1 --table '0 33552384 dust /dev/vdb1 0 512'

(For a device with a block size of 4096 bytes)

::

        $ sudo dmsetup create dust1 --table '0 33552384 dust /dev/vdb1 0 4096'

Check the status of the read behavior ("bypass" indicates that all I/O
will be passed through to the underlying device; "verbose" indicates that
bad block additions, removals, and remaps will be verbosely logged)::

        $ sudo dmsetup status dust1
        0 33552384 dust 252:17 bypass verbose

        $ sudo dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=128 iflag=direct
        128+0 records in
        128+0 records out

        $ sudo dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct
        128+0 records in
        128+0 records out

Adding and removing bad blocks
------------------------------

At any time (i.e.: whether the device has the "bad block" emulation
enabled or disabled), bad blocks may be added or removed from the
device via the "addbadblock" and "removebadblock" messages::

        $ sudo dmsetup message dust1 0 addbadblock 60
        kernel: device-mapper: dust: badblock added at block 60

        $ sudo dmsetup message dust1 0 addbadblock 67
        kernel: device-mapper: dust: badblock added at block 67

        $ sudo dmsetup message dust1 0 addbadblock 72
        kernel: device-mapper: dust: badblock added at block 72

These bad blocks will be stored in the "bad block list".
While the device is in "bypass" mode, reads and writes will succeed::

        $ sudo dmsetup status dust1
        0 33552384 dust 252:17 bypass

Enabling block read failures
----------------------------

To enable the "fail read on bad block" behavior, send the "enable" message::

        $ sudo dmsetup message dust1 0 enable
        kernel: device-mapper: dust: enabling read failures on bad sectors

        $ sudo dmsetup status dust1
        0 33552384 dust 252:17 fail_read_on_bad_block

With the device in "fail read on bad block" mode, attempting to read a
block will encounter an "Input/output error"::

        $ sudo dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=1 skip=67 iflag=direct
        dd: error reading '/dev/mapper/dust1': Input/output error
        0+0 records in
        0+0 records out
        0 bytes copied, 0.00040651 s, 0.0 kB/s

...and writing to the bad blocks will remove the blocks from the list,
therefore emulating the "remap" behavior of hard disk drives::

        $ sudo dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct
        128+0 records in
        128+0 records out

        kernel: device-mapper: dust: block 60 removed from badblocklist by write
        kernel: device-mapper: dust: block 67 removed from badblocklist by write
        kernel: device-mapper: dust: block 72 removed from badblocklist by write
        kernel: device-mapper: dust: block 87 removed from badblocklist by write

Bad block add/remove error handling
-----------------------------------

Attempting to add a bad block that already exists in the list will
result in an "Invalid argument" error, as well as a helpful message::

        $ sudo dmsetup message dust1 0 addbadblock 88
        device-mapper: message ioctl on dust1  failed: Invalid argument
        kernel: device-mapper: dust: block 88 already in badblocklist

Attempting to remove a bad block that doesn't exist in the list will
result in an "Invalid argument" error, as well as a helpful message::

        $ sudo dmsetup message dust1 0 removebadblock 87
        device-mapper: message ioctl on dust1  failed: Invalid argument
        kernel: device-mapper: dust: block 87 not found in badblocklist

Counting the number of bad blocks in the bad block list
-------------------------------------------------------

To count the number of bad blocks configured in the device, run the
following message command::

        $ sudo dmsetup message dust1 0 countbadblocks

A message will print with the number of bad blocks currently
configured on the device::

        countbadblocks: 895 badblock(s) found

Querying for specific bad blocks
--------------------------------

To find out if a specific block is in the bad block list, run the
following message command::

        $ sudo dmsetup message dust1 0 queryblock 72

The following message will print if the block is in the list::

        dust_query_block: block 72 found in badblocklist

The following message will print if the block is not in the list::

        dust_query_block: block 72 not found in badblocklist

The "queryblock" message command will work in both the "enabled"
and "disabled" modes, allowing the verification of whether a block
will be treated as "bad" without having to issue I/O to the device,
or having to "enable" the bad block emulation.

Clearing the bad block list
---------------------------

To clear the bad block list (without needing to individually run
a "removebadblock" message command for every block), run the
following message command::

        $ sudo dmsetup message dust1 0 clearbadblocks

After clearing the bad block list, the following message will appear::

        dust_clear_badblocks: badblocks cleared

If there were no bad blocks to clear, the following message will
appear::

        dust_clear_badblocks: no badblocks found

Listing the bad block list
--------------------------

To list all bad blocks in the bad block list (using an example device
with blocks 1 and 2 in the bad block list), run the following message
command::

        $ sudo dmsetup message dust1 0 listbadblocks
        1
        2

If there are no bad blocks in the bad block list, the command will
execute with no output::

        $ sudo dmsetup message dust1 0 listbadblocks

Message commands list
---------------------

Below is a list of the messages that can be sent to a dust device:

Operations on blocks (requires a <blknum> argument)::

        addbadblock <blknum>
        queryblock <blknum>
        removebadblock <blknum>

...where <blknum> is a block number within range of the device
(corresponding to the block size of the device.)

Single argument message commands::

        countbadblocks
        clearbadblocks
        listbadblocks
        disable
        enable
        quiet

Device removal
--------------

When finished, remove the device via the "dmsetup remove" command::

        $ sudo dmsetup remove dust1

Quiet mode
----------

On test runs with many bad blocks, it may be desirable to avoid
excessive logging (from bad blocks added, removed, or "remapped").
This can be done by enabling "quiet mode" via the following message::

        $ sudo dmsetup message dust1 0 quiet

This will suppress log messages from add / remove / removed by write
operations.  Log messages from "countbadblocks" or "queryblock"
message commands will still print in quiet mode.

The status of quiet mode can be seen by running "dmsetup status"::

        $ sudo dmsetup status dust1
        0 33552384 dust 252:17 fail_read_on_bad_block quiet

To disable quiet mode, send the "quiet" message again::

        $ sudo dmsetup message dust1 0 quiet

        $ sudo dmsetup status dust1
        0 33552384 dust 252:17 fail_read_on_bad_block verbose

(The presence of "verbose" indicates normal logging.)

"Why not...?"
-------------

scsi_debug has a "medium error" mode that can fail reads on one
specified sector (sector 0x1234, hardcoded in the source code), but
it uses RAM for the persistent storage, which drastically decreases
the potential device size.

dm-flakey fails all I/O from all block locations at a specified time
frequency, and not a given point in time.

When a bad sector occurs on a hard disk drive, reads to that sector
are failed by the device, usually resulting in an error code of EIO
("I/O error") or ENODATA ("No data available").  However, a write to
the sector may succeed, and result in the sector becoming readable
after the device controller no longer experiences errors reading the
sector (or after a reallocation of the sector).  However, there may
be bad sectors that occur on the device in the future, in a different,
unpredictable location.

This target seeks to provide a device that can exhibit the behavior
of a bad sector at a known sector location, at a known time, based
on a large storage device (at least tens of gigabytes, not occupying
system memory).

repo_name: angery/linux


code:
======
dm-ebs
======

This target is similar to the linear target except that it emulates
a smaller logical block size on a device with a larger logical block
size.  Its main purpose is to provide emulation of 512 byte sectors on
devices that do not provide this emulation (i.e. 4K native disks).

Supported emulated logical block sizes 512, 1024, 2048 and 4096.

Underlying block size can be set to > 4K to test buffering larger units.

Table parameters
----------------
  <dev path> <offset> <emulated sectors> [<underlying sectors>]

Mandatory parameters:

    <dev path>:
        Full pathname to the underlying block-device,
        or a "major:minor" device-number.
    <offset>:
        Starting sector within the device;
        has to be a multiple of <emulated sectors>.
    <emulated sectors>:
        Number of sectors defining the logical block size to be emulated;
        1, 2, 4, 8 sectors of 512 bytes supported.

Optional parameter:

    <underyling sectors>:
        Number of sectors defining the logical block size of <dev path>.
        2^N supported, e.g. 8 = emulate 8 sectors of 512 bytes = 4KiB.
        If not provided, the logical block size of <dev path> will be used.

Examples:

Emulate 1 sector = 512 bytes logical block size on /dev/sda starting at
offset 1024 sectors with underlying devices block size automatically set:

ebs /dev/sda 1024 1

Emulate 2 sector = 1KiB logical block size on /dev/sda starting at
offset 128 sectors, enforce 2KiB underlying device block size.
This presumes 2KiB logical blocksize on /dev/sda or less to work:

ebs /dev/sda 128 2 4


code:
=========
dm-flakey
=========

This target is the same as the linear target except that it exhibits
unreliable behaviour periodically.  It's been found useful in simulating
failing devices for testing purposes.

Starting from the time the table is loaded, the device is available for
<up interval> seconds, then exhibits unreliable behaviour for <down
interval> seconds, and then this cycle repeats.

Also, consider using this in combination with the dm-delay target too,
which can delay reads and writes and/or send them to different
underlying devices.

Table parameters
----------------

::

  <dev path> <offset> <up interval> <down interval> \
    [<num_features> [<feature arguments>]]

Mandatory parameters:

    <dev path>:
        Full pathname to the underlying block-device, or a
        "major:minor" device-number.
    <offset>:
        Starting sector within the device.
    <up interval>:
        Number of seconds device is available.
    <down interval>:
        Number of seconds device returns errors.

Optional feature parameters:

  If no feature parameters are present, during the periods of
  unreliability, all I/O returns errors.

  drop_writes:
	All write I/O is silently ignored.
	Read I/O is handled correctly.

  error_writes:
	All write I/O is failed with an error signalled.
	Read I/O is handled correctly.

  corrupt_bio_byte <Nth_byte> <direction> <value> <flags>:
	During <down interval>, replace <Nth_byte> of the data of
	each matching bio with <value>.

    <Nth_byte>:
	The offset of the byte to replace.
	Counting starts at 1, to replace the first byte.
    <direction>:
	Either 'r' to corrupt reads or 'w' to corrupt writes.
	'w' is incompatible with drop_writes.
    <value>:
	The value (from 0-255) to write.
    <flags>:
	Perform the replacement only if bio->bi_opf has all the
	selected flags set.

Examples:

Replaces the 32nd byte of READ bios with the value 1::

  corrupt_bio_byte 32 r 1 0

Replaces the 224th byte of REQ_META (=32) bios with the value 0::

  corrupt_bio_byte 224 w 0 32


code:
======
dm-ima
======

For a given system, various external services/infrastructure tools
(including the attestation service) interact with it - both during the
setup and during rest of the system run-time.  They share sensitive data
and/or execute critical workload on that system.  The external services
may want to verify the current run-time state of the relevant kernel
subsystems before fully trusting the system with business-critical
data/workload.

Device mapper plays a critical role on a given system by providing
various important functionalities to the block devices using various
target types like crypt, verity, integrity etc.  Each of these target
types’ functionalities can be configured with various attributes.
The attributes chosen to configure these target types can significantly
impact the security profile of the block device, and in-turn, of the
system itself.  For instance, the type of encryption algorithm and the
key size determines the strength of encryption for a given block device.

Therefore, verifying the current state of various block devices as well
as their various target attributes is crucial for external services before
fully trusting the system with business-critical data/workload.

IMA kernel subsystem provides the necessary functionality for
device mapper to measure the state and configuration of
various block devices -

- by device mapper itself, from within the kernel,
- in a tamper resistant way,
- and re-measured - triggered on state/configuration change.

Setting the IMA Policy:
=======================
For IMA to measure the data on a given system, the IMA policy on the
system needs to be updated to have following line, and the system needs
to be restarted for the measurements to take effect.

::

 /etc/ima/ima-policy
    measure func=CRITICAL_DATA label=device-mapper template=ima-buf

The measurements will be reflected in the IMA logs, which are located at:

::

 /sys/kernel/security/integrity/ima/ascii_runtime_measurements
 /sys/kernel/security/integrity/ima/binary_runtime_measurements

Then IMA ASCII measurement log has the following format:

::

 <PCR> <TEMPLATE_DATA_DIGEST> <TEMPLATE_NAME> <TEMPLATE_DATA>

 PCR := Platform Configuration Register, in which the values are registered.
       This is applicable if TPM chip is in use.

 TEMPLATE_DATA_DIGEST := Template data digest of the IMA record.
 TEMPLATE_NAME := Template name that registered the integrity value (e.g. ima-buf).

 TEMPLATE_DATA := <ALG> ":" <EVENT_DIGEST> <EVENT_NAME> <EVENT_DATA>
                  It contains data for the specific event to be measured,
                  in a given template data format.

 ALG := Algorithm to compute event digest
 EVENT_DIGEST := Digest of the event data
 EVENT_NAME := Description of the event (e.g. 'dm_table_load').
 EVENT_DATA := The event data to be measured.

|

| *NOTE #1:*
| The DM target data measured by IMA subsystem can alternatively
 be queried from userspace by setting DM_IMA_MEASUREMENT_FLAG with
 DM_TABLE_STATUS_CMD.

|

| *NOTE #2:*
| The Kernel configuration CONFIG_IMA_DISABLE_HTABLE allows measurement of duplicate records.
| To support recording duplicate IMA events in the IMA log, the Kernel needs to be configured with
 CONFIG_IMA_DISABLE_HTABLE=y.

Supported Device States:
========================
Following device state changes will trigger IMA measurements:

 1. Table load
 #. Device resume
 #. Device remove
 #. Table clear
 #. Device rename

1. Table load:
---------------
When a new table is loaded in a device's inactive table slot,
the device information and target specific details from the
targets in the table are measured.

The IMA measurement log has the following format for 'dm_table_load':

::

 EVENT_NAME := "dm_table_load"
 EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <table_load_data>

 dm_version_str := "dm_version=" <N> "." <N> "." <N>
                  Same as Device Mapper driver version.
 device_metadata := <device_name> "," <device_uuid> "," <device_major> "," <device_minor> ","
                   <minor_count> "," <num_device_targets> ";"

 device_name := "name=" <dm-device-name>
 device_uuid := "uuid=" <dm-device-uuid>
 device_major := "major=" <N>
 device_minor := "minor=" <N>
 minor_count := "minor_count=" <N>
 num_device_targets := "num_targets=" <N>
 dm-device-name := Name of the device. If it contains special characters like '\', ',', ';',
                   they are prefixed with '\'.
 dm-device-uuid := UUID of the device. If it contains special characters like '\', ',', ';',
                   they are prefixed with '\'.

 table_load_data := <target_data>
                    Represents the data (as name=value pairs) from various targets in the table,
                    which is being loaded into the DM device's inactive table slot.
 target_data := <target_data_row> | <target_data><target_data_row>

 target_data_row := <target_index> "," <target_begin> "," <target_len> "," <target_name> ","
                    <target_version> "," <target_attributes> ";"
 target_index := "target_index=" <N>
                 Represents nth target in the table (from 0 to N-1 targets specified in <num_device_targets>)
                 If all the data for N targets doesn't fit in the given buffer - then the data that fits
                 in the buffer (say from target 0 to x) is measured in a given IMA event.
                 The remaining data from targets x+1 to N-1 is measured in the subsequent IMA events,
                 with the same format as that of 'dm_table_load'
                 i.e. <dm_version_str> ";" <device_metadata> ";" <table_load_data>.

 target_begin := "target_begin=" <N>
 target_len := "target_len=" <N>
 target_name := Name of the target. 'linear', 'crypt', 'integrity' etc.
                The targets that are supported for IMA measurements are documented below in the
                'Supported targets' section.
 target_version := "target_version=" <N> "." <N> "." <N>
 target_attributes := Data containing comma separated list of name=value pairs of target specific attributes.

 For instance, if a linear device is created with the following table entries,
  # dmsetup create linear1
  0 2 linear /dev/loop0 512
  2 2 linear /dev/loop0 512
  4 2 linear /dev/loop0 512
  6 2 linear /dev/loop0 512

 Then IMA ASCII measurement log will have the following entry:
 (converted from ASCII to text for readability)

 10 a8c5ff755561c7a28146389d1514c318592af49a ima-buf sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72
 dm_table_load
 dm_version=4.45.0;
 name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4;
 target_index=0,target_begin=0,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512;
 target_index=1,target_begin=2,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512;
 target_index=2,target_begin=4,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512;
 target_index=3,target_begin=6,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512;

2. Device resume:
------------------
When a suspended device is resumed, the device information and the hash of the
data from previous load of an active table are measured.

The IMA measurement log has the following format for 'dm_device_resume':

::

 EVENT_NAME := "dm_device_resume"
 EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <active_table_hash> ";" <current_device_capacity> ";"

 dm_version_str := As described in the 'Table load' section above.
 device_metadata := As described in the 'Table load' section above.
 active_table_hash := "active_table_hash=" <table_hash_alg> ":" <table_hash>
                      Rerpresents the hash of the IMA data being measured for the
                      active table for the device.
 table_hash_alg := Algorithm used to compute the hash.
 table_hash := Hash of the (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";")
               as described in the 'dm_table_load' above.
               Note: If the table_load data spans across multiple IMA 'dm_table_load'
               events for a given device, the hash is computed combining all the event data
               i.e. (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";")
               across all those events.
 current_device_capacity := "current_device_capacity=" <N>

 For instance, if a linear device is resumed with the following command,
 #dmsetup resume linear1

 then IMA ASCII measurement log will have an entry with:
 (converted from ASCII to text for readability)

 10 56c00cc062ffc24ccd9ac2d67d194af3282b934e ima-buf sha256:e7d12c03b958b4e0e53e7363a06376be88d98a1ac191fdbd3baf5e4b77f329b6
 dm_device_resume
 dm_version=4.45.0;
 name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4;
 active_table_hash=sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72;current_device_capacity=8;

3. Device remove:
------------------
When a device is removed, the device information and a sha256 hash of the
data from an active and inactive table are measured.

The IMA measurement log has the following format for 'dm_device_remove':

::

 EVENT_NAME := "dm_device_remove"
 EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <device_inactive_metadata> ";"
               <active_table_hash> "," <inactive_table_hash> "," <remove_all> ";" <current_device_capacity> ";"

 dm_version_str := As described in the 'Table load' section above.
 device_active_metadata := Device metadata that reflects the currently loaded active table.
                           The format is same as 'device_metadata' described in the 'Table load' section above.
 device_inactive_metadata := Device metadata that reflects the inactive table.
                             The format is same as 'device_metadata' described in the 'Table load' section above.
 active_table_hash := Hash of the currently loaded active table.
                      The format is same as 'active_table_hash' described in the 'Device resume' section above.
 inactive_table_hash :=  Hash of the inactive table.
                         The format is same as 'active_table_hash' described in the 'Device resume' section above.
 remove_all := "remove_all=" <yes_no>
 yes_no := "y" | "n"
 current_device_capacity := "current_device_capacity=" <N>

 For instance, if a linear device is removed with the following command,
  #dmsetup remove l1

 then IMA ASCII measurement log will have the following entry:
 (converted from ASCII to text for readability)

 10 790e830a3a7a31590824ac0642b3b31c2d0e8b38 ima-buf sha256:ab9f3c959367a8f5d4403d6ce9c3627dadfa8f9f0e7ec7899299782388de3840
 dm_device_remove
 dm_version=4.45.0;
 device_active_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=2;
 device_inactive_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1;
 active_table_hash=sha256:4a7e62efaebfc86af755831998b7db6f59b60d23c9534fb16a4455907957953a,
 inactive_table_hash=sha256:9d79c175bc2302d55a183e8f50ad4bafd60f7692fd6249e5fd213e2464384b86,remove_all=n;
 current_device_capacity=2048;

4. Table clear:
----------------
When an inactive table is cleared from the device, the device information and a sha256 hash of the
data from an inactive table are measured.

The IMA measurement log has the following format for 'dm_table_clear':

::

 EVENT_NAME := "dm_table_clear"
 EVENT_DATA := <dm_version_str> ";" <device_inactive_metadata> ";" <inactive_table_hash> ";" <current_device_capacity> ";"

 dm_version_str := As described in the 'Table load' section above.
 device_inactive_metadata := Device metadata that was captured during the load time inactive table being cleared.
                             The format is same as 'device_metadata' described in the 'Table load' section above.
 inactive_table_hash := Hash of the inactive table being cleared from the device.
                        The format is same as 'active_table_hash' described in the 'Device resume' section above.
 current_device_capacity := "current_device_capacity=" <N>

 For instance, if a linear device's inactive table is cleared,
  #dmsetup clear l1

 then IMA ASCII measurement log will have an entry with:
 (converted from ASCII to text for readability)

 10 77d347408f557f68f0041acb0072946bb2367fe5 ima-buf sha256:42f9ca22163fdfa548e6229dece2959bc5ce295c681644240035827ada0e1db5
 dm_table_clear
 dm_version=4.45.0;
 name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1;
 inactive_table_hash=sha256:75c0dc347063bf474d28a9907037eba060bfe39d8847fc0646d75e149045d545;current_device_capacity=1024;

5. Device rename:
------------------
When an device's NAME or UUID is changed, the device information and the new NAME and UUID
are measured.

The IMA measurement log has the following format for 'dm_device_rename':

::

 EVENT_NAME := "dm_device_rename"
 EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <new_device_name> "," <new_device_uuid> ";" <current_device_capacity> ";"

 dm_version_str := As described in the 'Table load' section above.
 device_active_metadata := Device metadata that reflects the currently loaded active table.
                           The format is same as 'device_metadata' described in the 'Table load' section above.
 new_device_name := "new_name=" <dm-device-name>
 dm-device-name := Same as <dm-device-name> described in 'Table load' section above
 new_device_uuid := "new_uuid=" <dm-device-uuid>
 dm-device-uuid := Same as <dm-device-uuid> described in 'Table load' section above
 current_device_capacity := "current_device_capacity=" <N>

 E.g 1: if a linear device's name is changed with the following command,
  #dmsetup rename linear1 --setuuid 1234-5678

 then IMA ASCII measurement log will have an entry with:
 (converted from ASCII to text for readability)

 10 8b0423209b4c66ac1523f4c9848c9b51ee332f48 ima-buf sha256:6847b7258134189531db593e9230b257c84f04038b5a18fd2e1473860e0569ac
 dm_device_rename
 dm_version=4.45.0;
 name=linear1,uuid=,major=253,minor=2,minor_count=1,num_targets=1;new_name=linear1,new_uuid=1234-5678;
 current_device_capacity=1024;

 E.g 2:  if a linear device's name is changed with the following command,
  # dmsetup rename linear1 linear=2

 then IMA ASCII measurement log will have an entry with:
 (converted from ASCII to text for readability)

 10 bef70476b99c2bdf7136fae033aa8627da1bf76f ima-buf sha256:8c6f9f53b9ef9dc8f92a2f2cca8910e622543d0f0d37d484870cb16b95111402
 dm_device_rename
 dm_version=4.45.0;
 name=linear1,uuid=1234-5678,major=253,minor=2,minor_count=1,num_targets=1;
 new_name=linear\=2,new_uuid=1234-5678;
 current_device_capacity=1024;

Supported targets:
==================

Following targets are supported to measure their data using IMA:

 1. cache
 #. crypt
 #. integrity
 #. linear
 #. mirror
 #. multipath
 #. raid
 #. snapshot
 #. striped
 #. verity

1. cache
---------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'cache' target.

::

 target_attributes := <target_name> "," <target_version> "," <metadata_mode> "," <cache_metadata_device> ","
                      <cache_device> "," <cache_origin_device> "," <writethrough> "," <writeback> ","
                      <passthrough> "," <no_discard_passdown> ";"

 target_name := "target_name=cache"
 target_version := "target_version=" <N> "." <N> "." <N>
 metadata_mode := "metadata_mode=" <cache_metadata_mode>
 cache_metadata_mode := "fail" | "ro" | "rw"
 cache_device := "cache_device=" <cache_device_name_string>
 cache_origin_device := "cache_origin_device=" <cache_origin_device_string>
 writethrough := "writethrough=" <yes_no>
 writeback := "writeback=" <yes_no>
 passthrough := "passthrough=" <yes_no>
 no_discard_passdown := "no_discard_passdown=" <yes_no>
 yes_no := "y" | "n"

 E.g.
 When a 'cache' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'cache' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;name=cache1,uuid=cache_uuid,major=253,minor=2,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=28672,target_name=cache,target_version=2.2.0,metadata_mode=rw,
 cache_metadata_device=253:4,cache_device=253:3,cache_origin_device=253:5,writethrough=y,writeback=n,
 passthrough=n,metadata2=y,no_discard_passdown=n;

2. crypt
---------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'crypt' target.

::

 target_attributes := <target_name> "," <target_version> "," <allow_discards> "," <same_cpu_crypt> ","
                      <submit_from_crypt_cpus> "," <no_read_workqueue> "," <no_write_workqueue> ","
                      <iv_large_sectors> "," <iv_large_sectors> "," [<integrity_tag_size> ","] [<cipher_auth> ","]
                      [<sector_size> ","] [<cipher_string> ","] <key_size> "," <key_parts> ","
                      <key_extra_size> "," <key_mac_size> ";"

 target_name := "target_name=crypt"
 target_version := "target_version=" <N> "." <N> "." <N>
 allow_discards := "allow_discards=" <yes_no>
 same_cpu_crypt := "same_cpu_crypt=" <yes_no>
 submit_from_crypt_cpus := "submit_from_crypt_cpus=" <yes_no>
 no_read_workqueue := "no_read_workqueue=" <yes_no>
 no_write_workqueue := "no_write_workqueue=" <yes_no>
 iv_large_sectors := "iv_large_sectors=" <yes_no>
 integrity_tag_size := "integrity_tag_size=" <N>
 cipher_auth := "cipher_auth=" <string>
 sector_size := "sector_size="  <N>
 cipher_string := "cipher_string="
 key_size := "key_size="  <N>
 key_parts := "key_parts="  <N>
 key_extra_size := "key_extra_size="  <N>
 key_mac_size := "key_mac_size="  <N>
 yes_no := "y" | "n"

 E.g.
 When a 'crypt' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'crypt' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=crypt1,uuid=crypt_uuid1,major=253,minor=0,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=1953125,target_name=crypt,target_version=1.23.0,
 allow_discards=y,same_cpu=n,submit_from_crypt_cpus=n,no_read_workqueue=n,no_write_workqueue=n,
 iv_large_sectors=n,cipher_string=aes-xts-plain64,key_size=32,key_parts=1,key_extra_size=0,key_mac_size=0;

3. integrity
-------------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'integrity' target.

::

 target_attributes := <target_name> "," <target_version> "," <dev_name> "," <start>
                      <tag_size> "," <mode> "," [<meta_device> ","] [<block_size> ","] <recalculate> ","
                      <allow_discards> "," <fix_padding> "," <fix_hmac> "," <legacy_recalculate> ","
                      <journal_sectors> "," <interleave_sectors> "," <buffer_sectors> ";"

 target_name := "target_name=integrity"
 target_version := "target_version=" <N> "." <N> "." <N>
 dev_name := "dev_name=" <device_name_str>
 start := "start=" <N>
 tag_size := "tag_size=" <N>
 mode := "mode=" <integrity_mode_str>
 integrity_mode_str := "J" | "B" | "D" | "R"
 meta_device := "meta_device=" <meta_device_str>
 block_size := "block_size=" <N>
 recalculate := "recalculate=" <yes_no>
 allow_discards := "allow_discards=" <yes_no>
 fix_padding := "fix_padding=" <yes_no>
 fix_hmac := "fix_hmac=" <yes_no>
 legacy_recalculate := "legacy_recalculate=" <yes_no>
 journal_sectors := "journal_sectors=" <N>
 interleave_sectors := "interleave_sectors=" <N>
 buffer_sectors := "buffer_sectors=" <N>
 yes_no := "y" | "n"

 E.g.
 When a 'integrity' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'integrity' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=integrity1,uuid=,major=253,minor=1,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=7856,target_name=integrity,target_version=1.10.0,
 dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,fix_padding=n,
 fix_hmac=n,legacy_recalculate=n,journal_sectors=88,interleave_sectors=32768,buffer_sectors=128;

4. linear
----------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'linear' target.

::

 target_attributes := <target_name> "," <target_version> "," <device_name> <,> <start> ";"

 target_name := "target_name=linear"
 target_version := "target_version=" <N> "." <N> "." <N>
 device_name := "device_name=" <linear_device_name_str>
 start := "start=" <N>

 E.g.
 When a 'linear' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'linear' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=linear1,uuid=linear_uuid1,major=253,minor=2,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=28672,target_name=linear,target_version=1.4.0,
 device_name=253:1,start=2048;

5. mirror
----------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'mirror' target.

::

 target_attributes := <target_name> "," <target_version> "," <nr_mirrors> ","
                      <mirror_device_data> "," <handle_errors> "," <keep_log> "," <log_type_status> ";"

 target_name := "target_name=mirror"
 target_version := "target_version=" <N> "." <N> "." <N>
 nr_mirrors := "nr_mirrors=" <NR>
 mirror_device_data := <mirror_device_row> | <mirror_device_data><mirror_device_row>
                       mirror_device_row is repeated <NR> times - for <NR> described in <nr_mirrors>.
 mirror_device_row := <mirror_device_name> "," <mirror_device_status>
 mirror_device_name := "mirror_device_" <X> "=" <mirror_device_name_str>
                       where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>.
 mirror_device_status := "mirror_device_" <X> "_status=" <mirror_device_status_char>
                         where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>.
 mirror_device_status_char := "A" | "F" | "D" | "S" | "R" | "U"
 handle_errors := "handle_errors=" <yes_no>
 keep_log := "keep_log=" <yes_no>
 log_type_status := "log_type_status=" <log_type_status_str>
 yes_no := "y" | "n"

 E.g.
 When a 'mirror' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'mirror' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=mirror1,uuid=mirror_uuid1,major=253,minor=6,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=2048,target_name=mirror,target_version=1.14.0,nr_mirrors=2,
    mirror_device_0=253:4,mirror_device_0_status=A,
    mirror_device_1=253:5,mirror_device_1_status=A,
 handle_errors=y,keep_log=n,log_type_status=;

6. multipath
-------------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'multipath' target.

::

 target_attributes := <target_name> "," <target_version> "," <nr_priority_groups>
                      ["," <pg_state> "," <priority_groups> "," <priority_group_paths>] ";"

 target_name := "target_name=multipath"
 target_version := "target_version=" <N> "." <N> "." <N>
 nr_priority_groups := "nr_priority_groups=" <NPG>
 priority_groups := <priority_groups_row>|<priority_groups_row><priority_groups>
 priority_groups_row := "pg_state_" <X> "=" <pg_state_str> "," "nr_pgpaths_" <X>  "=" <NPGP> ","
                        "path_selector_name_" <X> "=" <string> "," <priority_group_paths>
                        where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>.
 pg_state_str := "E" | "A" | "D"
 <priority_group_paths> := <priority_group_paths_row> | <priority_group_paths_row><priority_group_paths>
 priority_group_paths_row := "path_name_" <X> "_" <Y> "=" <string> "," "is_active_" <X> "_" <Y> "=" <is_active_str>
                             "fail_count_" <X> "_" <Y> "=" <N> "," "path_selector_status_" <X> "_" <Y> "=" <path_selector_status_str>
                             where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>,
                             and <Y> ranges from 0 to (<NPGP> -1) - for <NPGP> described in <priority_groups_row>.
 is_active_str := "A" | "F"

 E.g.
 When a 'multipath' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'multipath' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=mp,uuid=,major=253,minor=0,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=2097152,target_name=multipath,target_version=1.14.0,nr_priority_groups=2,
    pg_state_0=E,nr_pgpaths_0=2,path_selector_name_0=queue-length,
        path_name_0_0=8:16,is_active_0_0=A,fail_count_0_0=0,path_selector_status_0_0=,
        path_name_0_1=8:32,is_active_0_1=A,fail_count_0_1=0,path_selector_status_0_1=,
    pg_state_1=E,nr_pgpaths_1=2,path_selector_name_1=queue-length,
        path_name_1_0=8:48,is_active_1_0=A,fail_count_1_0=0,path_selector_status_1_0=,
        path_name_1_1=8:64,is_active_1_1=A,fail_count_1_1=0,path_selector_status_1_1=;

7. raid
--------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'raid' target.

::

 target_attributes := <target_name> "," <target_version> "," <raid_type> "," <raid_disks> "," <raid_state>
                      <raid_device_status> ["," journal_dev_mode] ";"

 target_name := "target_name=raid"
 target_version := "target_version=" <N> "." <N> "." <N>
 raid_type := "raid_type=" <raid_type_str>
 raid_disks := "raid_disks=" <NRD>
 raid_state := "raid_state=" <raid_state_str>
 raid_state_str := "frozen" | "reshape" |"resync" | "check" | "repair" | "recover" | "idle" |"undef"
 raid_device_status := <raid_device_status_row> | <raid_device_status_row><raid_device_status>
                       <raid_device_status_row> is repeated <NRD> times - for <NRD> described in <raid_disks>.
 raid_device_status_row := "raid_device_" <X> "_status=" <raid_device_status_str>
                           where <X> ranges from 0 to (<NRD> -1) - for <NRD> described in <raid_disks>.
 raid_device_status_str := "A" | "D" | "a" | "-"
 journal_dev_mode := "journal_dev_mode=" <journal_dev_mode_str>
 journal_dev_mode_str := "writethrough" | "writeback" | "invalid"

 E.g.
 When a 'raid' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'raid' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=raid_LV1,uuid=uuid_raid_LV1,major=253,minor=12,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=2048,target_name=raid,target_version=1.15.1,
 raid_type=raid10,raid_disks=4,raid_state=idle,
    raid_device_0_status=A,
    raid_device_1_status=A,
    raid_device_2_status=A,
    raid_device_3_status=A;

8. snapshot
------------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'snapshot' target.

::

 target_attributes := <target_name> "," <target_version> "," <snap_origin_name> ","
                      <snap_cow_name> "," <snap_valid> "," <snap_merge_failed> "," <snapshot_overflowed> ";"

 target_name := "target_name=snapshot"
 target_version := "target_version=" <N> "." <N> "." <N>
 snap_origin_name := "snap_origin_name=" <string>
 snap_cow_name := "snap_cow_name=" <string>
 snap_valid := "snap_valid=" <yes_no>
 snap_merge_failed := "snap_merge_failed=" <yes_no>
 snapshot_overflowed := "snapshot_overflowed=" <yes_no>
 yes_no := "y" | "n"

 E.g.
 When a 'snapshot' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'snapshot' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=snap1,uuid=snap_uuid1,major=253,minor=13,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=4096,target_name=snapshot,target_version=1.16.0,
 snap_origin_name=253:11,snap_cow_name=253:12,snap_valid=y,snap_merge_failed=n,snapshot_overflowed=n;

9. striped
-----------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'striped' target.

::

 target_attributes := <target_name> "," <target_version> "," <stripes> "," <chunk_size> ","
                      <stripe_data> ";"

 target_name := "target_name=striped"
 target_version := "target_version=" <N> "." <N> "." <N>
 stripes := "stripes=" <NS>
 chunk_size := "chunk_size=" <N>
 stripe_data := <stripe_data_row>|<stripe_data><stripe_data_row>
 stripe_data_row := <stripe_device_name> "," <stripe_physical_start> "," <stripe_status>
 stripe_device_name := "stripe_" <X> "_device_name=" <stripe_device_name_str>
                       where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>.
 stripe_physical_start := "stripe_" <X> "_physical_start=" <N>
                           where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>.
 stripe_status := "stripe_" <X> "_status=" <stripe_status_str>
                  where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>.
 stripe_status_str := "D" | "A"

 E.g.
 When a 'striped' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'striped' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=striped1,uuid=striped_uuid1,major=253,minor=5,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=640,target_name=striped,target_version=1.6.0,stripes=2,chunk_size=64,
    stripe_0_device_name=253:0,stripe_0_physical_start=2048,stripe_0_status=A,
    stripe_1_device_name=253:3,stripe_1_physical_start=2048,stripe_1_status=A;

10. verity
----------
The 'target_attributes' (described as part of EVENT_DATA in 'Table load'
section above) has the following data format for 'verity' target.

::

 target_attributes := <target_name> "," <target_version> "," <hash_failed> "," <verity_version> ","
                      <data_device_name> "," <hash_device_name> "," <verity_algorithm> "," <root_digest> ","
                      <salt> "," <ignore_zero_blocks> "," <check_at_most_once> ["," <root_hash_sig_key_desc>]
                      ["," <verity_mode>] ";"

 target_name := "target_name=verity"
 target_version := "target_version=" <N> "." <N> "." <N>
 hash_failed := "hash_failed=" <hash_failed_str>
 hash_failed_str := "C" | "V"
 verity_version := "verity_version=" <verity_version_str>
 data_device_name := "data_device_name=" <data_device_name_str>
 hash_device_name := "hash_device_name=" <hash_device_name_str>
 verity_algorithm := "verity_algorithm=" <verity_algorithm_str>
 root_digest := "root_digest=" <root_digest_str>
 salt := "salt=" <salt_str>
 salt_str := "-" <verity_salt_str>
 ignore_zero_blocks := "ignore_zero_blocks=" <yes_no>
 check_at_most_once := "check_at_most_once=" <yes_no>
 root_hash_sig_key_desc := "root_hash_sig_key_desc="
 verity_mode := "verity_mode=" <verity_mode_str>
 verity_mode_str := "ignore_corruption" | "restart_on_corruption" | "panic_on_corruption" | "invalid"
 yes_no := "y" | "n"

 E.g.
 When a 'verity' target is loaded, then IMA ASCII measurement log will have an entry
 similar to the following, depicting what 'verity' attributes are measured in EVENT_DATA
 for 'dm_table_load' event.
 (converted from ASCII to text for readability)

 dm_version=4.45.0;
 name=test-verity,uuid=,major=253,minor=2,minor_count=1,num_targets=1;
 target_index=0,target_begin=0,target_len=1953120,target_name=verity,target_version=1.8.0,hash_failed=V,
 verity_version=1,data_device_name=253:1,hash_device_name=253:0,verity_algorithm=sha256,
 root_digest=29cb87e60ce7b12b443ba6008266f3e41e93e403d7f298f8e3f316b29ff89c5e,
 salt=e48da609055204e89ae53b655ca2216dd983cf3cb829f34f63a297d106d53e2d,
 ignore_zero_blocks=n,check_at_most_once=n;


code:
================================
Early creation of mapped devices
================================

It is possible to configure a device-mapper device to act as the root device for
your system in two ways.

The first is to build an initial ramdisk which boots to a minimal userspace
which configures the device, then pivot_root(8) in to it.

The second is to create one or more device-mappers using the module parameter
"dm-mod.create=" through the kernel boot command line argument.

The format is specified as a string of data separated by commas and optionally
semi-colons, where:

 - a comma is used to separate fields like name, uuid, flags and table
   (specifies one device)
 - a semi-colon is used to separate devices.

So the format will look like this::

 dm-mod.create=<name>,<uuid>,<minor>,<flags>,<table>[,<table>+][;<name>,<uuid>,<minor>,<flags>,<table>[,<table>+]+]

Where::

	<name>		::= The device name.
	<uuid>		::= xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | ""
	<minor>		::= The device minor number | ""
	<flags>		::= "ro" | "rw"
	<table>		::= <start_sector> <num_sectors> <target_type> <target_args>
	<target_type>	::= "verity" | "linear" | ... (see list below)

The dm line should be equivalent to the one used by the dmsetup tool with the
`--concise` argument.

Target types
============

Not all target types are available as there are serious risks in allowing
activation of certain DM targets without first using userspace tools to check
the validity of associated metadata.

======================= =======================================================
`cache`			constrained, userspace should verify cache device
`crypt`			allowed
`delay`			allowed
`era`			constrained, userspace should verify metadata device
`flakey`		constrained, meant for test
`linear`		allowed
`log-writes`		constrained, userspace should verify metadata device
`mirror`		constrained, userspace should verify main/mirror device
`raid`			constrained, userspace should verify metadata device
`snapshot`		constrained, userspace should verify src/dst device
`snapshot-origin`	allowed
`snapshot-merge`	constrained, userspace should verify src/dst device
`striped`		allowed
`switch`		constrained, userspace should verify dev path
`thin`			constrained, requires dm target message from userspace
`thin-pool`		constrained, requires dm target message from userspace
`verity`		allowed
`writecache`		constrained, userspace should verify cache device
`zero`			constrained, not meant for rootfs
======================= =======================================================

If the target is not listed above, it is constrained by default (not tested).

Examples
========
An example of booting to a linear array made up of user-mode linux block
devices::

  dm-mod.create="lroot,,,rw, 0 4096 linear 98:16 0, 4096 4096 linear 98:32 0" root=/dev/dm-0

This will boot to a rw dm-linear target of 8192 sectors split across two block
devices identified by their major:minor numbers.  After boot, udev will rename
this target to /dev/mapper/lroot (depending on the rules). No uuid was assigned.

An example of multiple device-mappers, with the dm-mod.create="..." contents
is shown here split on multiple lines for readability::

  dm-linear,,1,rw,
    0 32768 linear 8:1 0,
    32768 1024000 linear 8:2 0;
  dm-verity,,3,ro,
    0 1638400 verity 1 /dev/sdc1 /dev/sdc2 4096 4096 204800 1 sha256
    ac87db56303c9c1da433d7209b5a6ef3e4779df141200cbd7c157dcb8dd89c42
    5ebfe87f7df3235b80a117ebc4078e44f55045487ad4a96581d1adb564615b51

Other examples (per target):

"crypt"::

  dm-crypt,,8,ro,
    0 1048576 crypt aes-xts-plain64
    babebabebabebabebabebabebabebabebabebabebabebabebabebabebabebabe 0
    /dev/sda 0 1 allow_discards

"delay"::

  dm-delay,,4,ro,0 409600 delay /dev/sda1 0 500

"linear"::

  dm-linear,,,rw,
    0 32768 linear /dev/sda1 0,
    32768 1024000 linear /dev/sda2 0,
    1056768 204800 linear /dev/sda3 0,
    1261568 512000 linear /dev/sda4 0

"snapshot-origin"::

  dm-snap-orig,,4,ro,0 409600 snapshot-origin 8:2

"striped"::

  dm-striped,,4,ro,0 1638400 striped 4 4096
  /dev/sda1 0 /dev/sda2 0 /dev/sda3 0 /dev/sda4 0

"verity"::

  dm-verity,,4,ro,
    0 1638400 verity 1 8:1 8:2 4096 4096 204800 1 sha256
    fb1a5a0f00deb908d8b53cb270858975e76cf64105d412ce764225d53b8f3cfd
    51934789604d1b92399c52e7cb149d1b3a1b74bbbcb103b2a0aaacbed5c08584

For setups using device-mapper on top of asynchronously probed block
devices (MMC, USB, ..), it may be necessary to tell dm-init to
explicitly wait for them to become available before setting up the
device-mapper tables. This can be done with the "dm-mod.waitfor="
module parameter, which takes a list of devices to wait for::

  dm-mod.waitfor=<device1>[,..,<deviceN>]


code:
============
dm-integrity
============

The dm-integrity target emulates a block device that has additional
per-sector tags that can be used for storing integrity information.

A general problem with storing integrity tags with every sector is that
writing the sector and the integrity tag must be atomic - i.e. in case of
crash, either both sector and integrity tag or none of them is written.

To guarantee write atomicity, the dm-integrity target uses journal, it
writes sector data and integrity tags into a journal, commits the journal
and then copies the data and integrity tags to their respective location.

The dm-integrity target can be used with the dm-crypt target - in this
situation the dm-crypt target creates the integrity data and passes them
to the dm-integrity target via bio_integrity_payload attached to the bio.
In this mode, the dm-crypt and dm-integrity targets provide authenticated
disk encryption - if the attacker modifies the encrypted device, an I/O
error is returned instead of random data.

The dm-integrity target can also be used as a standalone target, in this
mode it calculates and verifies the integrity tag internally. In this
mode, the dm-integrity target can be used to detect silent data
corruption on the disk or in the I/O path.

There's an alternate mode of operation where dm-integrity uses bitmap
instead of a journal. If a bit in the bitmap is 1, the corresponding
region's data and integrity tags are not synchronized - if the machine
crashes, the unsynchronized regions will be recalculated. The bitmap mode
is faster than the journal mode, because we don't have to write the data
twice, but it is also less reliable, because if data corruption happens
when the machine crashes, it may not be detected.

When loading the target for the first time, the kernel driver will format
the device. But it will only format the device if the superblock contains
zeroes. If the superblock is neither valid nor zeroed, the dm-integrity
target can't be loaded.

To use the target for the first time:

1. overwrite the superblock with zeroes
2. load the dm-integrity target with one-sector size, the kernel driver
   will format the device
3. unload the dm-integrity target
4. read the "provided_data_sectors" value from the superblock
5. load the dm-integrity target with the target size
   "provided_data_sectors"
6. if you want to use dm-integrity with dm-crypt, load the dm-crypt target
   with the size "provided_data_sectors"

Target arguments:

1. the underlying block device

2. the number of reserved sector at the beginning of the device - the
   dm-integrity won't read of write these sectors

3. the size of the integrity tag (if "-" is used, the size is taken from
   the internal-hash algorithm)

4. mode:

	D - direct writes (without journal)
		in this mode, journaling is
		not used and data sectors and integrity tags are written
		separately. In case of crash, it is possible that the data
		and integrity tag doesn't match.
	J - journaled writes
		data and integrity tags are written to the
		journal and atomicity is guaranteed. In case of crash,
		either both data and tag or none of them are written. The
		journaled mode degrades write throughput twice because the
		data have to be written twice.
	B - bitmap mode - data and metadata are written without any
		synchronization, the driver maintains a bitmap of dirty
		regions where data and metadata don't match. This mode can
		only be used with internal hash.
	R - recovery mode - in this mode, journal is not replayed,
		checksums are not checked and writes to the device are not
		allowed. This mode is useful for data recovery if the
		device cannot be activated in any of the other standard
		modes.

5. the number of additional arguments

Additional arguments:

journal_sectors:number
	The size of journal, this argument is used only if formatting the
	device. If the device is already formatted, the value from the
	superblock is used.

interleave_sectors:number
	The number of interleaved sectors. This values is rounded down to
	a power of two. If the device is already formatted, the value from
	the superblock is used.

meta_device:device
	Don't interleave the data and metadata on the device. Use a
	separate device for metadata.

buffer_sectors:number
	The number of sectors in one buffer. The value is rounded down to
	a power of two.

	The tag area is accessed using buffers, the buffer size is
	configurable. The large buffer size means that the I/O size will
	be larger, but there could be less I/Os issued.

journal_watermark:number
	The journal watermark in percents. When the size of the journal
	exceeds this watermark, the thread that flushes the journal will
	be started.

commit_time:number
	Commit time in milliseconds. When this time passes, the journal is
	written. The journal is also written immediately if the FLUSH
	request is received.

internal_hash:algorithm(:key)	(the key is optional)
	Use internal hash or crc.
	When this argument is used, the dm-integrity target won't accept
	integrity tags from the upper target, but it will automatically
	generate and verify the integrity tags.

	You can use a crc algorithm (such as crc32), then integrity target
	will protect the data against accidental corruption.
	You can also use a hmac algorithm (for example
	"hmac(sha256):0123456789abcdef"), in this mode it will provide
	cryptographic authentication of the data without encryption.

	When this argument is not used, the integrity tags are accepted
	from an upper layer target, such as dm-crypt. The upper layer
	target should check the validity of the integrity tags.

recalculate
	Recalculate the integrity tags automatically. It is only valid
	when using internal hash.

journal_crypt:algorithm(:key)	(the key is optional)
	Encrypt the journal using given algorithm to make sure that the
	attacker can't read the journal. You can use a block cipher here
	(such as "cbc(aes)") or a stream cipher (for example "chacha20"
	or "ctr(aes)").

	The journal contains history of last writes to the block device,
	an attacker reading the journal could see the last sector numbers
	that were written. From the sector numbers, the attacker can infer
	the size of files that were written. To protect against this
	situation, you can encrypt the journal.

journal_mac:algorithm(:key)	(the key is optional)
	Protect sector numbers in the journal from accidental or malicious
	modification. To protect against accidental modification, use a
	crc algorithm, to protect against malicious modification, use a
	hmac algorithm with a key.

	This option is not needed when using internal-hash because in this
	mode, the integrity of journal entries is checked when replaying
	the journal. Thus, modified sector number would be detected at
	this stage.

block_size:number
	The size of a data block in bytes.  The larger the block size the
	less overhead there is for per-block integrity metadata.
	Supported values are 512, 1024, 2048 and 4096 bytes.  If not
	specified the default block size is 512 bytes.

sectors_per_bit:number
	In the bitmap mode, this parameter specifies the number of
	512-byte sectors that corresponds to one bitmap bit.

bitmap_flush_interval:number
	The bitmap flush interval in milliseconds. The metadata buffers
	are synchronized when this interval expires.

allow_discards
	Allow block discard requests (a.k.a. TRIM) for the integrity device.
	Discards are only allowed to devices using internal hash.

fix_padding
	Use a smaller padding of the tag area that is more
	space-efficient. If this option is not present, large padding is
	used - that is for compatibility with older kernels.

fix_hmac
	Improve security of internal_hash and journal_mac:

	- the section number is mixed to the mac, so that an attacker can't
	  copy sectors from one journal section to another journal section
	- the superblock is protected by journal_mac
	- a 16-byte salt stored in the superblock is mixed to the mac, so
	  that the attacker can't detect that two disks have the same hmac
	  key and also to disallow the attacker to move sectors from one
	  disk to another

legacy_recalculate
	Allow recalculating of volumes with HMAC keys. This is disabled by
	default for security reasons - an attacker could modify the volume,
	set recalc_sector to zero, and the kernel would not detect the
	modification.

The journal mode (D/J), buffer_sectors, journal_watermark, commit_time and
allow_discards can be changed when reloading the target (load an inactive
table and swap the tables with suspend and resume). The other arguments
should not be changed when reloading the target because the layout of disk
data depend on them and the reloaded target would be non-functional.

Status line:

1. the number of integrity mismatches
2. provided data sectors - that is the number of sectors that the user
   could use
3. the current recalculating position (or '-' if we didn't recalculate)

The layout of the formatted block device:

* reserved sectors
    (they are not used by this target, they can be used for
    storing LUKS metadata or for other purpose), the size of the reserved
    area is specified in the target arguments

* superblock (4kiB)
	* magic string - identifies that the device was formatted
	* version
	* log2(interleave sectors)
	* integrity tag size
	* the number of journal sections
	* provided data sectors - the number of sectors that this target
	  provides (i.e. the size of the device minus the size of all
	  metadata and padding). The user of this target should not send
	  bios that access data beyond the "provided data sectors" limit.
	* flags
	    SB_FLAG_HAVE_JOURNAL_MAC
		- a flag is set if journal_mac is used
	    SB_FLAG_RECALCULATING
		- recalculating is in progress
	    SB_FLAG_DIRTY_BITMAP
		- journal area contains the bitmap of dirty
		  blocks
	* log2(sectors per block)
	* a position where recalculating finished
* journal
	The journal is divided into sections, each section contains:

	* metadata area (4kiB), it contains journal entries

	  - every journal entry contains:

		* logical sector (specifies where the data and tag should
		  be written)
		* last 8 bytes of data
		* integrity tag (the size is specified in the superblock)

	  - every metadata sector ends with

		* mac (8-bytes), all the macs in 8 metadata sectors form a
		  64-byte value. It is used to store hmac of sector
		  numbers in the journal section, to protect against a
		  possibility that the attacker tampers with sector
		  numbers in the journal.
		* commit id

	* data area (the size is variable; it depends on how many journal
	  entries fit into the metadata area)

	    - every sector in the data area contains:

		* data (504 bytes of data, the last 8 bytes are stored in
		  the journal entry)
		* commit id

	To test if the whole journal section was written correctly, every
	512-byte sector of the journal ends with 8-byte commit id. If the
	commit id matches on all sectors in a journal section, then it is
	assumed that the section was written correctly. If the commit id
	doesn't match, the section was written partially and it should not
	be replayed.

* one or more runs of interleaved tags and data.
    Each run contains:

	* tag area - it contains integrity tags. There is one tag for each
	  sector in the data area
	* data area - it contains data sectors. The number of data sectors
	  in one run must be a power of two. log2 of this value is stored
	  in the superblock.


code:
=====
dm-io
=====

Dm-io provides synchronous and asynchronous I/O services. There are three
types of I/O services available, and each type has a sync and an async
version.

The user must set up an io_region structure to describe the desired location
of the I/O. Each io_region indicates a block-device along with the starting
sector and size of the region::

   struct io_region {
      struct block_device *bdev;
      sector_t sector;
      sector_t count;
   };

Dm-io can read from one io_region or write to one or more io_regions. Writes
to multiple regions are specified by an array of io_region structures.

The first I/O service type takes a list of memory pages as the data buffer for
the I/O, along with an offset into the first page::

   struct page_list {
      struct page_list *next;
      struct page *page;
   };

   int dm_io_sync(unsigned int num_regions, struct io_region *where, int rw,
                  struct page_list *pl, unsigned int offset,
                  unsigned long *error_bits);
   int dm_io_async(unsigned int num_regions, struct io_region *where, int rw,
                   struct page_list *pl, unsigned int offset,
                   io_notify_fn fn, void *context);

The second I/O service type takes an array of bio vectors as the data buffer
for the I/O. This service can be handy if the caller has a pre-assembled bio,
but wants to direct different portions of the bio to different devices::

   int dm_io_sync_bvec(unsigned int num_regions, struct io_region *where,
                       int rw, struct bio_vec *bvec,
                       unsigned long *error_bits);
   int dm_io_async_bvec(unsigned int num_regions, struct io_region *where,
                        int rw, struct bio_vec *bvec,
                        io_notify_fn fn, void *context);

The third I/O service type takes a pointer to a vmalloc'd memory buffer as the
data buffer for the I/O. This service can be handy if the caller needs to do
I/O to a large region but doesn't want to allocate a large number of individual
memory pages::

   int dm_io_sync_vm(unsigned int num_regions, struct io_region *where, int rw,
                     void *data, unsigned long *error_bits);
   int dm_io_async_vm(unsigned int num_regions, struct io_region *where, int rw,
                      void *data, io_notify_fn fn, void *context);

Callers of the asynchronous I/O services must include the name of a completion
callback routine and a pointer to some context data for the I/O::

   typedef void (*io_notify_fn)(unsigned long error, void *context);

The "error" parameter in this callback, as well as the `*error` parameter in
all of the synchronous versions, is a bitset (instead of a simple error value).
In the case of an write-I/O to multiple regions, this bitset allows dm-io to
indicate success or failure on each individual region.

Before using any of the dm-io services, the user should call dm_io_get()
and specify the number of pages they expect to perform I/O on concurrently.
Dm-io will attempt to resize its mempool to make sure enough pages are
always available in order to avoid unnecessary waiting while performing I/O.

When the user is finished using the dm-io services, they should call
dm_io_put() and specify the same number of pages that were given on the
dm_io_get() call.


code:
=====================
Device-Mapper Logging
=====================
The device-mapper logging code is used by some of the device-mapper
RAID targets to track regions of the disk that are not consistent.
A region (or portion of the address space) of the disk may be
inconsistent because a RAID stripe is currently being operated on or
a machine died while the region was being altered.  In the case of
mirrors, a region would be considered dirty/inconsistent while you
are writing to it because the writes need to be replicated for all
the legs of the mirror and may not reach the legs at the same time.
Once all writes are complete, the region is considered clean again.

There is a generic logging interface that the device-mapper RAID
implementations use to perform logging operations (see
dm_dirty_log_type in include/linux/dm-dirty-log.h).  Various different
logging implementations are available and provide different
capabilities.  The list includes:

==============	==============================================================
Type		Files
==============	==============================================================
disk		drivers/md/dm-log.c
core		drivers/md/dm-log.c
userspace	drivers/md/dm-log-userspace* include/linux/dm-log-userspace.h
==============	==============================================================

The "disk" log type
-------------------
This log implementation commits the log state to disk.  This way, the
logging state survives reboots/crashes.

The "core" log type
-------------------
This log implementation keeps the log state in memory.  The log state
will not survive a reboot or crash, but there may be a small boost in
performance.  This method can also be used if no storage device is
available for storing log state.

The "userspace" log type
------------------------
This log type simply provides a way to export the log API to userspace,
so log implementations can be done there.  This is done by forwarding most
logging requests to userspace, where a daemon receives and processes the
request.

The structure used for communication between kernel and userspace are
located in include/linux/dm-log-userspace.h.  Due to the frequency,
diversity, and 2-way communication nature of the exchanges between
kernel and userspace, 'connector' is used as the interface for
communication.

There are currently two userspace log implementations that leverage this
framework - "clustered-disk" and "clustered-core".  These implementations
provide a cluster-coherent log for shared-storage.  Device-mapper mirroring
can be used in a shared-storage environment when the cluster log implementations
are employed.


code:
===============
dm-queue-length
===============

dm-queue-length is a path selector module for device-mapper targets,
which selects a path with the least number of in-flight I/Os.
The path selector name is 'queue-length'.

Table parameters for each path: [<repeat_count>]

::

	<repeat_count>: The number of I/Os to dispatch using the selected
			path before switching to the next path.
			If not given, internal default is used. To check
			the default value, see the activated table.

Status for each path: <status> <fail-count> <in-flight>

::

	<status>: 'A' if the path is active, 'F' if the path is failed.
	<fail-count>: The number of path failures.
	<in-flight>: The number of in-flight I/Os on the path.

Algorithm
=========

dm-queue-length increments/decrements 'in-flight' when an I/O is
dispatched/completed respectively.
dm-queue-length selects a path with the minimum 'in-flight'.

Examples
========
In case that 2 paths (sda and sdb) are used with repeat_count == 128.

::

  # echo "0 10 multipath 0 0 1 1 queue-length 0 2 1 8:0 128 8:16 128" \
    dmsetup create test
  #
  # dmsetup table
  test: 0 10 multipath 0 0 1 1 queue-length 0 2 1 8:0 128 8:16 128
  #
  # dmsetup status
  test: 0 10 multipath 2 0 0 0 1 1 E 0 2 1 8:0 A 0 0 8:16 A 0 0


code:
=======
dm-raid
=======

The device-mapper RAID (dm-raid) target provides a bridge from DM to MD.
It allows the MD RAID drivers to be accessed using a device-mapper
interface.

Mapping Table Interface
-----------------------
The target is named "raid" and it accepts the following parameters::

  <raid_type> <#raid_params> <raid_params> \
    <#raid_devs> <metadata_dev0> <dev0> [.. <metadata_devN> <devN>]

<raid_type>:

  ============= ===============================================================
  raid0		RAID0 striping (no resilience)
  raid1		RAID1 mirroring
  raid4		RAID4 with dedicated last parity disk
  raid5_n 	RAID5 with dedicated last parity disk supporting takeover
		Same as raid4

		- Transitory layout
  raid5_la	RAID5 left asymmetric

		- rotating parity 0 with data continuation
  raid5_ra	RAID5 right asymmetric

		- rotating parity N with data continuation
  raid5_ls	RAID5 left symmetric

		- rotating parity 0 with data restart
  raid5_rs 	RAID5 right symmetric

		- rotating parity N with data restart
  raid6_zr	RAID6 zero restart

		- rotating parity zero (left-to-right) with data restart
  raid6_nr	RAID6 N restart

		- rotating parity N (right-to-left) with data restart
  raid6_nc	RAID6 N continue

		- rotating parity N (right-to-left) with data continuation
  raid6_n_6	RAID6 with dedicate parity disks

		- parity and Q-syndrome on the last 2 disks;
		  layout for takeover from/to raid4/raid5_n
  raid6_la_6	Same as "raid_la" plus dedicated last Q-syndrome disk

		- layout for takeover from raid5_la from/to raid6
  raid6_ra_6	Same as "raid5_ra" dedicated last Q-syndrome disk

		- layout for takeover from raid5_ra from/to raid6
  raid6_ls_6	Same as "raid5_ls" dedicated last Q-syndrome disk

		- layout for takeover from raid5_ls from/to raid6
  raid6_rs_6	Same as "raid5_rs" dedicated last Q-syndrome disk

		- layout for takeover from raid5_rs from/to raid6
  raid10        Various RAID10 inspired algorithms chosen by additional params
		(see raid10_format and raid10_copies below)

		- RAID10: Striped Mirrors (aka 'Striping on top of mirrors')
		- RAID1E: Integrated Adjacent Stripe Mirroring
		- RAID1E: Integrated Offset Stripe Mirroring
		- and other similar RAID10 variants
  ============= ===============================================================

  Reference: Chapter 4 of
  https://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf

<#raid_params>: The number of parameters that follow.

<raid_params> consists of

    Mandatory parameters:
        <chunk_size>:
		      Chunk size in sectors.  This parameter is often known as
		      "stripe size".  It is the only mandatory parameter and
		      is placed first.

    followed by optional parameters (in any order):
	[sync|nosync]
		Force or prevent RAID initialization.

	[rebuild <idx>]
		Rebuild drive number 'idx' (first drive is 0).

	[daemon_sleep <ms>]
		Interval between runs of the bitmap daemon that
		clear bits.  A longer interval means less bitmap I/O but
		resyncing after a failure is likely to take longer.

	[min_recovery_rate <kB/sec/disk>]
		Throttle RAID initialization
	[max_recovery_rate <kB/sec/disk>]
		Throttle RAID initialization
	[write_mostly <idx>]
		Mark drive index 'idx' write-mostly.
	[max_write_behind <sectors>]
		See '--write-behind=' (man mdadm)
	[stripe_cache <sectors>]
		Stripe cache size (RAID 4/5/6 only)
	[region_size <sectors>]
		The region_size multiplied by the number of regions is the
		logical size of the array.  The bitmap records the device
		synchronisation state for each region.

        [raid10_copies   <# copies>], [raid10_format   <near|far|offset>]
		These two options are used to alter the default layout of
		a RAID10 configuration.  The number of copies is can be
		specified, but the default is 2.  There are also three
		variations to how the copies are laid down - the default
		is "near".  Near copies are what most people think of with
		respect to mirroring.  If these options are left unspecified,
		or 'raid10_copies 2' and/or 'raid10_format near' are given,
		then the layouts for 2, 3 and 4 devices	are:

		========	 ==========	   ==============
		2 drives         3 drives          4 drives
		========	 ==========	   ==============
		A1  A1           A1  A1  A2        A1  A1  A2  A2
		A2  A2           A2  A3  A3        A3  A3  A4  A4
		A3  A3           A4  A4  A5        A5  A5  A6  A6
		A4  A4           A5  A6  A6        A7  A7  A8  A8
		..  ..           ..  ..  ..        ..  ..  ..  ..
		========	 ==========	   ==============

		The 2-device layout is equivalent 2-way RAID1.  The 4-device
		layout is what a traditional RAID10 would look like.  The
		3-device layout is what might be called a 'RAID1E - Integrated
		Adjacent Stripe Mirroring'.

		If 'raid10_copies 2' and 'raid10_format far', then the layouts
		for 2, 3 and 4 devices are:

		========	     ============	  ===================
		2 drives             3 drives             4 drives
		========	     ============	  ===================
		A1  A2               A1   A2   A3         A1   A2   A3   A4
		A3  A4               A4   A5   A6         A5   A6   A7   A8
		A5  A6               A7   A8   A9         A9   A10  A11  A12
		..  ..               ..   ..   ..         ..   ..   ..   ..
		A2  A1               A3   A1   A2         A2   A1   A4   A3
		A4  A3               A6   A4   A5         A6   A5   A8   A7
		A6  A5               A9   A7   A8         A10  A9   A12  A11
		..  ..               ..   ..   ..         ..   ..   ..   ..
		========	     ============	  ===================

		If 'raid10_copies 2' and 'raid10_format offset', then the
		layouts for 2, 3 and 4 devices are:

		========       ==========         ================
		2 drives       3 drives           4 drives
		========       ==========         ================
		A1  A2         A1  A2  A3         A1  A2  A3  A4
		A2  A1         A3  A1  A2         A2  A1  A4  A3
		A3  A4         A4  A5  A6         A5  A6  A7  A8
		A4  A3         A6  A4  A5         A6  A5  A8  A7
		A5  A6         A7  A8  A9         A9  A10 A11 A12
		A6  A5         A9  A7  A8         A10 A9  A12 A11
		..  ..         ..  ..  ..         ..  ..  ..  ..
		========       ==========         ================

		Here we see layouts closely akin to 'RAID1E - Integrated
		Offset Stripe Mirroring'.

        [delta_disks <N>]
		The delta_disks option value (-251 < N < +251) triggers
		device removal (negative value) or device addition (positive
		value) to any reshape supporting raid levels 4/5/6 and 10.
		RAID levels 4/5/6 allow for addition of devices (metadata
		and data device tuple), raid10_near and raid10_offset only
		allow for device addition. raid10_far does not support any
		reshaping at all.
		A minimum of devices have to be kept to enforce resilience,
		which is 3 devices for raid4/5 and 4 devices for raid6.

        [data_offset <sectors>]
		This option value defines the offset into each data device
		where the data starts. This is used to provide out-of-place
		reshaping space to avoid writing over data while
		changing the layout of stripes, hence an interruption/crash
		may happen at any time without the risk of losing data.
		E.g. when adding devices to an existing raid set during
		forward reshaping, the out-of-place space will be allocated
		at the beginning of each raid device. The kernel raid4/5/6/10
		MD personalities supporting such device addition will read the data from
		the existing first stripes (those with smaller number of stripes)
		starting at data_offset to fill up a new stripe with the larger
		number of stripes, calculate the redundancy blocks (CRC/Q-syndrome)
		and write that new stripe to offset 0. Same will be applied to all
		N-1 other new stripes. This out-of-place scheme is used to change
		the RAID type (i.e. the allocation algorithm) as well, e.g.
		changing from raid5_ls to raid5_n.

	[journal_dev <dev>]
		This option adds a journal device to raid4/5/6 raid sets and
		uses it to close the 'write hole' caused by the non-atomic updates
		to the component devices which can cause data loss during recovery.
		The journal device is used as writethrough thus causing writes to
		be throttled versus non-journaled raid4/5/6 sets.
		Takeover/reshape is not possible with a raid4/5/6 journal device;
		it has to be deconfigured before requesting these.

	[journal_mode <mode>]
		This option sets the caching mode on journaled raid4/5/6 raid sets
		(see 'journal_dev <dev>' above) to 'writethrough' or 'writeback'.
		If 'writeback' is selected the journal device has to be resilient
		and must not suffer from the 'write hole' problem itself (e.g. use
		raid1 or raid10) to avoid a single point of failure.

<#raid_devs>: The number of devices composing the array.
	Each device consists of two entries.  The first is the device
	containing the metadata (if any); the second is the one containing the
	data. A Maximum of 64 metadata/data device entries are supported
	up to target version 1.8.0.
	1.9.0 supports up to 253 which is enforced by the used MD kernel runtime.

	If a drive has failed or is missing at creation time, a '-' can be
	given for both the metadata and data drives for a given position.

Example Tables
--------------

::

  # RAID4 - 4 data drives, 1 parity (no metadata devices)
  # No metadata devices specified to hold superblock/bitmap info
  # Chunk size of 1MiB
  # (Lines separated for easy reading)

  0 1960893648 raid \
          raid4 1 2048 \
          5 - 8:17 - 8:33 - 8:49 - 8:65 - 8:81

  # RAID4 - 4 data drives, 1 parity (with metadata devices)
  # Chunk size of 1MiB, force RAID initialization,
  #       min recovery rate at 20 kiB/sec/disk

  0 1960893648 raid \
          raid4 4 2048 sync min_recovery_rate 20 \
          5 8:17 8:18 8:33 8:34 8:49 8:50 8:65 8:66 8:81 8:82

Status Output
-------------
'dmsetup table' displays the table used to construct the mapping.
The optional parameters are always printed in the order listed
above with "sync" or "nosync" always output ahead of the other
arguments, regardless of the order used when originally loading the table.
Arguments that can be repeated are ordered by value.

'dmsetup status' yields information on the state and health of the array.
The output is as follows (normally a single line, but expanded here for
clarity)::

  1: <s> <l> raid \
  2:      <raid_type> <#devices> <health_chars> \
  3:      <sync_ratio> <sync_action> <mismatch_cnt>

Line 1 is the standard output produced by device-mapper.

Line 2 & 3 are produced by the raid target and are best explained by example::

        0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0

Here we can see the RAID type is raid4, there are 5 devices - all of
which are 'A'live, and the array is 2/490221568 complete with its initial
recovery.  Here is a fuller description of the individual fields:

	=============== =========================================================
	<raid_type>     Same as the <raid_type> used to create the array.
	<health_chars>  One char for each device, indicating:

			- 'A' = alive and in-sync
			- 'a' = alive but not in-sync
			- 'D' = dead/failed.
	<sync_ratio>    The ratio indicating how much of the array has undergone
			the process described by 'sync_action'.  If the
			'sync_action' is "check" or "repair", then the process
			of "resync" or "recover" can be considered complete.
	<sync_action>   One of the following possible states:

			idle
				- No synchronization action is being performed.
			frozen
				- The current action has been halted.
			resync
				- Array is undergoing its initial synchronization
				  or is resynchronizing after an unclean shutdown
				  (possibly aided by a bitmap).
			recover
				- A device in the array is being rebuilt or
				  replaced.
			check
				- A user-initiated full check of the array is
				  being performed.  All blocks are read and
				  checked for consistency.  The number of
				  discrepancies found are recorded in
				  <mismatch_cnt>.  No changes are made to the
				  array by this action.
			repair
				- The same as "check", but discrepancies are
				  corrected.
			reshape
				- The array is undergoing a reshape.
	<mismatch_cnt>  The number of discrepancies found between mirror copies
			in RAID1/10 or wrong parity values found in RAID4/5/6.
			This value is valid only after a "check" of the array
			is performed.  A healthy array has a 'mismatch_cnt' of 0.
	<data_offset>   The current data offset to the start of the user data on
			each component device of a raid set (see the respective
			raid parameter to support out-of-place reshaping).
	<journal_char>	- 'A' - active write-through journal device.
			- 'a' - active write-back journal device.
			- 'D' - dead journal device.
			- '-' - no journal device.
	=============== =========================================================

Message Interface
-----------------
The dm-raid target will accept certain actions through the 'message' interface.
('man dmsetup' for more information on the message interface.)  These actions
include:

	========= ================================================
	"idle"    Halt the current sync action.
	"frozen"  Freeze the current sync action.
	"resync"  Initiate/continue a resync.
	"recover" Initiate/continue a recover process.
	"check"   Initiate a check (i.e. a "scrub") of the array.
	"repair"  Initiate a repair of the array.
	========= ================================================

Discard Support
---------------
The implementation of discard support among hardware vendors varies.
When a block is discarded, some storage devices will return zeroes when
the block is read.  These devices set the 'discard_zeroes_data'
attribute.  Other devices will return random data.  Confusingly, some
devices that advertise 'discard_zeroes_data' will not reliably return
zeroes when discarded blocks are read!  Since RAID 4/5/6 uses blocks
from a number of devices to calculate parity blocks and (for performance
reasons) relies on 'discard_zeroes_data' being reliable, it is important
that the devices be consistent.  Blocks may be discarded in the middle
of a RAID 4/5/6 stripe and if subsequent read results are not
consistent, the parity blocks may be calculated differently at any time;
making the parity blocks useless for redundancy.  It is important to
understand how your hardware behaves with discards if you are going to
enable discards with RAID 4/5/6.

Since the behavior of storage devices is unreliable in this respect,
even when reporting 'discard_zeroes_data', by default RAID 4/5/6
discard support is disabled -- this ensures data integrity at the
expense of losing some performance.

Storage devices that properly support 'discard_zeroes_data' are
increasingly whitelisted in the kernel and can thus be trusted.

For trusted devices, the following dm-raid module parameter can be set
to safely enable discard support for RAID 4/5/6:

    'devices_handle_discards_safely'

Version History
---------------

::

 1.0.0	Initial version.  Support for RAID 4/5/6
 1.1.0	Added support for RAID 1
 1.2.0	Handle creation of arrays that contain failed devices.
 1.3.0	Added support for RAID 10
 1.3.1	Allow device replacement/rebuild for RAID 10
 1.3.2	Fix/improve redundancy checking for RAID10
 1.4.0	Non-functional change.  Removes arg from mapping function.
 1.4.1	RAID10 fix redundancy validation checks (commit 55ebbb5).
 1.4.2	Add RAID10 "far" and "offset" algorithm support.
 1.5.0	Add message interface to allow manipulation of the sync_action.
	New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt.
 1.5.1	Add ability to restore transiently failed devices on resume.
 1.5.2	'mismatch_cnt' is zero unless [last_]sync_action is "check".
 1.6.0	Add discard support (and devices_handle_discard_safely module param).
 1.7.0	Add support for MD RAID0 mappings.
 1.8.0	Explicitly check for compatible flags in the superblock metadata
	and reject to start the raid set if any are set by a newer
	target version, thus avoiding data corruption on a raid set
	with a reshape in progress.
 1.9.0	Add support for RAID level takeover/reshape/region size
	and set size reduction.
 1.9.1	Fix activation of existing RAID 4/10 mapped devices
 1.9.2	Don't emit '- -' on the status table line in case the constructor
	fails reading a superblock. Correctly emit 'maj:min1 maj:min2' and
	'D' on the status line.  If '- -' is passed into the constructor, emit
	'- -' on the table line and '-' as the status line health character.
 1.10.0	Add support for raid4/5/6 journal device
 1.10.1	Fix data corruption on reshape request
 1.11.0	Fix table line argument order
	(wrong raid10_copies/raid10_format sequence)
 1.11.1	Add raid4/5/6 journal write-back support via journal_mode option
 1.12.1	Fix for MD deadlock between mddev_suspend() and md_write_start() available
 1.13.0	Fix dev_health status at end of "recover" (was 'a', now 'A')
 1.13.1	Fix deadlock caused by early md_stop_writes().  Also fix size an
	state races.
 1.13.2	Fix raid redundancy validation and avoid keeping raid set frozen
 1.14.0	Fix reshape race on small devices.  Fix stripe adding reshape
	deadlock/potential data corruption.  Update superblock when
	specific devices are requested via rebuild.  Fix RAID leg
	rebuild errors.
 1.15.0 Fix size extensions not being synchronized in case of new MD bitmap
        pages allocated;  also fix those not occurring after previous reductions
 1.15.1 Fix argument count and arguments for rebuild/write_mostly/journal_(dev|mode)
        on the status line.


code:
===============
dm-service-time
===============

dm-service-time is a path selector module for device-mapper targets,
which selects a path with the shortest estimated service time for
the incoming I/O.

The service time for each path is estimated by dividing the total size
of in-flight I/Os on a path with the performance value of the path.
The performance value is a relative throughput value among all paths
in a path-group, and it can be specified as a table argument.

The path selector name is 'service-time'.

Table parameters for each path:

    [<repeat_count> [<relative_throughput>]]
	<repeat_count>:
			The number of I/Os to dispatch using the selected
			path before switching to the next path.
			If not given, internal default is used.  To check
			the default value, see the activated table.
	<relative_throughput>:
			The relative throughput value of the path
			among all paths in the path-group.
			The valid range is 0-100.
			If not given, minimum value '1' is used.
			If '0' is given, the path isn't selected while
			other paths having a positive value are available.

Status for each path:

    <status> <fail-count> <in-flight-size> <relative_throughput>
	<status>:
		'A' if the path is active, 'F' if the path is failed.
	<fail-count>:
		The number of path failures.
	<in-flight-size>:
		The size of in-flight I/Os on the path.
	<relative_throughput>:
		The relative throughput value of the path
		among all paths in the path-group.

Algorithm
=========

dm-service-time adds the I/O size to 'in-flight-size' when the I/O is
dispatched and subtracts when completed.
Basically, dm-service-time selects a path having minimum service time
which is calculated by::

	('in-flight-size' + 'size-of-incoming-io') / 'relative_throughput'

However, some optimizations below are used to reduce the calculation
as much as possible.

	1. If the paths have the same 'relative_throughput', skip
	   the division and just compare the 'in-flight-size'.

	2. If the paths have the same 'in-flight-size', skip the division
	   and just compare the 'relative_throughput'.

	3. If some paths have non-zero 'relative_throughput' and others
	   have zero 'relative_throughput', ignore those paths with zero
	   'relative_throughput'.

If such optimizations can't be applied, calculate service time, and
compare service time.
If calculated service time is equal, the path having maximum
'relative_throughput' may be better.  So compare 'relative_throughput'
then.

Examples
========
In case that 2 paths (sda and sdb) are used with repeat_count == 128
and sda has an average throughput 1GB/s and sdb has 4GB/s,
'relative_throughput' value may be '1' for sda and '4' for sdb::

  # echo "0 10 multipath 0 0 1 1 service-time 0 2 2 8:0 128 1 8:16 128 4" \
    dmsetup create test
  #
  # dmsetup table
  test: 0 10 multipath 0 0 1 1 service-time 0 2 2 8:0 128 1 8:16 128 4
  #
  # dmsetup status
  test: 0 10 multipath 2 0 0 0 1 1 E 0 2 2 8:0 A 0 0 1 8:16 A 0 0 4

Or '2' for sda and '8' for sdb would be also true::

  # echo "0 10 multipath 0 0 1 1 service-time 0 2 2 8:0 128 2 8:16 128 8" \
    dmsetup create test
  #
  # dmsetup table
  test: 0 10 multipath 0 0 1 1 service-time 0 2 2 8:0 128 2 8:16 128 8
  #
  # dmsetup status
  test: 0 10 multipath 2 0 0 0 1 1 E 0 2 2 8:0 A 0 0 2 8:16 A 0 0 8